org.forgerock / org.forgerock.opendj

02 Jul, 2015

3 commits

  • 27f5f2cf   OPENDJ-2183 ECL: changes for new suffix are missing ... Browse Code » 
    OPENDJ-2184 ECL: wrong number of changes after adding a new replica

Code review: CR-7431



This problem is linked to OPENDJ-2141, where we introduced a "default" value for a CSN when starting from a changeNumberIndex record.

Explanation: when the server starts up, the newest record is read from the change number index db, so the change number indexer knows where to start.
From there, we must start all the cursors, and we are using the newest CSN as a starting point.
The triplet (LESS_THAN_OR_EQUAL_TO_KEY, ON_MATCHING_KEY, newestCsn) is then used to start all cursors, including the cursors for replica DBs created dynamically much after server startup.
When opening a cursor for any new replica, the provided default CSN cannot be found in the replica DB. Because of this condition and the provided triplet, the file-based changelog code creates an empty cursor.
Empty cursors cannot be recycled to iterate over the populated replica DB (And in that case, the replica DB already has the change csn3!!).

I think this is incorrect. This change fixes this.



LogTest.java, LogFileTest.java, BlockLogReaderWriterTest:
Changed the unit tests, so the returned values match what we need.

Log.java:
In getCursor(), changed the condition that creates a valid cursor to also include LESS_THAN_OR_EQUAL_TO_KEY.

BlockLogReader.java:
In positionToKeySequentially() and getMatchingRecord(), removed the code handling LESS_THAN_OR_EQUAL_TO_KEY and AFTER_MATCHING_KEY.
Renamed positionToKeySequentially() to positionToKey().


git-svn-id: https://svn.forgerock.org/opendj/trunk@12605 41b1ffd8-f28e-4786-ab96-9950f0a78031
    JnRouvignac
     
  • c47902af   OPENDJ-2189: Empty EntryIDSet are kept rather than deleted. ... Browse Code » 
    git-svn-id: https://svn.forgerock.org/opendj/trunk@12603 41b1ffd8-f28e-4786-ab96-9950f0a78031
    ylecaillez
     
  • 363563d3   Fix ChangelogBackendTestCase randomness ... Browse Code » 
    ChangelogBackendTestCase.java:
Introduced assertLastCookieIsEqualTo() to do an equivalent to "wait until keyword suceeds"
In searchInCookieModeOnTwoSuffixesWithPrivateBackend(), do not forget to close the domain.
In isOldestCSNForReplica(), fixed error message.
Overrode getGenerationId() to have a fixed generationId. This ensures messages can always sent from ReplicationBroker to LDAPReplicationDomain and processed.

git-svn-id: https://svn.forgerock.org/opendj/trunk@12600 41b1ffd8-f28e-4786-ab96-9950f0a78031
    JnRouvignac
     

01 Jul, 2015

5 commits

  • aff111d3   OPENDJ-2189: Empty EntryIDSet are kept rather than deleted. ... Browse Code » 
    git-svn-id: https://svn.forgerock.org/opendj/trunk@12597 41b1ffd8-f28e-4786-ab96-9950f0a78031
    ylecaillez
     
  • a3f4fdd5   CR-7464 OPENDJ-2187 Add server Javadoc to OpenDJ site ... Browse Code » 
    This patch works for me with Java 7 and the following command:
MAVEN_OPTS="-Xmx4g -XX:MaxPermSize=512m -Djava.awt.headless=true" mvn
-Pman-pages,docs -DskipTests=true clean install site

git-svn-id: https://svn.forgerock.org/opendj/trunk@12595 41b1ffd8-f28e-4786-ab96-9950f0a78031
    mark
     
  • 4876fc14   ChangelogBackendTestCase.java: ... Browse Code » 
    Hopefully say forever goodbye to randomness of searchInChangeNumberModeOnOneSuffixMultipleTimes().

The fixxed assertion error message was:
Failure Cause: java.lang.AssertionError: [In entry dn:
firstchangenumber: 1
lastchangenumber: 8
changelog: cn=changelog
lastExternalChangelogCookie: o=test:00000149a520c3ed04b10000000a;
incorrect value for attr 'lastchangenumber']
Expecting:
<"8">
to be equal to:
<"9">
ignoring case considerations
org.opends.server.backends.ChangelogBackendTestCase.assertAttributeValue(ChangelogBackendTestCase.java:1381)
org.opends.server.backends.ChangelogBackendTestCase.assertChangelogAttributesInRootDSE(ChangelogBackendTestCase.java:857)
org.opends.server.backends.ChangelogBackendTestCase.searchInChangeNumberModeOnOneSuffixMultipleTimes(ChangelogBackendTestCase.java:588)


The error was due to the CSN of the 9nth change coming after the previous CSN, based on time.
How is that possible? I have absolutely no idea since both were created by using which should never go backwards AFAIK.

git-svn-id: https://svn.forgerock.org/opendj/trunk@12594 41b1ffd8-f28e-4786-ab96-9950f0a78031
    JnRouvignac
     
  • fbee61b5   Attempt to fix randomness in ChangelogBackendTest ... Browse Code » 
    ChangelogBackendTest.java:
Only stop replication domains and brokers after the test method has finished, not during the test. Otherwise MultimasterReplication.isECLEnabledDomain(DN) risked returning false for valid domains.

git-svn-id: https://svn.forgerock.org/opendj/trunk@12593 41b1ffd8-f28e-4786-ab96-9950f0a78031
    JnRouvignac
     
  • a97f889d   Used try-with-resources as much as possible ... Browse Code » 
    git-svn-id: https://svn.forgerock.org/opendj/trunk@12592 41b1ffd8-f28e-4786-ab96-9950f0a78031
    JnRouvignac
     

30 Jun, 2015

3 commits

29 Jun, 2015

2 commits

  • 3f8fdf4d   Make checkstyle happy (sigh) ... Browse Code » 
    git-svn-id: https://svn.forgerock.org/opendj/trunk@12582 41b1ffd8-f28e-4786-ab96-9950f0a78031
    JnRouvignac
     
  • 07abf78d   OPENDJ-2179 (CR-7413) ECL: full resync step is no longer valid ... Browse Code » 
    

This was found due to one functional test that was working with 2.6 code base, but no longer works with 2.8.

Here is the sequence of events:

time1 change 1 => changenumber=1, changelogcookie=dc=com:csn1;
time2 ldapsearch (changenumber=1) returns entry with changenumber=1 and changelogcookie=dc=com:csn1;
time3 enable domain dc=fr
time4 change 2 => changenumber=2, changelogcookie=dc=com:csn1;dc=fr:csn2;

2.6:
time5 ldapsearch (changenumber=1) returns entry with changenumber=1 and changelogcookie=dc=com:csn1;dc=fr;
time6 ldapsearch (changelogcookie=dc=com:csn1;) resultcode=UNWILLING_TO_PERFORM message=Full resynch is required...[...]...use dc=com:csn1;dc=fr;
time6 ldapsearch (changelogcookie=dc=com:csn1;dc=fr;) resultcode=SUCCESS returns changes with csn1 and csn2

2.8:
time5 ldapsearch (changenumber=1) returns entry with changenumber=1 and changelogcookie=dc=com:csn1;
time6 ldapsearch (changelogcookie=dc=com:csn1;) resultcode=UNWILLING_TO_PERFORM message=Full resynch is required...[...]...use dc=com:csn1;dc=fr;
time6 ldapsearch (changelogcookie=dc=com:csn1;dc=fr;) resultcode=SUCCESS returns changes with csn1 and csn2

Note the different values for changelogcookie attribute at time4.

I think this difference is due to the fact the change number index DB was storing the previous cookie (thus returning it), while the new change number index DB is not storing it anymore (thus building it on the fly). So the new domain baseDN cannot be included until one change has been seen for it.

I think this error message was originally added as the result of a leaky abstraction: In 2.6, doing a search in the cookie mode was driven by the cookie provided by the user. So when a search was performed using a cookie that did not have all domains, no results would be returned for the missing domains. Hence this restriction was added to force users to iterate over all the domains.

With the new changelog implementation, all domains are iterated by default and this restriction is no longer necessary.

By removing this limitation here is how the results would compare:

time1 change 1 => changenumber=1, changelogcookie=dc=com:csn1;
time2 ldapsearch (changenumber=1) returns entry with changenumber=1 and changelogcookie=dc=com:csn1;
time3 enable domain dc=fr
time4 change 2 => changenumber=2, changelogcookie=dc=com:csn1;dc=fr:csn2;

2.6:
time5 ldapsearch (changenumber=1) returns entry with changenumber=1 and changelogcookie=dc=com:csn1;dc=fr;
time6 ldapsearch (changelogcookie=dc=com:csn1;) resultcode=UNWILLING_TO_PERFORM message=Full resynch is required...[...]...use dc=com:csn1;dc=fr;
time6 ldapsearch (changelogcookie=dc=com:csn1;dc=fr;) resultcode=SUCCESS returns changes with csn1 and csn2

2.8:
time5 ldapsearch (changenumber=1) returns entry with changenumber=1 and changelogcookie=dc=com:csn1;
time6 ldapsearch (changelogcookie=dc=com:csn1;) resultcode=SUCCESS returns changes with csn1 and csn2



ReplicationServer.java:
Removed error ERR_RESYNC_REQUIRED_MISSING_DOMAIN_IN_PROVIDED_COOKIE, no longer necessary with the new changelog implementation.

git-svn-id: https://svn.forgerock.org/opendj/trunk@12580 41b1ffd8-f28e-4786-ab96-9950f0a78031
    JnRouvignac
     

26 Jun, 2015

5 commits

25 Jun, 2015

6 commits

24 Jun, 2015

3 commits

  • ab468b52   CR-7377 OPENDJ-2110 Doc how proxy auth affects resource limits ... Browse Code » 
    This patch indicates in both parts of the docs
that resource limits under proxied authorization
apply to the bind DN user, not the proxied authz identity.

git-svn-id: https://svn.forgerock.org/opendj/trunk@12562 41b1ffd8-f28e-4786-ab96-9950f0a78031
    mark
     
  • 35f40612   MultimasterReplication.java: ... Browse Code » 
    Removed getDomain(), it is better to use Map.get() here.

git-svn-id: https://svn.forgerock.org/opendj/trunk@12561 41b1ffd8-f28e-4786-ab96-9950f0a78031
    JnRouvignac
     
  • eb31277d   Fix site links ... Browse Code » 
    Nightly builds are currently for 3.0.0-SNAPSHOT.
This aligns the site links to docs with the nightlies.

git-svn-id: https://svn.forgerock.org/opendj/trunk@12560 41b1ffd8-f28e-4786-ab96-9950f0a78031
    mark
     

23 Jun, 2015

6 commits

  • e2d94feb   CR-7365 OPENDJ-2132 Mention OPENDJ_JAVA_HOME ... Browse Code » 
    This patch mentions the need to use OPENDJ_JAVA_HOME
(or OPENDJ_JAVA_BIN) to set the Java environment correctly
for GUI installation.

This also means that OpenDJ is dropping support for Java WebStart.

git-svn-id: https://svn.forgerock.org/opendj/trunk@12558 41b1ffd8-f28e-4786-ab96-9950f0a78031
    mark
     
  • 0f06398d   Fixed missed javadoc indentation ... Browse Code » 
    git-svn-id: https://svn.forgerock.org/opendj/trunk@12557 41b1ffd8-f28e-4786-ab96-9950f0a78031
    JnRouvignac
     
  • f4bb4754   Renamed openMethod local variables to importCommand. ... Browse Code » 
    git-svn-id: https://svn.forgerock.org/opendj/trunk@12556 41b1ffd8-f28e-4786-ab96-9950f0a78031
    JnRouvignac
     
  • d62ae5e4   OPENDJ-2148 CR-7317 "NullPointerException" when running import-ldif online with exclude branch ... Browse Code » 
    Take out include/exclude branch reduction from the import code to a new class, returning what kind of policy should be used for creating a Suffix.
The same class can be used by upper layers during import to decide if the operation is supported or not.



git-svn-id: https://svn.forgerock.org/opendj/trunk@12552 41b1ffd8-f28e-4786-ab96-9950f0a78031
    fabiop
     
  • 42012e7f   Fixed unit tests after r12545 ... Browse Code » 
    git-svn-id: https://svn.forgerock.org/opendj/trunk@12549 41b1ffd8-f28e-4786-ab96-9950f0a78031
    JnRouvignac
     
  • b6ca83dc   OPENDJ-2141 (CR-7337) Cannot find entry in cn=changelog when searching with changelogCookie filter ... Browse Code » 
    Joint effort with Nicolas Capponi. Follow up of preparatory refactoring in r12534.

The problem lies in ChangelogCookie.initializeCookieForChangeNumberMode(). Here, the cookie is correctly initialized for the starting change number index record, but incorrectly for all the other replicas. The effect is that other replicas are incorrectly cursoring from the beginning of the replicaDB rather than the change immediately preceding the change number index record.
The fix is to make all replicaDBs cursor from the change immediately preceding the change number index record.


ChangelogBackend.java:
Added updateCookieToMediumConsistencyPoint() method to initialize a cookie from a ECLMultiDomainDBCursor.


ECLMultiDomainDBCursor.java:
Removed badly implemented methods getSnapshot(), toCookie() which returned the wrong result.

CompositeDBCursor.java
Removed badly implemented method getSnapshot() which returned the wrong result.
In cursors comparator, ensured more consistent ordering when the CSNs are equal.

ChangeNumberIndexer.java:
Added cookie field to replace the use of ECLMultiDomainDBCursor.toCookie().
Reimplemented initializeNextChangeCursor(), by calling ChangelogBackend.updateUpTo() + removed getCookieInitializedWithNewestCSN().


FileChangelogDB.java, JEChangelogDB.java:
In getCursorFrom(baseDN, serverId, startCSN, options), catered for the case where the provided startCSN does not come from this serverId.

FileReplicaDB.java, JEReplicaDB.java:
In generateCursorFrom(), catered for the case where the provided startCSN does not come from this serverId.

git-svn-id: https://svn.forgerock.org/opendj/trunk@12545 41b1ffd8-f28e-4786-ab96-9950f0a78031
    JnRouvignac
     

19 Jun, 2015

3 commits

  • 053fc193   OPENDJ-2154 (CR-7325) search on cn=monitor changenumber index database times out ... Browse Code » 
    
With a very large changelogDb, when issuing queries on cn=monitor, the code counts the number of entries in the changeNumberIndexDb. In order to do the count, it cursors through the changeNumberIndexDb, decoding each and every change in the changeNumberIndexDb.
This is a killer with very large changelogDb.
Any query on cn=monitor with very large file-based changelogDb is now at risk to timeout.

JE has a direct API to count the number of records in a tree.
File based changelog does not have such API, thus, a first implementation was to count the changes by cursoring through the changeNumberIndexDb.
However, since the changeNumberIndexDb purge has been reimplemented, there are no longer "holes" in the changeNumbers. As a consequence, a simple way to count the number of changes is to do "newestChangeNumber - oldestChangeNumber".



FileChangeNumberIndexDB.java:
In DbMonitorProvider.getMonitorData(), changed the code to make a diff between first and last changeNumber.
Introduced ChangeNumberType enum to get rid of unreadable boolean parameter.
Extracted toAttribute() from createChangeNumberAttribute().

git-svn-id: https://svn.forgerock.org/opendj/trunk@12535 41b1ffd8-f28e-4786-ab96-9950f0a78031
    JnRouvignac
     
  • a78405c8   OPENDJ-2141 Preliminary work - introduce CursorOptions ... Browse Code » 
    Code review: Jean-Noel 

Preliminary work to fix the issue

Add new class DBCursor.CursorOptions to hold options for
 cursor creation

git-svn-id: https://svn.forgerock.org/opendj/trunk@12534 41b1ffd8-f28e-4786-ab96-9950f0a78031
    nicolas.capponi@forgerock.com
     
  • a9bc177a   CR-7318 OPENDJ-2152 fix boolean properties in tools.properties files ... Browse Code » 
    git-svn-id: https://svn.forgerock.org/opendj/trunk@12533 41b1ffd8-f28e-4786-ab96-9950f0a78031
    cjr
     

17 Jun, 2015

3 commits

16 Jun, 2015

1 commit