org.forgerock / org.forgerock.opendj

19 Jul, 2014

1 commit

18 Jul, 2014

3 commits

  • 24850d2b   Fixed CompositeDBCursorTest.recycleTwoElementsCursor() test. ... Browse Code » 
    git-svn-id: https://svn.forgerock.org/opendj/trunk@10913 41b1ffd8-f28e-4786-ab96-9950f0a78031
    JnRouvignac
     
  • 10df020c   OPENDJ-1441 (CR-4037) Persistent searches on external changelog do not return ch… ... Browse Code » 
    …anges for new replicas and new domains



This commit only fixes the "new replicas" case.

For this, I made the existing CompositeDBCursor abstract and implemented 2 new cursors: DomainDBCursor and MultiDomainDBCursor which iterate on a single replication domain or in a cross domain fashion.

These new cursors are able to react to topology changes like a new domain or a new replica.
However, ECL persistent search cannot take advantage of them right now because their list of ECLServerHandler.domainCtxts is initialized at the start of the search and never updated after.


Please note that I added new methods to ReplicationDomainDB, but several of them are just called by DomainDBCursor, MultiDomainDBCursor or ChangeNumberIndexer which are considered internal classes to the changelog DB. So maybe should we find a way to hide them from code client to the changelogDB.

These changes meant I could remove the awful double way to use the CompositeDBCursor + I could also remove most cursor management from ChangeNumberIndexer. I think I might find a way in a subsequent commit to also get rid of ChangeNumberIndexer.replicasOffline.

Alas the same changes are duplicated in JE + file based changelog. 



DomainDBCursor.java, MultiDomainDBCursor.java: ADDED

ReplicationDomainDB.java:
Added getCursorFrom(MultiDomainServerState startAfterState) and unregisterCursor(DBCursor) called by DomainDBCursor and MultiDomainDBCursor.

FileChangelogDB.java, JEChangelogDB.java:
Added registeredDomainCursors and registeredMultiDomainCursors fields.
In getExistingOrNewDomainMap(), updated MultiDomainDBCursors when a new domain is created.
In getCursorFrom(DN baseDN, int serverId, CSN), created the ReplicaOfflineCSN there from getCursorFrom(DN baseDN, ServerState).
Added newDomainDBCursor().
Reworked newOfflineCSN().
Implemented new methods in ReplicationDomainDB.
In getOrCreateReplicaDB(), updated DomainDBCursors when a new replica is created.
Synchronized the two files to ease diffing them together.

ChangeNumberIndexer.java:
Removed the responsibility to manage cursors from this class.
Removed allCursors, newCursors fields.
In publishUpdateMsg(), initialize(), moveForwardMediumConsistencyPoint() and run() removed code that dealt with creating/opening/recreating/removing cursors.
Removed resetNextChangeForInsertDBCursor(), ensureCursorExists(), removeCursors(), getCursor(), recycleExhaustedCursors(), createNewCursors(), getPrecedingCSN().
Made getPrecedingCSN() public static.
Added logUnexpectedException().


CompositeDBCursor.java:
Now abstract.
Removed ctor.
Removed recycleExhaustedCursors field.
Added incorporateNewCursors, isCursorNoLongerNeededFor(), cursorRemoved() and addCursor() + used them in next().
In next(), extracted recycleExhaustedCursors + used newly added removeNoLongerNeededCursors().
In close(), completed code.

ChangeNumberIndexerTest.java:
Consequence of the changes to ChangeNumberIndexer.
Renamed cursors field to replicaDBCursors.
Added multiDomainCursor, domainDBCursors fields.
Added eclEnabledDomains field to separate it from startCNIndexer().
Changed setup() and addReplica().

CompositeDBCursorTest.java:
Consequence of the change to CompositeDBCursor.

git-svn-id: https://svn.forgerock.org/opendj/trunk@10912 41b1ffd8-f28e-4786-ab96-9950f0a78031
    JnRouvignac
     
  • ec19c95f   Fix OPENDJ-1516: Trunk sends MSG_TYPE_REPLICA_OFFLINE messages to a v7 DS ... Browse Code » 
    git-svn-id: https://svn.forgerock.org/opendj/trunk@10910 41b1ffd8-f28e-4786-ab96-9950f0a78031
    cjr
     

16 Jul, 2014

1 commit

15 Jul, 2014

1 commit

10 Jul, 2014

3 commits

09 Jul, 2014

1 commit

08 Jul, 2014

1 commit

07 Jul, 2014

4 commits

  • 4b38c4f4   Fix unit-test breakage due to changes with the isEqualsTo method. ... Browse Code » 
    git-svn-id: https://svn.forgerock.org/opendj/trunk@10870 41b1ffd8-f28e-4786-ab96-9950f0a78031
    ludo
     
  • 99bef4e4   Fixed NPE on querying cn=changelog with persistent search. ... Browse Code » 
    Regression was introduced in r10655.

ECLServerHandler.java:
In buildDomainContexts(), fixed the code to match the code behaviour pre r10655.

git-svn-id: https://svn.forgerock.org/opendj/trunk@10869 41b1ffd8-f28e-4786-ab96-9950f0a78031
    JnRouvignac
     
  • fa189ecd   OPENDJ-1453 Replica offline messages should be synced with updates ... Browse Code » 
    In r10840, the change to JE/FileChangelogDB.getCursorFrom(DN, ServerState) unnecessarily triggers a lot of calls to the underlying DB (JE or file based) to retrieve the ChangelogState.
As an optimization, keeping an in-memory version of the ChangelogState in synch with the on-disk version will help.


ChangelogState.java:
Now thread safe.
Added removeOfflineReplica(), isEqualTo().
Changed domainToServerIds from Map<DN, List<Integer>> to Map<DN, Set<Integer>>.
In getOfflineReplicas(), now return a MultiDomainServerState.

ChangeNumberIndexer.java:
Consequence of the changes to ChangelogState.


ReplicationDbEnv.java:
Added changelogState field, updated at the same time as the on-disk changelogstate DB
Added getChangelogState(), called by client code instead of readChangelogState().
Renamed readChangelogState() to private readOnDiskChangelogState().
Added stateLock field to sync updates to in-memory and on-disk changelog state.

ReplicationEnvironment.java:
Added changelogState field, updated at the same time as the on-disk changelogstate DB
Added getChangelogState(), called by client code instead of readChangelogState().
Renamed readChangelogState() to private readOnDiskChangelogState().
Renamed domainLock field to domainsLock.

replication.properties:
Removed now unused error message.

FileChangelogDB.java, JEChangelogDB.java:
Consequence of the changes to ChangelogState and ReplicationEnvironment/ReplicationDbEnv.


MultiDomainServerState.java, ServerState.java:
Added getSnapshot() for unit tests.

ReplicationEnvironmentTest.java, ReplicationDbEnvTest.java:
Consequence of the changes to ReplicationEnvironment and ChangelogState.
Used the fake server.

git-svn-id: https://svn.forgerock.org/opendj/trunk@10868 41b1ffd8-f28e-4786-ab96-9950f0a78031
    JnRouvignac
     
  • 7a62dbf2   OPENDJ-1509 (CR-3963) The opendj-ldap-sdk is redundant and can be removed ... Browse Code » 
    Remove the opendj-ldap-sdk project and update dependencies in all pom files and in the project documentation.
* all pom.xml
** Removing reference to the opendj-ldap-sdk project and add dependencies to opendj-core and openj-grizzly
* opendj-core/.../index.xml.vm
** Updating the opendj-core documentation with the opendj-ldap-sdk old documentation page
* opendj-project site.xml chap-get-sdk.xml
** Updating project documentation menu and developer guide

git-svn-id: https://svn.forgerock.org/opendj/trunk@10867 41b1ffd8-f28e-4786-ab96-9950f0a78031
    gaetan
     

04 Jul, 2014

3 commits

03 Jul, 2014

4 commits

  • 6f6d1e62   OPENDJ-1453 (CR-3938) Replica offline messages should be synced with updates ... Browse Code » 
    This changed managed to solve the combined DS/RS case. Mostly, because robot test is sometimes failing yet.
This is fixed by introducing a DSRSShutdownSync class which ensures that when the DS sends a ReplicaOfflineMsg, the RS will relay it to the rest of the topology.
Second main change is to ensure ReplicaOfflineMsgs do not update the domain's ServerState otherwise the functional tests detect an inconsistency in the backend ds-sync-state data between the 2 replica: the one that is stopped and the one that is never stopped. For this I added UpdateMsg.contributesToDomainState() and used it throughout.
The rest of the change is passing the DSRSShutdownSync down method calls.



DSRSShutdownSync.java: ADDED

UpdateMsg.java, ReplicaOfflineMsg.java:
Added contributesToDomainState()



MultimasterReplication.java:
Added dsrsShutdownSync field + created it here + passed it down creation of LDAPReplicationDomain and ReplicationServerListener.
Code cleanup.

LDAPReplicationDomain.java:
Added dsrsShutdownSync field + used it in publishReplicaOfflineMsg().

ReplicationServerListener.java
Added dsrsShutdownSync field + used it when creating ReplicationServer.
Code cleanup.

ReplicationServer.java:
Added dsrsShutdownSync field + added getDSRSShutdownSync() getter.

ServerHandler.java
Called ReplicationServer.getDSRSShutdownSync() when creating ServerWriter.
Code cleanup.

ServerWriter.java:
Added dsrsShutdownSync field + used it in run().



ECLServerWriter.java:
Consequence of the change to ServerWriter.
Code cleanup.



RemotePendingChanges.java, ReplicationDomain.java, MessageHandler.java:
Used UpdateMsg.contributesToDomainState().

git-svn-id: https://svn.forgerock.org/opendj/trunk@10861 41b1ffd8-f28e-4786-ab96-9950f0a78031
    JnRouvignac
     
  • 115c364e   Code cleanup. ... Browse Code » 
    Made use of constants and serverConstants where applicable.
Fixed typos.
Simplified tests.
Fixed a couple of erroneous impossible condition tests.

git-svn-id: https://svn.forgerock.org/opendj/trunk@10860 41b1ffd8-f28e-4786-ab96-9950f0a78031
    ludo
     
  • 89512276   Factorized code into CommonArguments.getSearchScope(). ... Browse Code » 
    git-svn-id: https://svn.forgerock.org/opendj/trunk@10859 41b1ffd8-f28e-4786-ab96-9950f0a78031
    JnRouvignac
     
  • 60213812   Code cleanup. ... Browse Code » 
    
ReplicationServerDomain.java:
In put(), renamed update parameter to updateMsg.
Extracted methods getPreparedAssuredInfo(), getAssuredServers(), isDifferentGenerationId(), isUpdateMsgFiltered().
Dramatically simplified addUpdate().

ServerReader.java, ServerWriter.java:
Extracted method isUpdateMsgFiltered().

git-svn-id: https://svn.forgerock.org/opendj/trunk@10858 41b1ffd8-f28e-4786-ab96-9950f0a78031
    JnRouvignac
     

02 Jul, 2014

4 commits

  • 05d1a6d0   Code cleanup. ... Browse Code » 
    AssuredReplicationServerTest.java:
Added SafeReadAssertions with sensible defaults to allow for fluent style assertions on of the FakeReplicationDomain stats.
In checkTimeAndMonitoringSafeData(), extracted methods checkAckOccured() and checkTimeOutOccured().
Used assertj SoftAssertions for blocks of asserts.
In checkServerErrors(), used assertj Assertions.assertThat()
Removed checkServerErrorListsAreEqual() superseded by assertj Assertions.assertThat(Map).

git-svn-id: https://svn.forgerock.org/opendj/trunk@10857 41b1ffd8-f28e-4786-ab96-9950f0a78031
    JnRouvignac
     
  • 83426fb1   Code cleanup. ... Browse Code » 
    AssuredReplicationServerTest.java:
Removed FakeReplicationDomain.everyUpdatesAreOk duplicating FakeReplicationDomain.nWrongReceivedUpdates.
Added assert methods to inner classes + removed unneeded getters.
Replaced assertContainsOnly() with using assertj.

git-svn-id: https://svn.forgerock.org/opendj/trunk@10856 41b1ffd8-f28e-4786-ab96-9950f0a78031
    JnRouvignac
     
  • 7d54f469   Code cleanup: ... Browse Code » 
    - hid ReplicationDomain internals
- moved test code out of production code
- removed duplicated test code


ReplicationDomain.java:
In ctor, delegate to other ctor.
In startPublishService(), removed ReplicationDomainCfg parameter and get the config from field instead.
Removed getConfig(), not used anymore.
Moved publish(byte[]) to ReplicationDomainTest.

LDAPReplicationDomain.java, AssuredReplicationServerTest.java, FakeStressReplicationDomain.java:
Consequence of the change to ReplicationDomain.startPublishService().


FakeReplicationDomain.java:
Expanded the code to support what FractionalReplicationTest.FakeReplicationDomain was doing.
Consequence of the change to ReplicationDomain.startPublishService().

FractionalReplicationTest.java:
Removed FakeReplicationDomain inner class, replaced by org.opends.server.replication.service.FakeReplicationDomain.
Removed newConfig(), now unused.

ReplicationDomainTest.java:
Moved ReplicationDomain.publish(byte[]) here.
Extracted method publishRepeatedly().

git-svn-id: https://svn.forgerock.org/opendj/trunk@10855 41b1ffd8-f28e-4786-ab96-9950f0a78031
    JnRouvignac
     
  • 520afa5b   OPENDJ-1471 File based changelog : improve cursor behavior ... Browse Code » 
    CR-3920

Third and last step : all cursors related to ChangeNumberindexDB now 
 behave like java.sql.ResultSet API. 

* FileChangeNumberIndexDBCursor, JEChangeNumberIndexDBCursor: 
 now behave like ResultSet API

* ECLServerHandler : adapt to changes of underlying cursor 
 on ChangeNumberIndexDB

* FileChangeNumberIndexDBTest, JEChangeNumberIndexDBTest: 
 adapt to changes on cursor

 * ChangeNumberIndexDB : javadoc fix
 
 * ReplicationEnvironment : change default log file size to 10M
 

git-svn-id: https://svn.forgerock.org/opendj/trunk@10854 41b1ffd8-f28e-4786-ab96-9950f0a78031
    nicolas.capponi@forgerock.com
     

01 Jul, 2014

4 commits

  • 3e926ae5   Code cleanups. ... Browse Code » 
    

MessageHandler.java:
Extracted method fillLateQueue().

MsgQueue.java:
Added toString().

PersistentServerState.java:
Renamed field baseDn => baseDN.
Made updateStateEntry() return a boolean instead of a ResultCode.
Used Collections.singleton*() methods.
Added toString().



JEReplicaDBCursor.java:
In next(), factorized code.

ExternalChangeLogTest.java:
Inlined a few passthrough test methods.
Renamed createControls() to createCookieControl().
In getControls(), removed an esoteric way to call Assert.fail(). Let the exception bubble up dammit!

InternalSearchMonitorTestCase.java:
Simplified code calling processSearch().
In getMonitorNames(), do not copy the monitor names.


BrowserController.java:
Reduce fields + methods visibilities.
Added final keyword to fields.
Removed unused methods removeSuffix(), showAttributeName(), removeBrowserEventListener(), notifyChildEntryChanged(), notifyChildEntryAdded(), notifyChildEntryDeleted(), startRefresh(), shutDown(), getAttrsForGreenSearch(), entryArrayFromCollection(), nodeArrayFromCollection().
Extracted methods getAciCount(), getNewIcon() and toInt().
Used early returns.

IconPool.java:
Removed unused method maskedIcon().

LDAPConnectionPool.java:
Reduce fields + methods visibilities.
Added final keyword to fields.
Used static import for ConnectionUtils.
Changed makeLDAPUrl() to accept a single InitialLdapContext parameter.
Removed unused methods removeReferralAuthenticationListener(), flush(), getAuthDN(), getAuthPassword(), getRegisteredAuthentication().

git-svn-id: https://svn.forgerock.org/opendj/trunk@10853 41b1ffd8-f28e-4786-ab96-9950f0a78031
    JnRouvignac
     
  • 79d068c0   Fix OPENDJ-1187: DSML Gateway SOAP 1.2 Support ... Browse Code » 
    git-svn-id: https://svn.forgerock.org/opendj/trunk@10851 41b1ffd8-f28e-4786-ab96-9950f0a78031
    cjr
     
  • 80d7e2e5   Fix test mistake that sets side effect to followup tests (leaves the server with… ... Browse Code » 
    … strict syntax for telephone number which makes some import tests to fail later)..

git-svn-id: https://svn.forgerock.org/opendj/trunk@10850 41b1ffd8-f28e-4786-ab96-9950f0a78031
    ludo
     
  • 9d68f2f1   Checkpoint commit for OPENDJ-1471 File based changelog : improve cursor behavior ... Browse Code » 
    CR-3911

Second step : all cursors related to ReplicaDB now behave like java.sql.ResultSet API
 in both JE and file-based implementations

* FileChangelogDB#getCursorFrom(DN, int, CSN) now returns a java.sql.ResultSet style cursor

* JEChangelogDB#getCursorFrom(DN, int, CSN) now returns a java.sql.ResultSet style cursor

* ChangeNumberIndex : advance cursor to first record to keep current behavior

* Update Javadoc in several classes : DBCursor,  ReplicationDomainDB, FileReplicaDB, JEReplicaDB,
 JEReplicaDBCursor, Log, LogFile

* SequentialDBCursor (used in tests only) now behaves like java.sql.ResultSet API

* ChangeNumberIndexerTest : adapt to changes of SequentialDBCursor

* CompositeDBCursorTest : add more tests

* ReplicaOfflineCursorTest : adapt to changes of SequentialDBCursor


git-svn-id: https://svn.forgerock.org/opendj/trunk@10849 41b1ffd8-f28e-4786-ab96-9950f0a78031
    nicolas.capponi@forgerock.com
     

30 Jun, 2014

5 commits

26 Jun, 2014

1 commit

  • 68233346   OPENDJ-1453 (CR-3870) Replica offline messages should be synced with updates ... Browse Code » 
    Added a new ReplicaOfflineMsg to communicate that a replica is offline. ReplicaOfflineMsg extends UpdateMsg. It is possible that OPENDJ-1260 will piggy back on this new message type.
This patch makes a DS send a ReplicaOfflineMsg to its preferred RS on shutdown.
This works when we have split DS-RS, but works only 50% (guesstimate) of the time with combined DS-RS. There is a race condition between shutdown and the ReplicaOfflineMsg being forwarded by the collocated RS.
Last but not least, RSs can communicate such ReplicaOfflineMsg to other RSs via ServerWriter / ServerReader.

Another last item: Due to current change, replication is querying the changelogstate a lot more often. This is not playing nice with ExternalChangeLogTest and GenerationIdTest and required changing the File-based changelog ReplicationEnvironment class.
It might be necessary to reduce I/O to maintain an in memory copy of the changelogstate. I might do this in a subsequent commit.




ReplicaOfflineMsg.java: ADDED

ReplicationMsg.java:
Added support for ReplicaOfflineMsg.

ProtocolVersion.java:
Updated javadoc for REPLICATION_PROTOCOL_V8.

LDAPReplicationDomain.java, ReplicationDomain.java:
Added publishReplicaOfflineMsg().

ReplicationBroker.java:
In stop(), called ReplicationDomain.publishReplicaOfflineMsg().

ReplicationServerDomain.java:
In publishUpdateMsg(UpdateMsg), handled ReplicaOfflineMsg.



PendingChange.java:
Changed msg field + getMsg() return type from LDAPUpdateMsg to UpdateMsg.
Added getLDAPUpdateMsg().

PendingChanges.java:
Added putReplicaOfflineMsg().
In pushCommittedChanges(), changed the code as a consequence of the change to PendingChange.getMsg().

RemotePendingChanges.java:
Changed the code as a consequence of the change to PendingChange.getMsg().



JEChangelogDB.java:
Added ReplicaOfflineCursor inner class, decorator for a DBCursor + used it in getCursorFrom() to return ReplicaOfflineMsg to ServerWriter thread.

JEChangelogDBTest.java: ADDED
Tested ReplicaOfflineCursor.

ECLServerHandler.java:
In getNextMessage(), ignore ReplicaOfflineMsg which are useless to searches on cn=changelog.



SynchronizationMsgTest.java:
Added a test for ReplicaOfflineMsg (de)serialization.

FakeUpdateMsg.java: ADDED, extracted from CompositeDBCursorTest

CompositeDBCursorTest.java:
Replaced call to newUpdateMsg() by calling FakeUpdateMsg ctor.



ReplicationEnvironment.java:
Fixes to make ExternalChangeLogTest and GenerationIdTest work.

replication.properties:
Removed now useless messages.

git-svn-id: https://svn.forgerock.org/opendj/trunk@10840 41b1ffd8-f28e-4786-ab96-9950f0a78031
    JnRouvignac
     

25 Jun, 2014

4 commits