07 Jul, 2014

2 commits

  • Regression was introduced in r10655.
    
    ECLServerHandler.java:
    In buildDomainContexts(), fixed the code to match the code behaviour pre r10655.
    
    git-svn-id: https://svn.forgerock.org/opendj/trunk@10869 41b1ffd8-f28e-4786-ab96-9950f0a78031
    JnRouvignac
     
  • In r10840, the change to JE/FileChangelogDB.getCursorFrom(DN, ServerState) unnecessarily triggers a lot of calls to the underlying DB (JE or file based) to retrieve the ChangelogState.
    As an optimization, keeping an in-memory version of the ChangelogState in synch with the on-disk version will help.
    
    
    ChangelogState.java:
    Now thread safe.
    Added removeOfflineReplica(), isEqualTo().
    Changed domainToServerIds from Map<DN, List<Integer>> to Map<DN, Set<Integer>>.
    In getOfflineReplicas(), now return a MultiDomainServerState.
    
    ChangeNumberIndexer.java:
    Consequence of the changes to ChangelogState.
    
    
    ReplicationDbEnv.java:
    Added changelogState field, updated at the same time as the on-disk changelogstate DB
    Added getChangelogState(), called by client code instead of readChangelogState().
    Renamed readChangelogState() to private readOnDiskChangelogState().
    Added stateLock field to sync updates to in-memory and on-disk changelog state.
    
    ReplicationEnvironment.java:
    Added changelogState field, updated at the same time as the on-disk changelogstate DB
    Added getChangelogState(), called by client code instead of readChangelogState().
    Renamed readChangelogState() to private readOnDiskChangelogState().
    Renamed domainLock field to domainsLock.
    
    replication.properties:
    Removed now unused error message.
    
    FileChangelogDB.java, JEChangelogDB.java:
    Consequence of the changes to ChangelogState and ReplicationEnvironment/ReplicationDbEnv.
    
    
    MultiDomainServerState.java, ServerState.java:
    Added getSnapshot() for unit tests.
    
    ReplicationEnvironmentTest.java, ReplicationDbEnvTest.java:
    Consequence of the changes to ReplicationEnvironment and ChangelogState.
    Used the fake server.
    
    git-svn-id: https://svn.forgerock.org/opendj/trunk@10868 41b1ffd8-f28e-4786-ab96-9950f0a78031
    JnRouvignac
     

04 Jul, 2014

2 commits


03 Jul, 2014

3 commits

  • This changed managed to solve the combined DS/RS case. Mostly, because robot test is sometimes failing yet.
    This is fixed by introducing a DSRSShutdownSync class which ensures that when the DS sends a ReplicaOfflineMsg, the RS will relay it to the rest of the topology.
    Second main change is to ensure ReplicaOfflineMsgs do not update the domain's ServerState otherwise the functional tests detect an inconsistency in the backend ds-sync-state data between the 2 replica: the one that is stopped and the one that is never stopped. For this I added UpdateMsg.contributesToDomainState() and used it throughout.
    The rest of the change is passing the DSRSShutdownSync down method calls.
    
    
    
    DSRSShutdownSync.java: ADDED
    
    UpdateMsg.java, ReplicaOfflineMsg.java:
    Added contributesToDomainState()
    
    
    
    MultimasterReplication.java:
    Added dsrsShutdownSync field + created it here + passed it down creation of LDAPReplicationDomain and ReplicationServerListener.
    Code cleanup.
    
    LDAPReplicationDomain.java:
    Added dsrsShutdownSync field + used it in publishReplicaOfflineMsg().
    
    ReplicationServerListener.java
    Added dsrsShutdownSync field + used it when creating ReplicationServer.
    Code cleanup.
    
    ReplicationServer.java:
    Added dsrsShutdownSync field + added getDSRSShutdownSync() getter.
    
    ServerHandler.java
    Called ReplicationServer.getDSRSShutdownSync() when creating ServerWriter.
    Code cleanup.
    
    ServerWriter.java:
    Added dsrsShutdownSync field + used it in run().
    
    
    
    ECLServerWriter.java:
    Consequence of the change to ServerWriter.
    Code cleanup.
    
    
    
    RemotePendingChanges.java, ReplicationDomain.java, MessageHandler.java:
    Used UpdateMsg.contributesToDomainState().
    
    git-svn-id: https://svn.forgerock.org/opendj/trunk@10861 41b1ffd8-f28e-4786-ab96-9950f0a78031
    JnRouvignac
     
  • Made use of constants and serverConstants where applicable.
    Fixed typos.
    Simplified tests.
    Fixed a couple of erroneous impossible condition tests.
    
    git-svn-id: https://svn.forgerock.org/opendj/trunk@10860 41b1ffd8-f28e-4786-ab96-9950f0a78031
    ludo
     
  • 
    ReplicationServerDomain.java:
    In put(), renamed update parameter to updateMsg.
    Extracted methods getPreparedAssuredInfo(), getAssuredServers(), isDifferentGenerationId(), isUpdateMsgFiltered().
    Dramatically simplified addUpdate().
    
    ServerReader.java, ServerWriter.java:
    Extracted method isUpdateMsgFiltered().
    
    git-svn-id: https://svn.forgerock.org/opendj/trunk@10858 41b1ffd8-f28e-4786-ab96-9950f0a78031
    JnRouvignac
     

02 Jul, 2014

2 commits

  • - hid ReplicationDomain internals
    - moved test code out of production code
    - removed duplicated test code
    
    
    ReplicationDomain.java:
    In ctor, delegate to other ctor.
    In startPublishService(), removed ReplicationDomainCfg parameter and get the config from field instead.
    Removed getConfig(), not used anymore.
    Moved publish(byte[]) to ReplicationDomainTest.
    
    LDAPReplicationDomain.java, AssuredReplicationServerTest.java, FakeStressReplicationDomain.java:
    Consequence of the change to ReplicationDomain.startPublishService().
    
    
    FakeReplicationDomain.java:
    Expanded the code to support what FractionalReplicationTest.FakeReplicationDomain was doing.
    Consequence of the change to ReplicationDomain.startPublishService().
    
    FractionalReplicationTest.java:
    Removed FakeReplicationDomain inner class, replaced by org.opends.server.replication.service.FakeReplicationDomain.
    Removed newConfig(), now unused.
    
    ReplicationDomainTest.java:
    Moved ReplicationDomain.publish(byte[]) here.
    Extracted method publishRepeatedly().
    
    git-svn-id: https://svn.forgerock.org/opendj/trunk@10855 41b1ffd8-f28e-4786-ab96-9950f0a78031
    JnRouvignac
     
  • CR-3920
    
    Third and last step : all cursors related to ChangeNumberindexDB now 
     behave like java.sql.ResultSet API. 
    
    * FileChangeNumberIndexDBCursor, JEChangeNumberIndexDBCursor: 
     now behave like ResultSet API
    
    * ECLServerHandler : adapt to changes of underlying cursor 
     on ChangeNumberIndexDB
    
    * FileChangeNumberIndexDBTest, JEChangeNumberIndexDBTest: 
     adapt to changes on cursor
    
     * ChangeNumberIndexDB : javadoc fix
     
     * ReplicationEnvironment : change default log file size to 10M
     
    
    git-svn-id: https://svn.forgerock.org/opendj/trunk@10854 41b1ffd8-f28e-4786-ab96-9950f0a78031
    nicolas.capponi@forgerock.com
     

01 Jul, 2014

3 commits

  • 
    
    MessageHandler.java:
    Extracted method fillLateQueue().
    
    MsgQueue.java:
    Added toString().
    
    PersistentServerState.java:
    Renamed field baseDn => baseDN.
    Made updateStateEntry() return a boolean instead of a ResultCode.
    Used Collections.singleton*() methods.
    Added toString().
    
    
    
    JEReplicaDBCursor.java:
    In next(), factorized code.
    
    ExternalChangeLogTest.java:
    Inlined a few passthrough test methods.
    Renamed createControls() to createCookieControl().
    In getControls(), removed an esoteric way to call Assert.fail(). Let the exception bubble up dammit!
    
    InternalSearchMonitorTestCase.java:
    Simplified code calling processSearch().
    In getMonitorNames(), do not copy the monitor names.
    
    
    BrowserController.java:
    Reduce fields + methods visibilities.
    Added final keyword to fields.
    Removed unused methods removeSuffix(), showAttributeName(), removeBrowserEventListener(), notifyChildEntryChanged(), notifyChildEntryAdded(), notifyChildEntryDeleted(), startRefresh(), shutDown(), getAttrsForGreenSearch(), entryArrayFromCollection(), nodeArrayFromCollection().
    Extracted methods getAciCount(), getNewIcon() and toInt().
    Used early returns.
    
    IconPool.java:
    Removed unused method maskedIcon().
    
    LDAPConnectionPool.java:
    Reduce fields + methods visibilities.
    Added final keyword to fields.
    Used static import for ConnectionUtils.
    Changed makeLDAPUrl() to accept a single InitialLdapContext parameter.
    Removed unused methods removeReferralAuthenticationListener(), flush(), getAuthDN(), getAuthPassword(), getRegisteredAuthentication().
    
    git-svn-id: https://svn.forgerock.org/opendj/trunk@10853 41b1ffd8-f28e-4786-ab96-9950f0a78031
    JnRouvignac
     
  • git-svn-id: https://svn.forgerock.org/opendj/trunk@10851 41b1ffd8-f28e-4786-ab96-9950f0a78031
    cjr
     
  • CR-3911
    
    Second step : all cursors related to ReplicaDB now behave like java.sql.ResultSet API
     in both JE and file-based implementations
    
    * FileChangelogDB#getCursorFrom(DN, int, CSN) now returns a java.sql.ResultSet style cursor
    
    * JEChangelogDB#getCursorFrom(DN, int, CSN) now returns a java.sql.ResultSet style cursor
    
    * ChangeNumberIndex : advance cursor to first record to keep current behavior
    
    * Update Javadoc in several classes : DBCursor,  ReplicationDomainDB, FileReplicaDB, JEReplicaDB,
     JEReplicaDBCursor, Log, LogFile
    
    * SequentialDBCursor (used in tests only) now behaves like java.sql.ResultSet API
    
    * ChangeNumberIndexerTest : adapt to changes of SequentialDBCursor
    
    * CompositeDBCursorTest : add more tests
    
    * ReplicaOfflineCursorTest : adapt to changes of SequentialDBCursor
    
    
    git-svn-id: https://svn.forgerock.org/opendj/trunk@10849 41b1ffd8-f28e-4786-ab96-9950f0a78031
    nicolas.capponi@forgerock.com
     

30 Jun, 2014

1 commit


26 Jun, 2014

1 commit

  • Added a new ReplicaOfflineMsg to communicate that a replica is offline. ReplicaOfflineMsg extends UpdateMsg. It is possible that OPENDJ-1260 will piggy back on this new message type.
    This patch makes a DS send a ReplicaOfflineMsg to its preferred RS on shutdown.
    This works when we have split DS-RS, but works only 50% (guesstimate) of the time with combined DS-RS. There is a race condition between shutdown and the ReplicaOfflineMsg being forwarded by the collocated RS.
    Last but not least, RSs can communicate such ReplicaOfflineMsg to other RSs via ServerWriter / ServerReader.
    
    Another last item: Due to current change, replication is querying the changelogstate a lot more often. This is not playing nice with ExternalChangeLogTest and GenerationIdTest and required changing the File-based changelog ReplicationEnvironment class.
    It might be necessary to reduce I/O to maintain an in memory copy of the changelogstate. I might do this in a subsequent commit.
    
    
    
    
    ReplicaOfflineMsg.java: ADDED
    
    ReplicationMsg.java:
    Added support for ReplicaOfflineMsg.
    
    ProtocolVersion.java:
    Updated javadoc for REPLICATION_PROTOCOL_V8.
    
    LDAPReplicationDomain.java, ReplicationDomain.java:
    Added publishReplicaOfflineMsg().
    
    ReplicationBroker.java:
    In stop(), called ReplicationDomain.publishReplicaOfflineMsg().
    
    ReplicationServerDomain.java:
    In publishUpdateMsg(UpdateMsg), handled ReplicaOfflineMsg.
    
    
    
    PendingChange.java:
    Changed msg field + getMsg() return type from LDAPUpdateMsg to UpdateMsg.
    Added getLDAPUpdateMsg().
    
    PendingChanges.java:
    Added putReplicaOfflineMsg().
    In pushCommittedChanges(), changed the code as a consequence of the change to PendingChange.getMsg().
    
    RemotePendingChanges.java:
    Changed the code as a consequence of the change to PendingChange.getMsg().
    
    
    
    JEChangelogDB.java:
    Added ReplicaOfflineCursor inner class, decorator for a DBCursor + used it in getCursorFrom() to return ReplicaOfflineMsg to ServerWriter thread.
    
    JEChangelogDBTest.java: ADDED
    Tested ReplicaOfflineCursor.
    
    ECLServerHandler.java:
    In getNextMessage(), ignore ReplicaOfflineMsg which are useless to searches on cn=changelog.
    
    
    
    SynchronizationMsgTest.java:
    Added a test for ReplicaOfflineMsg (de)serialization.
    
    FakeUpdateMsg.java: ADDED, extracted from CompositeDBCursorTest
    
    CompositeDBCursorTest.java:
    Replaced call to newUpdateMsg() by calling FakeUpdateMsg ctor.
    
    
    
    ReplicationEnvironment.java:
    Fixes to make ExternalChangeLogTest and GenerationIdTest work.
    
    replication.properties:
    Removed now useless messages.
    
    git-svn-id: https://svn.forgerock.org/opendj/trunk@10840 41b1ffd8-f28e-4786-ab96-9950f0a78031
    JnRouvignac
     

25 Jun, 2014

2 commits

  • First step : improve cursor behavior for file-based implementation only
    
    * Log.java, LogFile.java, FileReplicaDBCursor.java : change cursors to behave 
       like java.sql.ResultSet, ie cursor is positionned before the first record
     
    * FileReplicaDBCursor.java : fix behavior for case when record with
     start key is not available when cursor is created 
    
    * FileChangeNumberIndexDBCursor.java : update to adapt to new behavior
      of underlying cursor, but still behave differently than  java.sql.ResultSet
      (to be changed in the next step)
    
    * LogTest.java, LogFileTest.java : adapt tests to new behavior
    
    * FileReplicaDBTest.java : update test related to exhausted cursor 
     for better coverage of edge cases 
    
    
    git-svn-id: https://svn.forgerock.org/opendj/trunk@10839 41b1ffd8-f28e-4786-ab96-9950f0a78031
    nicolas.capponi@forgerock.com
     
  • git-svn-id: https://svn.forgerock.org/opendj/trunk@10838 41b1ffd8-f28e-4786-ab96-9950f0a78031
    mark
     

24 Jun, 2014

1 commit


23 Jun, 2014

6 commits


20 Jun, 2014

1 commit

  • * config.ldiff
    ** Add the 'changelog-read' value to the 'ds-default-root-privilege-name' multi-valued attribute
    * GlobalConfiguration.xml RootDNConfiguration.xml ADSContext.java Privilege.java RootPrivilegeChangeListener.java
    ** Add the 'changelog-read' privilege where is was needed
    * GlobalCfgDefn.properties RootDNCfgDefn.properties
    ** Add 'changelog-read' privilege definition
    * replication.properties replication_fr.properties
    ** Add messages to prevent user that he needs to have the 'changelog-read' privilege if he wants to search on changelog
    * ECLSearchOperation.java
    ** Add a check to verify that the current connection has the 'changelog-read' privilege before starting the changelog search
    * ExternalChangeLogTest.java
    ** Unit test which ensure that is not possible to perform a changelog search without the 'changelog-read' privilege 
    
    
    git-svn-id: https://svn.forgerock.org/opendj/trunk@10820 41b1ffd8-f28e-4786-ab96-9950f0a78031
    gaetan
     

19 Jun, 2014

1 commit


17 Jun, 2014

1 commit

  • …ver from a previous failure
    CR-3768
    
    * LogFile.java
    ** Add a check of file validity in constructor when log file is write-enabled
    ** Recover the file if it is corrupted (partially written record) by truncating it
    
    * BlockLogReader.java
    ** Add method checkLogIsValid() to check validity of file
     
    * replication.properties
    ** Add new messages related to check and recovery
    
    * LogFileTest.java
    ** Add tests for recovery after log file corruption
    
    * Minor changes in other files
    
    git-svn-id: https://svn.forgerock.org/opendj/trunk@10816 41b1ffd8-f28e-4786-ab96-9950f0a78031
    nicolas.capponi@forgerock.com
     

13 Jun, 2014

3 commits

  • Issue: On resetting the generationId in RS2, the ServerReader thread is blocked waiting on the ChangeNumberIndexer thread that is waiting on new changes.
    The fix consisted in ensuring the replicaDB cursors are released by the ChangeNumberIndexer thread when a replicaDB is being cleared.
    
    
    ChangeNumberIndexer.java:
    Changed AtomicBoolean doClear field into ConcurrentSkipListSet<DN> domainsToClear.
    Changed removeAllCursors() to removeCursors(DN baseDN).
    In clear(), added a DN parameter.
    
    JEChangelogDB.java, FileChangelogDB.java:
    Called ChangeNumberIndexer.clear() from removeDomain() rather than from clearDB(). 
    
    git-svn-id: https://svn.forgerock.org/opendj/trunk@10809 41b1ffd8-f28e-4786-ab96-9950f0a78031
    JnRouvignac
     
  • In decodeRecord(), changed changeNumber local variable from long to Long.
    Code cleanup.
    
    git-svn-id: https://svn.forgerock.org/opendj/trunk@10808 41b1ffd8-f28e-4786-ab96-9950f0a78031
    JnRouvignac
     
  • 
    ReplicationBroker.java:
    Reverted one change from r10786 which broke replication.fractional functional tests.
    
    git-svn-id: https://svn.forgerock.org/opendj/trunk@10807 41b1ffd8-f28e-4786-ab96-9950f0a78031
    JnRouvignac
     

12 Jun, 2014

2 commits


11 Jun, 2014

1 commit

  • …ing ReplicationBackend class
    
    Added upgrade tasks for:
    - Removing 'dc=replicationchanges' backend
    - Removing ACI for 'dc=replicationchanges'
    
    
    Upgrade.java:
    Registered upgrade tasks.
    
    UpgradeTasks.java:
    Added deleteConfigEntry().
    Factorized code by extracting method perform0().
    Fixed javadocs.
    
    UpgradeUtils.java:
    In updateConfigFile(), renamed lines parameter to ldifLines + changed String dn local variable to DN ldifDN + added support for deleting entries.
    
    tools.properties:
    Added messages for upgrade tasks.
    Fixed typos.
    
    
    admin_tool.properties:
    Fixed typos.
    
    SaltedMD5PasswordStorageSchemeTestCase.java:
    Removed unused import.
    
    git-svn-id: https://svn.forgerock.org/opendj/trunk@10800 41b1ffd8-f28e-4786-ab96-9950f0a78031
    JnRouvignac
     

10 Jun, 2014

1 commit

  • DirectoryServer.getClassLoader() is used by SNMPClassLoaderProvider, so made it public again.
    
    git-svn-id: https://svn.forgerock.org/opendj/trunk@10796 41b1ffd8-f28e-4786-ab96-9950f0a78031
    JnRouvignac
     

06 Jun, 2014

1 commit

  • Left over changes from a failed attempt at sending replica offline messages after all update messages have been sent on replica shutdown (see the JIRA issue for details):
    - Shutdown sequence improvements (reordered shutdown stages)
    - Code cleanups / refactorings / clarifications
    
    
    
    DirectoryServer.java:
    In shutDown(), removed dead code + reordered stages to: shutdown connection handlers, then work queue, then replication, then the rest.
    shutting down the work queue waits for the worker threads to exit for ServerShutdownMonitor.WAIT_TIME. To be improved by OPENDJ-1469 2 phase shutdown
    Made several constants final.
    Made several methods private.
    
    ServerShutdownMonitor.java:
    Extracted WAIT_TIME constants to reuse it in DirectoryServer.shutDown().
    
    ReplicationBroker.java
    Reordered shutdown sequence: first shutdown changeTime heartbeat publisher thread, then RS heartbeat monitoring thread, then set no connected RS.
    
    
    ReplicationDomain.java, DummyReplicationDomain.java:
    Made status private + added signalNewStatus().
    
    LDAPReplicationDomain.java:
    Consequence of the change to signalNewStatus().
    
    ReplicationServerDomain.java:
    Added PendingStatusMessages.toString().
    Made some methods private.
    In sendPendingTopologyMsgs(), avoid building a topology message if there is no RSs to send it to.
    
    
    PendingChange.java:
    Code cleanup.
    Removed useless field/methods targetDN, getTargetDN(), setOp().
    Added toString().
    
    PendingChanges.java:
    In putLocalOperation(), avoid storing synchronization operations because they will never be sent (see code in pushCommittedChanges()).
    In pushCommittedChanges(), made better use of TreeMap API + do not return int anymore (it was never used).
    In commitAndPushCommittedChanges(), do not return int anymore (it was never used).
    
    RemotePendingChanges.java:
    Consequence of the change to PendingChange.getTargetDN().
    Renamed targetDn local variables to targetDN.
    
    
    TraditionalWorkerThread.java, CSNGenerator.java:
    Code cleanup.
    
    git-svn-id: https://svn.forgerock.org/opendj/trunk@10786 41b1ffd8-f28e-4786-ab96-9950f0a78031
    JnRouvignac
     

05 Jun, 2014

1 commit

  • Added a test case for the change number computation + fixed the code to support it.
    In ChangeNumberIndexer.run(), fixed a bug where shutdown forces an insert in the Change number index DB.
    
    ChangeNumberIndexerTest.java:
    Added emptyDBTwoDSsOneKilled test.
    Fixed timestamps in a few tests.
    
    ChangeNumberIndexer.java:
    Initialized nextChangeForInsertDBCursor field to avoid NPE on startup.
    Added getOldestLastAliveCSN(), mightMoveForwardMediumConsistencyPoint() + used it in tryNotify().
    Changed canMoveForwardMediumConsistencyPoint() to canMoveForwardMediumConsistencyPoint(CSN).
    Removed mediumConsistency field, unused now.
    In run(), fixed a bug where shutdown forces an insert in the Change number index DB.
    Updated javadocs.
    
    MultiDomainServerState.java, MultiDomainServerStateTest.java:
    Added getOldestCSNExcluding() and isReplicaExcluded().
    
    git-svn-id: https://svn.forgerock.org/opendj/trunk@10781 41b1ffd8-f28e-4786-ab96-9950f0a78031
    JnRouvignac
     

04 Jun, 2014

3 commits

  • git-svn-id: https://svn.forgerock.org/opendj/trunk@10780 41b1ffd8-f28e-4786-ab96-9950f0a78031
    mark
     
  • replication topology with file-based changelog
    OPENDJ-1487 : File based changelog : cursors opened when clearing the log
    
    Log.java :
    * Ensure cursors opened on log are temporarily disabled before rotating
     the head (to avoid renaming the file while reader are opened on it)
    * Allow cursors to be opened on log when performing a clear on the log, 
     turning them into empty cursors. 
       
    
    git-svn-id: https://svn.forgerock.org/opendj/trunk@10778 41b1ffd8-f28e-4786-ab96-9950f0a78031
    nicolas.capponi@forgerock.com
     
  • Problem was due to an NPE while deserializing StartECLSessionMsg: ByteArrayScanner.nextString(), can return null, while previous code was always creating an empty string when using the older API.
    
    Thanks Nicolas Capponi for bisecting.
    
    
    StartECLSessionMsg.java:
    In ctor, added a null check to prevent NPE.
    
    git-svn-id: https://svn.forgerock.org/opendj/trunk@10777 41b1ffd8-f28e-4786-ab96-9950f0a78031
    JnRouvignac
     

03 Jun, 2014

2 commits