org.forgerock / org.forgerock.opendj

02 Sep, 2007

1 commit

  • 990f858a   There are several improvements to the unit test framework in this commit. ... Browse Dir » 
    * Test methods are no longer interleaved between classes.  All test
methods in a class are run together, with @BeforeClass and @AfterClass
methods called immediately before and after the methods are run.  As
part of this fix, you are now required to include sequential=true in
every class level @Test annotation.  If you don't do this, the build
will complain.

* Added a TestCaseUtils.restartServer() method that will do an in core 
restart of the directory server during the tests.  This can be used in a 
@BeforeClass method to ensure that the tests start with a clean 
directory server, and also in an @AfterClass method to cleanup after a 
test that makes a lot of configuration changes.  So if you introduce a 
new test that runs fine in isolation but fails when run with other 
tests, you could try calling TestCaseUtils.restartServer() in its 
@BeforeClass method.  The TestCaseUtils.restartServer() method will 
reinitialize the server and reload the original test configuration, but 
it's not quite the same as creating a completely new process.  
Specifically, it cannot undo any changes that were made to static member 
variables of a class.  I've fixed a handful of places in the server 
where this was a problem, but there might be more lurking.  If you write 
a test that changes static member variables of a class, please make sure 
that it cleans up after itself in an @AfterMethod or @AfterClass test.

* The tests now use significantly much less memory.  I saw a peak of
only 80MB.  There were two main problems.  1) TestNG holds on to all
parameters and results for the whole test, and 2) since the test
classes themselves live for the duration of the tests, their member
variables were holding onto a lot of garbage.  The in-core restart
made this problem much worse because we ended up with lots of copies
of the Schema, ConfigEntryS, etc.  I've introduced some hacks to fix
this.  Basically the code uses devious methos to go in and null out
the parameters and member variables after the test has run.  If you're
curious about the details of how we've addressed this take a look at
the comments in DirectoryServerTestCase.  From now on, all test
classes must inherit directly or indirectly from
DirectoryServerTestCase.  The build will fail if they don't.

* Upgrade to TestNG b5.7.  There is a fix in this release that helps
our tests to run in order, and I've had to make a couple of more
fixes, which they will eventually put back into the trunk.

* In classes with a class-level @Test annotation, TestNG treats any
public method (except @Before/After* methods) as a test method.  The
build now points this out and asks you to either add a specific @Test
annotation to the method or change the method to be non-public.  I've
fixed up several places where a test wasn't annotated and others where
a non-test method was being treated as a test method.

* The tests now report progress as they run.  Run 'build testhelp' to
see details on how to control the output.


I've also added some new test properties, mainly to make debugging the 
tests easier

* test.remote.debug.port:  This test property allows you to remotely 
attach a debugger to the unit tests.  If you provide a valid port value, 
the unit tests will not start to run until the debugger is attached.

* testng.verbosity0to5:  This test property controls the debugging 
output of TestNG.  This output is useful to check the order in which 
test methods are invoked or other details on what TestNG is doing.  
Valid values are integer values from 0 (no output) to 5 (maximum 
output).  Since this implicitly sets 
-Dorg.opends.test.suppressOutput=false, other stderr/stdout output 
generated by the unit tests will also be displayed.

* org.opends.test.copyClassesToTestPackage=true:  This test property 
copies the classes into the test server root.  This enables you to run 
the server tools on the test server.  It can slow down the test startup 
so the files are no longer copied by default.

Thanks to Neil for trying out these changes and to him and Bo for
helping me track down some of the memory leaks.


git-svn-id: https://svn.forgerock.org/opendj/trunk@2902 41b1ffd8-f28e-4786-ab96-9950f0a78031
    davidely
     

22 Jan, 2007

1 commit

  • 45449b89   1. Wrapped TestNG suite runner with our own (SuiteRunner.java). When there is … ... Browse Dir » 
    …an uncaught exception in the tests, this gives us the stack trace and forces the tests to exit.  This should make it easier to track down OutOfMemoryErrors in the future.

2.  Added our own listener for the tests.  This does the following
  * As soon as a test fails, it prints it out all of the info to the screen.  Test method, error message, stacktrace, parameters
  * At the end of the tests, it summarizes all of the tests that failed.
  * It generates a new test report in text format.  The html report generator was a real memory hog.

3.  Changes to build.xml
   * Test output is suppressed by default.
   * The coverage report is generated even if the tests failed.
   * Exposed the ability to rerun only the failed tests with -Dtest.failed=true.
   * The 'runtests' target prints out information about additional test options.
   * Two small fixes so that ant doesn't recompile files that haven't changed.
   * Removed tabs.  This was mostly in the status panel code.
   * Reverted the -Xmx256M temporary fix for the unit tests.




git-svn-id: https://svn.forgerock.org/opendj/trunk@1005 41b1ffd8-f28e-4786-ab96-9950f0a78031
    davidely
     

03 Sep, 2006

1 commit

28 Jun, 2006

1 commit