How to use ARB unit testing --------------------------- 1. Invocation Edit config.makefile. Set UNIT_TESTS to 1 cd $ARBHOME make unit_tests 'make all' runs tests as well ('make build' doesn't) 2. Add unit test code Go to the unit containing your_function() and add something like the following to the end of the file. #if (UNIT_TESTS == 1) #include void TEST_your_function() { } #endif The names of the test functions are important: - they have to start with 'TEST_' and - need to be visible at global scope. (see also 'Building tests' at end of file) Test result: If TEST_your_function() does nothing special, it is assumed the unit test passed "ok". If your function fails an assertion (TEST_ASSERT) or raises SIGSEGV in any other manner, it is assumed the unit test failed and an error message is printed to the console. (Any other test function in the same unit will nevertheless get called.) Post conditions: If a unit defines one or more tests called void TEST_POSTCOND_xxx() each of these post condition tests will be run after EACH single unit test. These post condition tests are intended to check for unwanted global side-effects of test-functions. Such side effects often let other (unrelated) test-functions fail unexpectedly. Use them to undo these side-effects. If the post-condition fails, the test itself will be reported as failure. If the test itself has failed for other reasons, post condition tests will nevertheless be executed (and may do their cleanups), but their failures will not be shown in the log. Failing tests: If you have some broken behavior that you cannot fix now, please do NOT leave the test as "failing". Instead change TEST_ASSERT into TEST_ASSERT_BROKEN(failing_condition); That will print a warning as reminder as long as the condition fails. When the behavior was fixed (so that the condition is fulfilled now), it will abort the test as failed! Just change TEST_ASSERT_BROKEN back into TEST_ASSERT then. Set WARN_LEVEL to 0 in Makefile.setup.local to disable warnings. Random failures: Some tests tend to fail randomly if started from inside jenkins (due to parallel builds), whereas they behave proper and provide safety during normal development. You should disable such tests for jenkins by enclosing the whole test with: #if !defined(DEVEL_JENKINS) #endif One example for such a test is ../CORE/arb_cs.cxx@TEST_open_socket Another possibility to handle random failures is to allow multiple tries. This can be done using the macro TEST_ALLOW_RETRY(count). If called from testcode this will allow the test to run 'count' times, failing only if all 'count' calls of the test function result in failure. Note: TEST_ALLOW_RETRY has no effect if a test already gets retried! Missing tests: You may drop a reminder like MISSING_TEST(describe what is missing); which will appear as warning during test run. Order of tests: Tests are executed in the order they appear in the code. Multiple Files are sorted alphabethically. Exceptions: - If test name starts with 'TEST_###' (where ### are digits), it declares a test with priority ###. The default priority is 100. Lower numbers mean higher priority, i.e. start more early. - Other predefined priorities are - TEST_BASIC_ (=20) - TEST_EARLY_ (=50) - TEST_ (=100 default) - TEST_LATE_ (=200) - TEST_SLOW_ (=900) - TEST_AFTER_SLOW_ (=910) TEST_SLOW_... is meant to indicate "slow" tests (i.e. tests which need more than a second to execute). Order of test units: Tests from each unit (aka library) run consecutive. Tests of different units are executed asynchronously. Code differences when compiled with UNIT_TESTS=1 Some unit tests require differences in production code, e.g. - to inspect class internals - to provoke special conditions (e.g. reduce some buffer size to simulate big amount of processed data) Such differences should be marked with a comment containing 'UT_DIFF'. 3. Valgrinding test code Note: valgrind was not used for some time and might have rotten. I've completely switched to using sanitizers -> see ../config.makefile If you set VALGRIND=A in UNIT_TESTER/Makefile.setup.local, valgrind will be started on the test-binary after the normal unit tests passed with success. Valgrind errors/warnings will not raise an error or abort testing. If you set VALGRIND=B valgrind will be started BEFORE running tests. Useful if test fail unexpectedly and you suspect a memory bug. See also ENABLE_CRASH_TESTS in test_unit.h (all crash tests are reported by valgrind) If you set VALGRIND=E some external tools like arb_pt_server will be spawned valgrinded and the generated reports will be printed on test termination. Any combination of the above works as well. 4. Check test-coverage set 'COVERAGE=1' in config.makefile and 'make rebuild' - prints out uncovered code to compile log - leaves info for partly covered code files in yourcode.cxx.cov Note: You can silence lines which are never reached while testing with a comment containing 'NEED_NO_COV'. call 'make clean_coverage_results' to get rid of .cov files (see also showCoverageForAll in gcov2msg.pl) 5. Define which tests shall run [look into Makefile.setup.local for examples] a. Restrict tests to one or several libraries: Set 'RESTRICT_LIB=lib1[:libN]*' to run only tests defined in the specified libraries. Set 'RESTRICT_LIB=' to test all libraries. b. Restrict tests to specific modules: Set "RESTRICT_MODULE='regexpr'" to run only tests located in matching modules. Use '.' to match any modules. Prefix whole regexpr with '!' to match all but listed modules. c. Restrict tests by name: Set "RESTRICT_FUN='regexpr'" to run only tests whose function-name matches the regexpr. Use '.' to match any test. Prefix whole regexpr with '!' to match all but listed tests. d. Restrict test frequency: Set 'SKIP_SLOW=min' to restrict frequency of "slow tests". Slow tests are all tests named 'TEST_SLOW_.*'. These tests will not run more often than the specified amount of minutes. Set 'SKIP_SLOW=0' to always run all tests (default). e. Run unchanged tests: By default only changed unit-tests will run. Set 'ONLY_CHANGED_TESTS=0' to force testing unchanged code. Tests will re-run if .. - the test code was changed, - any needed library was changed or if - Makefile.setup.local or anything in the UnitTester itself was changed. Calling 'make clean' in $ARBHOME will also force all tests to re-run. 6. Global test environments Environments provide a test situation which takes some reasonable time for startup. Multiple tests will be permitted to use one environment, but they will be serialized (even if tests are running in multiple processes). These environments have to be coded directly in TestEnvironment.cxx (for existing environments see TestEnvironment.cxx@ExistingEnvironments) If you write a test that wishes to use one of the environments simply write TEST_SETUP_GLOBAL_ENVIRONMENT("name"); // starts environment 'name' 7. Auto-patch generation Whenever unit tests complete successfully, a patch (svn diff) is generated and stored inside $ARBHOME/patches.arb/ After PATCHES_KEEP_HOURS seconds the patch will be deleted automatically (see Makefile.setup.local), considering at least PATCHES_MIN_KEPT patches remain. 8. Building tests To build and execute a test-executable for a previously untested subdir the following steps are necessary: - add the test to the list of performed tests in ../Makefile@UNITS_TESTED - if the test subject is a shared library, be sure to mention it a Makefile.test@SHAREDLIBS - if the test subject is an executable, be sure to mention it a Makefile.test@EXEOBJDIRS