Running the Unit Tests

Overview

We use CTest for building and running our unit tests. This wraps the underlying cxxtest or other (e.g. pyunit) test code.

CMake/CTest: Command Line

The unit tests are currently excluded from the ‘all’ target. To build all of the tests, including the dependent framework code run:

cmake --build . --target AllTests

To build only one package of tests (and its dependencies):

cmake --build . --target KernelTest

To run all the tests:

ctest -j8

To build and run all the tests in one shot:

cmake --build . --target check

To run a specific test or set of tests (will run all those that match the search string):

ctest -R KernelTest_TimerTest

So to run all tests in a suite (using a search string):

ctest -j8 -R KernelTest

To exclude things from your tests (matches the string as with the -R option) - useful for those building the performance tests:

ctest -j8 -E Performance

Useful CTest Options

ctest -h gives the full list of command line arguments. Some useful ones to note are:

  • --output-on-failure: displays the log and any stderr output that is otherwise hidden by CTest.

  • --schedule-random: run the tests in a random order. Useful to weed out accidental dependencies

  • --repeat-until-fail: require each test to run times without failing in order to pass. Useful to try and find random failures

Running Unit Tests Directly

Starting in your build folder:

  • Running an entire test suite (e.g. KernelTest):

    ./bin/KernelTest
    
  • Running a specific test class.

    ./bin/KernelTest MyTestClassName
    
  • Running a specific test from a CxxTest test class (not possible via CTest).

    ./bin/KernelTest MyTestClassName MySingleTestName
    
  • Running a specific test from a Python unittest test class (not possible via CTest).

    PYTHONPATH=/path/to/build/bin python3 /path/to/src/Framework/PythonInterface/test/python/plugins/algorithms/MeanTest.py MeanTest.test_mean
    

Running Unit Tests With Visual Studio and ctest

Open the Mantid solution in Visual Studio. To run a subset of tests (for example UnitTests/AlgorithmsTest);

  • In the Solution Explorer, right click the project for the tests (in this case UnitTests/AlgorithmsTest) and select Properties.

  • In the Debugging tab of Properties change the Command Arguments box to the name of the test, for example “AddNoteTest”.

  • Right click the directory again and select Debug->Start new instance.

  • Once the build has finished, open a file browser and navigate to the mantid build directory, run the command-prompt.bat file to open a command prompt and run

    ctest -C Debug -V -R AddNoteTest
    

    For this example, there should be several lines of output ending with the time taken and the line

    100% tests passed, 0 tests failed out of 1
    

    Omitting the -R AddNoteTest option runs all the tests, but note that any tests which were not built according to the above instructions will fail. Adding the -V increases the output verbosity.

Running Unit Tests With Visual Studio

The unit tests can be run from within Visual Studio, following steps 1-3 above, with the addition in step 2 of;

  • Add the name of the test to the Target Name field in the General tab of Properties. Then add a breakpoint somewhere in the test header file.

Debugging unit tests

See the instructions here