How to add tests to Parthenon
Unit Tests
Unit tests are straightforward to implement. Open the tst/unit
directory to see the current test suites. You may either add a new
file to this directory (and the associated CMakeLists.txt
file, or
extend an existing file.
Parthenon uses the Catch2 unit test framework. Tests are typically written in the following format:
TEST_CASE("Name", "[category1][category2]") {
GIVEN("Set up code") {
// some code
WHEN("Trigger") {
THEN("Condition") {
REQUIRE(some_bool_expression);
}
}
}
}
See the Catch documentation for more details.
Regression Tests
The regression test infrastructure is more complicated, and our
regression test infrastructure is built on a mix of Python and
CMake. Each test is defined by a test suite. You can find the test
suites in the tst/regression/test_suites
directory. Each test
suite is a Python module that defines a TestCase
class, which
inherits from the abstract base class provided by the
utils.test_case
module included in the test suite. A TestCase
class must implement the following methods:
Prepare(self, parameters, step)
is the python code which sets up a simulation run. It modifies an included input deck for a given test, based on the test design. Theparameters
input contains a list of command line arguments that should modify the parthenon run. These are passed in to the test infrastructure via CMake (described below). Thestep
argument is an integer. It is used for regression tests that require multiple simulation runs, such as a convergence test.Analyze(self, parameters)
is the post-processing step that checks whether or not the test passed. Some tests compare to gold files (described further below) and some simply compare to a known solution.
A test suite needs to have not only the python file containing the
TestCase
class, but an empty __init__.py
file to match the
Python module API.
After adding a module, you must also modify the file
tst/regression/CMakeLists.txt
. In particular for a new regression
test, you must add a set of arguments like these:
list(APPEND TEST_DIRS name_of_folder )
list(APPEND TEST_PROCS ${NUM_MPI_PROC_TESTING})
list(APPEND TEST_ARGS args_to_pass_to_run_test.py )
list(APPEND EXTRA_TEST_LABELS "my-awesome-test")
The first argument specifies the name of the folder containing the new python module for the new test. The second argument specifies number of MPI ranks if the test should be run with MPI (specify 1 if not). The third argument specifies arguments to pass to your test suite, for example
"--driver ${PROJECT_BINARY_DIR}/example/advection/advection-example --driver_input ${CMAKE_CURRENT_SOURCE_DIR}/test_suites/advection_performance/parthinput.advection_performance --num_steps 4"
which would specify which application to run for the test, as well as the input deck and the number of steps.
The final argument specifies labels attached to the test for use with CTest.
Gold Files
Many tests use so-called gold files, files are files containing
known results to compare against. Parthenon bundles its gold files as
part of releases. These files are automatically downloaded and are
located in tst/regression/gold_standard
. To add a new gold file
(or update an old one), place it in this directory.
To make the new (or updated) test official, you must add it to the
official test suite. First update
tst/regression/gold_standard/README.rst
and add a new version of
the test suite, corresponding to the commit where you added the
relevant test code and an explanation for why the gold files needed to
change. Then run the script `make_tarball.sh
as
bash make_tarball.sh NEW_VERSION
where NEW_VERSION
is the new version of the gold files (not
necessarily tied to the version of the code release). You can then ask
a maintainer to create a new goldfile release and attach the resultant
tarball to the release.
As a sanity check, Parthenon checks against the sha512
hash of the
tarball. The make tarball script will output the hash. The new version
and new hash must be set as the default values of the
REGRESSION_GOLD_STANDARD_VER
and REGRESSION_GOLD_STANDARD_HASH
in the top level CMakeLists.txt
file.