######################################################## Thursday B: Adding a filter test to JEDI ######################################################## In this session you will practice adding a test to a new QC filter you developed on Wednesday(a). Requirements ------------ - A terminal session connected to your AWS node. You can use either SSH or the web-based JupyterLab interface. - Have successfully added :code:`PracticalBoundsCheck` filter to UFO filters by following Wednesday(a) activity instruction. - Have built and compiled fv3-bundle Introduction ------------ In the activity session a on Wednesday, you practiced adding a new filter to UFO. Today we will develop a test for this filter to ensure it works properly. Testing in JEDI ~~~~~~~~~~~~~~~ Each JEDI bundle has it's own suite of tests. After building and compiling the bundle, you can run the tests using :code:`ctest`. .. code:: bash cd /ufo ctest This will run all tests in ufo. When the tests are complete, ctest will print out a summary, highlighting which tests, if any, failed. To run a single test, you can use :code:`-R` followed by the test's name, for example: .. code:: bash ctest -R ufo_coding_norms The output from these tests will be printed to the screen and written to the file :code:`LastTest.log` in the directory :code:`/Testing/Temporary`. In the same directory :code:`LastTestsFailed.log` lists the last tests that failed. You can run ctests with the verbose option to get more information which can be helpful for debugging. .. code:: bash ctest -V -R test_ufo_geovals and for extra verbose: .. code:: bash ctest -VV -R test_ufo_geovals Another way to get more information is to set one or both of these environment variables before you run ctest: .. code:: bash export OOPS_DEBUG=1 export OOPS_TRACE=1 ctest also has an option to only re-run the tests that failed last time: .. code:: bash ctest --rerun-failed Please refer to (`JEDI documentation`_) for more information on JEDI test suite. Keep in mind that when you add a new feature to the JEDI repository you need to write a test for your code. This way you ensure your code is working properly and it will help us review and merge your code quicker. Step 1: Access Your AWS instance -------------------------------- Follow the same method as Monday to connect to your AWS compute node. Don't forget to enter the singularity container using this command: .. code:: bash singularity shell -e jedi-gnu-openmpi-dev_latest.sif Step 2: Review the new added filter ----------------------------------- In this exercise, we will write and add a test for :code:`PracticalBoundsCheck` filter you added in the practical session on Wednesday (a). We want to push the changes back to the GitHub repository so make sure you are in the correct branch ( :code:`feature/new_qc_filter_example` ) on the ufo repository. Enter :code:`~/jedi/fv3-bundle-day3/ufo` and check the branch name by using command: .. code:: bash git branch Let's review how we added :code:`PracticalBoundsCheck` filter to the ufo repository yesterday. In :code:`ufo/src/ufo/filters` you have added :code:`PracticalBoundsCheck.h` and :code:`PracticalBoundsCheck.cc`. This filter checks to see if your data is within a specified range or not. Take a look that the function :code:`PracticalBoundsCheck::applyFilter` in :code:`PracticalBoundsCheck.cc` for more details: .. code:: C++ void PracticalBoundsCheck::applyFilter(const std::vector & apply, const Variables & filtervars, std::vector> & flagged) const { const float missing = util::missingValue(missing); ufo::Variables testvars; // This filter is applied based on var@ObsValue values. testvars += ufo::Variables(filtervars, "ObsValue"); // Read minvalue and maxvalue from YAML configuration const float vmin = config_.getFloat("minvalue", missing); const float vmax = config_.getFloat("maxvalue", missing); // Sanity checks if (filtervars.nvars() == 0) { oops::Log::error() << "No variables will be filtered out in filter " << config_ << std::endl; ABORT("No variables specified to be filtered out in filter"); } // Loop over all variables to filter for (size_t jv = 0; jv < testvars.nvars(); ++jv) { // get test data for this variable std::vector testdata; data_.get(testvars.variable(jv), testdata); // apply the filter // filter out var@ObsValue >vmax and vmax) flagged[jv][jobs] = true; } } } } Step 3: Add YAML configuration file for the new test ----------------------------------------------------- We will use :code:`filters_testdata.nc4`, a simplified IODA format file, for testing our new filter. Take a look at this dataset by using :code:`ncdump` command. .. code:: bash cd /ufo/test/Data/ufo/testinput_tier_1 ncdump filters_testdata.nc4 | less If :code:`/ufo/test/Data/ufo` is empty this means you haven't run the ufo tests. :code:`ufo_get_*` tests download ioda, ufo, and crtm files. You can run all :code:`ufo_get_*` tests using the command below: .. code:: bash ctest -R ufo_get To test your filter you need to first add YAML configuration file in :code:`ufo/test/testinput` . In this directory, YAML configuration files with prefix :code:`qc_` are used for testing various filters in UFO. In this file, you can specify the details of how you want to test your filter. For example, the name of the file and list of variables in the file you want to apply the filter on. Create a new YAML file in :code:`ufo/test/testinput` called :code:`qc_practical_boundscheck.yaml`. Copy and paste this to your YAML file. .. code:: yaml window begin: 2018-01-01T00:00:00Z window end: 2019-01-01T00:00:00Z observations: - obs space: name: test data obsdatain: obsfile: Data/ufo/testinput_tier_1/filters_testdata.nc4 simulated variables: [variable1, variable2, variable3] obs filters: # Filter names are listed in # ufo/src/ufo/instantiateObsFilterFactory.h - filter: Bounds Check # test min/max value with all variables filter variables: - name: variable1 - name: variable2 - name: variable3 minvalue: 14.0 maxvalue: 19.0 # Compare variables with minvalue/maxvalue # variable1@ObsValue = 10, 11, 12, 13, 14, 15, 16, 17, 18, 19 # variable2@ObsValue = 10, 12, 14, 16, 18, 20, 22, 24, 26, 28 # variable3@ObsValue = 25, 24, 23, 22, 21, 20, 19, 18, 17, 16 # number of data points passing the filter passedBenchmark: 13 The test will pass when the number of data points that pass the filter is equal to :code:`passedBenchmark` value. The developer of the test is responsible for finding the correct :code:`passedBenchmark` value. You can define multiple mini-tests for your filter in this YAML configuration file. Now add a new test to filter out data points with ObsValues less than 15.0 and greater 20.0 only for variable2 and variable3 using your new :code:`Bounds Check` filter. Notice that all data points in variable1 will pass because variable1 is not specified in this test. You can copy :code:`obs filters` section from the previous test and modify it. Or you can simply use this template below to add this test. .. code-block:: yaml observations: - obs space: name: test data obsdatain: obsfile: Data/ufo/testinput_tier_1/filters_testdata.nc4 simulated variables: [variable2, variable3] obs filters: # Filter names are listed in # ufo/src/ufo/instantiateObsFilterFactory.h - filter: filter variables: - name: - name: minvalue: maxvalue: passedBenchmark: Step 4: Register your test to CMakeLists.txt -------------------------------------------- Now you need to register your new test to CMake by adding it to :code:`ufo/test/CMakeLists.txt`. First, add your YAML configure file to :code:`ufo_test_input` list. Next, under :code:`Test UFO ObsFilters (generic)` section add your test using :code:`ecbuild_add_test` command. .. code:: cmake ecbuild_add_test( TARGET test_ufo_qc_gen_practical_boundscheck COMMAND ${CMAKE_BINARY_DIR}/bin/test_ObsFilters.x ARGS "testinput/qc_practical_boundscheck.yaml" ENVIRONMENT OOPS_TRAPFPE=1 DEPENDS test_ObsFilters.x TEST_DEPENDS ufo_get_ufo_test_data ) Step 5: Run your new test ------------------------- Now you are ready to test your filter! Don't forget to rebuild UFO first. To rebuild UFO with the new changes you need to enter :code:`/ufo` and simply run the command :code:`make -j4`. Next you can list all the UFO test using :code:`ctest -N`. Can you find your new test in the list? Now run your test using: .. code:: bash ctest -R name_of_your_test Did your test pass? When writing a new test, it is always a good idea to also test failure conditions. Modify your YAML configuration file to make your test fail. You do not need to rebuild the bundle if you are only making changes to the YAML files. You can simply rerun your test after modifying the YAML file. Run your test in verbose mode to see the detailed output. .. code:: bash ctest -VV -R name_of_your_test Did your test fail as expected? Don't forget to change your YAML file back to the passing condition. You can add more tests to your YAML configuration file to make your new filter robust. .. _JEDI documentation: https://jointcenterforsatellitedataassimilation-jedi-docs.readthedocs-hosted.com/en/latest/inside/testing/unit_testing.html