######################################################## Thursday A: Adding a filter test to JEDI ######################################################## In this session you will practice adding a test to a new QC filter you developed on Tuesday(b). Requirements: ---------------------- - A terminal session connected to your AWS node. You can use either SSH or the web-based JupyterLab interface. - Have successfully added :code:`PracticalBoundsCheck` filter to UFO filters by following Tuesday(b) activity instruction. - Have built and compiled ufo bundle Introduction: ------------- In the activity session b on Tuesday, you practiced adding a new filter to UFO. Today we will develop a test for this filter to ensure it works properly. Testing in JEDI ~~~~~~~~~~~~~~~ Each JEDI bundle has it's own suite of tests. After building and compiling the bundle, you can run the tests using :code:`ctest`. .. code:: bash cd /ufo ctest This will run all tests in ufo. When the tests are complete, ctest will print out a summary, highlighting which tests, if any, failed. To run a single test, you can use :code:`-R` option, for example: .. code:: bash ctest -R ufo_coding_norms The output from these tests will be printed to the screen and written to the file :code:`LastTest.log` in the directory :code:`/Testing/Temporary`. In the same directory :code:`LastTestsFailed.log` lists the last tests that failed. You can run ctests with the verbose option to get more information which can be helpful for debugging. .. code:: bash ctest -V -R test_ufo_geovals and for extra verbose: .. code:: bash ctest -VV -R test_ufo_geovals Another way to get more information is to set one or both of these environment variables before you run ctest: .. code:: bash export OOPS_DEBUG=1 export OOPS_TRACE=1 ctest also has an option to only re-run the tests that failed last time: .. code:: bash ctest --rerun-failed Please refer to (`JEDI documentation`_) for more information on JEDI test suite. Keep in mind that when you add a new feature to the JEDI repository you need to write a test for your code. This way you can ensure that your code is working properly and it will help us review and merge your code quicker. Adding a test for your filter: ------------------------------ In this exercise, we will write and add a test for :code:`PracticalBoundsCheck` filter you added in the practical session on Tuesday (b). A quick reminder about :code:`PracticalBoundsCheck` filter : In :code:`ufo/src/ufo/filters` you have added :code:`PracticalBoundsCheck.h` and :code:`PracticalBoundsCheck.cc`. This filter checks to see if your data is within a specified range or not. Take a look that the function :code:`PracticalBoundsCheck::applyFilter` in :code:`PracticalBoundsCheck.cc` for more details: .. code:: C++ void PracticalBoundsCheck::applyFilter(const std::vector & apply, const Variables & filtervars, std::vector> & flagged) const { const float missing = util::missingValue(missing); ufo::Variables testvars; // This filter is applied based on var@ObsValue values. testvars += ufo::Variables(filtervars, "ObsValue"); // Read minvalue and maxvalue from YAML configuration const float vmin = config_.getFloat("minvalue", missing); const float vmax = config_.getFloat("maxvalue", missing); // Sanity checks if (filtervars.nvars() == 0) { oops::Log::error() << "No variables will be filtered out in filter " << config_ << std::endl; ABORT("No variables specified to be filtered out in filter"); } // Loop over all variables to filter for (size_t jv = 0; jv < testvars.nvars(); ++jv) { // get test data for this variable std::vector testdata; data_.get(testvars.variable(jv), testdata); // apply the filter // filter out var@ObsValue >vmax and vmax) flagged[jv][jobs] = true; } } } } Add YAML Configuration File ~~~~~~~~~~~~~~~~~~~~~~~~~~~ We will use :code:`filter_testdata.nc4`, a simplified IODA format file, for testing our new filter. Take a look at this dataset by using :code:`ncdump` command. .. code:: bash cd /ufo/test/Data ncdump filter_testdata.nc4 | less To test your filter you need to first add YAML configuration file in :code:`ufo/test/testinput`. YAML configuration files used in testing filters are usually named :code:`qc_*.yaml`. create a new YAML file called :code:`qc_practical_boundscheck.yaml`. Copy and paste this to your YAML file. .. code:: yaml window_begin: 2018-04-14T21:00:00Z window_end: 2018-04-15T03:00:00Z Observations: ObsTypes: - ObsSpace: name: test data # Ioda file we want to apply the filter to ObsDataIn: obsfile: Data/filters_testdata.nc4 simulate: variables: [variable1, variable2, variable3] ObsFilters: # name of the filter. List of filter names are in # ufo/src/ufo/instantiateObsFilterFactory.h - Filter: Bounds Check # test min/max value with all variables # variables you want to apply the filter to filter variables: - name: variable1 - name: variable2 - name: variable3 minvalue: 14.0 maxvalue: 19.0 # variable1@ObsValue = 10, 11, 12, 13, 14, 15, 16, 17, 18, 19 # variable2@ObsValue = 10, 12, 14, 16, 18, 20, 22, 24, 26, 28 # variable3@ObsValue = 25, 24, 23, 22, 21, 20, 19, 18, 17, 16 # how many data points will pass the filter? passedBenchmark: 13 The test will pass when the number of data points that pass the filter is equal to :code:`passedBenchmark` value. The developer of the test is responsible for finding the correct :code:`passedBenchmark` value. You can define multiple tests for your filter in this YAML configuration file. Now add a new test to filter out data points with ObsValues less than 15.0 and greater 20.0 only for variable2 and variable3. Notice that all data points in variable1 will pass because variable1 is not specified in this test. you can copy :code:`ObsType:` section from the previous test and modify it. Or you can use this template to add this test. .. code:: yaml ObsTypes: - ObsSpace: name: test data ObsDataIn: obsfile: Data/filters_testdata.nc4 simulate: variables: [variable1, variable2, variable3] ObsFilters: - Filter: filter variables: - name: minvalue: maxvalue: passedBenchmark: Register your test to CMakeLists.txt ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Now you need to register your new test to CMake by adding it to :code:`ufo/test/CMakeLists.txt`. First, add your YAML configure file to :code:`ufo_test_input` list. Next, under :code:`Test UFO ObsFilters (generic)` section add your test using :code:`ecbuild_add_test` command. .. code:: cmake ecbuild_add_test( TARGET test_ufo_qc_gen_practical_boundscheck COMMAND ${CMAKE_BINARY_DIR}/bin/test_ObsFilters.x ARGS "testinput/qc_practical_boundscheck.yaml" ENVIRONMENT OOPS_TRAPFPE=1 DEPENDS test_ObsFilters.x ) Now you are ready to test your filter! You need to rebuilt ufo first. In the build directory cd into ufo and list all the ufo test using :code:`ctest -N`. Can you find your test in the list? Now run your test using .. code:: bash ctest -R name_of_your_test Did your test pass? You can run your test with the verbose option on. .. code:: bash ctest -VV -R name_of_your_test You can add more tests to your YAML configuration file to make your new filter more robust. You do not need to rebuild the bundle if you are only making changes to the YAML files. .. _JEDI documentation: https://jointcenterforsatellitedataassimilation-jedi-docs.readthedocs-hosted.com/en/latest/developer/building_and_testing/unit_testing.html