Thursday A: Adding a filter test to JEDI

In this session you will practice adding a test to a new QC filter you developed on Tuesday(b).

Requirements:

• A terminal session connected to your AWS node. You can use either SSH or the web-based JupyterLab interface.
• Have successfully added PracticalBoundsCheck filter to UFO filters by following Tuesday(b) activity instruction.
• Have built and compiled ufo bundle

Introduction:

In the activity session b on Tuesday, you practiced adding a new filter to UFO. Today we will develop a test for this filter to ensure it works properly.

Testing in JEDI

Each JEDI bundle has it’s own suite of tests. After building and compiling the bundle, you can run the tests using ctest.

cd <build-directory>/ufo
ctest

This will run all tests in ufo. When the tests are complete, ctest will print out a summary, highlighting which tests, if any, failed. To run a single test, you can use -R option, for example:

ctest -R ufo_coding_norms

The output from these tests will be printed to the screen and written to the file LastTest.log in the directory <build-directory>/Testing/Temporary. In the same directory LastTestsFailed.log lists the last tests that failed. You can run ctests with the verbose option to get more information which can be helpful for debugging.

ctest -V -R test_ufo_geovals

and for extra verbose:

ctest -VV -R test_ufo_geovals

Another way to get more information is to set one or both of these environment variables before you run ctest:

export OOPS_DEBUG=1
export OOPS_TRACE=1

ctest also has an option to only re-run the tests that failed last time:

ctest --rerun-failed

Please refer to (JEDI documentation) for more information on JEDI test suite.

Keep in mind that when you add a new feature to the JEDI repository you need to write a test for your code. This way you can ensure that your code is working properly and it will help us review and merge your code quicker.

Adding a test for your filter:

In this exercise, we will write and add a test for PracticalBoundsCheck filter you added in the practical session on Tuesday (b).

A quick reminder about PracticalBoundsCheck filter :

In ufo/src/ufo/filters you have added PracticalBoundsCheck.h and PracticalBoundsCheck.cc. This filter checks to see if your data is within a specified range or not. Take a look that the function PracticalBoundsCheck::applyFilter in PracticalBoundsCheck.cc for more details:

void PracticalBoundsCheck::applyFilter(const std::vector<bool> & apply,
                                  const Variables & filtervars,
                                  std::vector<std::vector<bool>> & flagged) const {
  const float missing = util::missingValue(missing);
  ufo::Variables testvars;

  // This filter is applied based on var@ObsValue values.
  testvars += ufo::Variables(filtervars, "ObsValue");

  // Read minvalue and maxvalue from YAML configuration
  const float vmin = config_.getFloat("minvalue", missing);
  const float vmax = config_.getFloat("maxvalue", missing);

  //  Sanity checks
  if (filtervars.nvars() == 0) {
    oops::Log::error() << "No variables will be filtered out in filter "
                       << config_ << std::endl;
    ABORT("No variables specified to be filtered out in filter");
  }

// Loop over all variables to filter
    for (size_t jv = 0; jv < testvars.nvars(); ++jv) {
      //  get test data for this variable
      std::vector<float> testdata;
      data_.get(testvars.variable(jv), testdata);
      //  apply the filter
      //  filter out var@ObsValue >vmax and <vmin
      for (size_t jobs = 0; jobs < obsdb_.nlocs(); ++jobs) {
        if (apply[jobs]) {
          ASSERT(testdata[jobs] != missing);
          if (vmin != missing && testdata[jobs] < vmin) flagged[jv][jobs] = true;
          if (vmax != missing && testdata[jobs] > vmax) flagged[jv][jobs] = true;
        }
      }
    }
}

Add YAML Configuration File

We will use filter_testdata.nc4, a simplified IODA format file, for testing our new filter. Take a look at this dataset by using ncdump command.

cd <build-directory>/ufo/test/Data
ncdump filter_testdata.nc4 | less

To test your filter you need to first add YAML configuration file in ufo/test/testinput. YAML configuration files used in testing filters are usually named qc_*.yaml. create a new YAML file called qc_practical_boundscheck.yaml. Copy and paste this to your YAML file.

window_begin: 2018-04-14T21:00:00Z
window_end: 2018-04-15T03:00:00Z

Observations:
 ObsTypes:
 - ObsSpace:
     name: test data
     #  Ioda file we want to apply the filter to
     ObsDataIn:
       obsfile: Data/filters_testdata.nc4
     simulate:
       variables: [variable1, variable2, variable3]
   ObsFilters:
   #  name of the filter. List of filter names are in
   #  ufo/src/ufo/instantiateObsFilterFactory.h
   - Filter: Bounds Check        # test min/max value with all variables
     #  variables you want to apply the filter to
     filter variables:
     - name: variable1
     - name: variable2
     - name: variable3
     minvalue: 14.0
     maxvalue: 19.0
   #  variable1@ObsValue = 10, 11, 12, 13, 14, 15, 16, 17, 18, 19
   #  variable2@ObsValue = 10, 12, 14, 16, 18, 20, 22, 24, 26, 28
   #  variable3@ObsValue = 25, 24, 23, 22, 21, 20, 19, 18, 17, 16
   #  how many data points will pass the filter?
   passedBenchmark: 13

The test will pass when the number of data points that pass the filter is equal to passedBenchmark value. The developer of the test is responsible for finding the correct passedBenchmark value.

You can define multiple tests for your filter in this YAML configuration file. Now add a new test to filter out data points with ObsValues less than 15.0 and greater 20.0 only for variable2 and variable3. Notice that all data points in variable1 will pass because variable1 is not specified in this test. you can copy ObsType: section from the previous test and modify it. Or you can use this template to add this test.

ObsTypes:
- ObsSpace:
    name: test data
    ObsDataIn:
      obsfile: Data/filters_testdata.nc4
    simulate:
      variables: [variable1, variable2, variable3]
  ObsFilters:
  - Filter:
    filter variables:
    - name:
    minvalue:
    maxvalue:
  passedBenchmark:

Register your test to CMakeLists.txt

Now you need to register your new test to CMake by adding it to ufo/test/CMakeLists.txt. First, add your YAML configure file to ufo_test_input list. Next, under Test UFO ObsFilters (generic) section add your test using ecbuild_add_test command.

ecbuild_add_test( TARGET  test_ufo_qc_gen_practical_boundscheck
                  COMMAND ${CMAKE_BINARY_DIR}/bin/test_ObsFilters.x
                  ARGS    "testinput/qc_practical_boundscheck.yaml"
                  ENVIRONMENT OOPS_TRAPFPE=1
                  DEPENDS test_ObsFilters.x )

Now you are ready to test your filter!

You need to rebuilt ufo first. In the build directory cd into ufo and list all the ufo test using ctest -N. Can you find your test in the list? Now run your test using

ctest -R name_of_your_test

Did your test pass? You can run your test with the verbose option on.

ctest -VV -R name_of_your_test

You can add more tests to your YAML configuration file to make your new filter more robust. You do not need to rebuild the bundle if you are only making changes to the YAML files.