Thursday B: Adding a filter test to JEDI¶

In this session you will practice adding a test to a new QC filter you developed on Wednesday(a).

Requirements¶

A terminal session connected to your AWS node. You can use either SSH or the web-based JupyterLab interface.
Have successfully added PracticalBoundsCheck filter to UFO filters by following Wednesday(a) activity instruction.
Have built and compiled fv3-bundle

Introduction¶

In the activity session a on Wednesday, you practiced adding a new filter to UFO. Today we will develop a test for this filter to ensure it works properly.

Testing in JEDI¶

Each JEDI bundle has it’s own suite of tests. After building and compiling the bundle, you can run the tests using ctest.

cd <build-directory>/ufo
ctest

Here <build-directory> is $HOME/jedi/build. This will run all tests in ufo. When the tests are complete, ctest will print out a summary, highlighting which tests, if any, failed. To run a single test, you can use -R followed by the test’s name, for example:

ctest -R ufo_coding_norms

The output from these tests will be printed to the screen and written to the file LastTest.log in the directory <build-directory>/Testing/Temporary or in this example $HOME/jedi/build/ufo/Testing/Temporary. In the same directory LastTestsFailed.log lists the last tests that failed. You can run ctests with the verbose option to get more information which can be helpful for debugging.

ctest -V -R test_ufo_geovals

and for extra verbose:

ctest -VV -R test_ufo_geovals

Another way to get more information is to set one or both of these environment variables before you run ctest:

export OOPS_DEBUG=1
export OOPS_TRACE=1

ctest also has an option to only re-run the tests that failed last time:

ctest --rerun-failed

Please refer to (JEDI documentation) for more information on JEDI test suite.

Keep in mind that when you add a new feature to the JEDI repository you need to write a test for your code. This way you ensure your code is working properly and it will help us review and merge your code quicker.

Step 1: Access Your AWS instance¶

Follow the same method as Monday to connect to your AWS compute node. Don’t forget to enter the singularity container using this command:

singularity shell -e jedi-gnu-openmpi-dev_latest.sif

Make sure you set:

ulimit -s unlimited
ulimit -v unlimited
export LOCAL_PATH_JEDI_TESTFILES=$HOME/jedi/test_data

Step 2: Review the new added filter¶

In this exercise, we will write and add a test for PracticalBoundsCheck filter you added in the practical session on Wednesday (a). We want to push the changes back to the GitHub repository so make sure you are in the correct branch ( feature/new_qc_filter_example ) on the ufo repository. Enter $HOME/jedi/fv3-bundle/ufo and check the branch name by using command:

git branch

Let’s review how we added PracticalBoundsCheck filter to the ufo repository yesterday. In ufo/src/ufo/filters you have added PracticalBoundsCheck.h and PracticalBoundsCheck.cc. This filter checks to see if your data is within a specified range or not. Take a look that the function PracticalBoundsCheck::applyFilter in PracticalBoundsCheck.cc for more details:

void PracticalBoundsCheck::applyFilter(const std::vector<bool> & apply,
                                  const Variables & filtervars,
                                  std::vector<std::vector<bool>> & flagged) const {
  const float missing = util::missingValue(missing);
  ufo::Variables testvars;

  // This filter is applied based on var@ObsValue values.
  testvars += ufo::Variables(filtervars, "ObsValue");

  // Read minvalue and maxvalue from YAML configuration
  const float vmin = config_.getFloat("minvalue", missing);
  const float vmax = config_.getFloat("maxvalue", missing);

  //  Sanity checks
  if (filtervars.nvars() == 0) {
    oops::Log::error() << "No variables will be filtered out in filter "
                       << config_ << std::endl;
    ABORT("No variables specified to be filtered out in filter");
  }

// Loop over all variables to filter
    for (size_t jv = 0; jv < testvars.nvars(); ++jv) {
      //  get test data for this variable
      std::vector<float> testdata;
      data_.get(testvars.variable(jv), testdata);
      //  apply the filter
      //  filter out var@ObsValue >vmax and <vmin
      for (size_t jobs = 0; jobs < obsdb_.nlocs(); ++jobs) {
        if (apply[jobs]) {
          ASSERT(testdata[jobs] != missing);
          if (vmin != missing && testdata[jobs] < vmin) flagged[jv][jobs] = true;
          if (vmax != missing && testdata[jobs] > vmax) flagged[jv][jobs] = true;
        }
      }
    }
}

Step 3: Add YAML configuration file for the new test¶

We will use filters_testdata.nc4, a simplified IODA format file, for testing our new filter. Take a look at this dataset by using ncdump command.

cd <build-directory>/ufo/test/Data/ufo/testinput_tier_1
ncdump filters_testdata.nc4 | less

Here <build-directory> is $HOME/jedi/build.

To test your filter you need to first add YAML configuration file in your source directory $HOME/jedi/fv3-bundle/ufo/test/testinput. In this directory, YAML configuration files with prefix qc_ are used for testing various filters in UFO. In this file, you can specify the details of how you want to test your filter. For example, the name of the file and list of variables in the file you want to apply the filter on.

Create a new YAML file in ufo/test/testinput called qc_practical_boundscheck.yaml. Copy and paste this to your YAML file.

window begin: 2018-01-01T00:00:00Z
window end: 2019-01-01T00:00:00Z

observations:
- obs space:
    name: test data
    obsdatain:
      obsfile: Data/ufo/testinput_tier_1/filters_testdata.nc4
    simulated variables: [variable1, variable2, variable3]
  obs filters:
  #  Filter names are listed in
  #  ufo/src/ufo/instantiateObsFilterFactory.h
  - filter: Practical Bounds Check        # test min/max value with all variables
    filter variables:
    - name: variable1
    - name: variable2
    - name: variable3
    minvalue: 14.0
    maxvalue: 19.0
#  Compare variables with minvalue/maxvalue
#  variable1@ObsValue = 10, 11, 12, 13, 14, 15, 16, 17, 18, 19
#  variable2@ObsValue = 10, 12, 14, 16, 18, 20, 22, 24, 26, 28
#  variable3@ObsValue = 25, 24, 23, 22, 21, 20, 19, 18, 17, 16
#  number of data points passing the filter
  passedBenchmark: 13

The test will pass when the number of data points that pass the filter is equal to passedBenchmark value. The developer of the test is responsible for finding the correct passedBenchmark value.

You can define multiple mini-tests for your filter in this YAML configuration file. Now add a new test to filter out data points with ObsValues less than 15.0 and greater 20.0 only for variable2 and variable3 using your new Practical Bounds Check filter. Notice that all data points in variable1 will pass because variable1 is not specified in this test. You can copy obs filters section from the previous test and modify it. Or you can simply use this template below to add this test.

observations:
- obs space:
    name: test data
    obsdatain:
      obsfile: Data/ufo/testinput_tier_1/filters_testdata.nc4
    simulated variables: [variable2, variable3]
  obs filters:
  #  Filter names are listed in
  #  ufo/src/ufo/instantiateObsFilterFactory.h
  - filter:
    filter variables:
    - name:
    - name:
    minvalue:
    maxvalue:
  passedBenchmark:

Step 4: Register your test to CMakeLists.txt¶

Now you need to register your new test to CMake by adding it to $HOME/jedi/fv3-bundle/ufo/test/CMakeLists.txt. First, add your YAML configure file to ufo_test_input list. Next, under Test UFO ObsFilters (generic) section add your test using ecbuild_add_test command.

ecbuild_add_test( TARGET  test_ufo_qc_gen_practical_boundscheck
                  COMMAND ${CMAKE_BINARY_DIR}/bin/test_ObsFilters.x
                  ARGS    "testinput/qc_practical_boundscheck.yaml"
                  ENVIRONMENT OOPS_TRAPFPE=1
                  DEPENDS test_ObsFilters.x
                  TEST_DEPENDS ufo_get_ufo_test_data )

Step 5: Run your new test¶

Now you are ready to test your filter! Don’t forget to rebuild UFO first. To rebuild UFO with the new changes you need to enter <build-directory>/ufo and simply run the command make -j4. Next you can list all the UFO test using ctest -N. Can you find your new test in the list? Now run your test using:

ctest -R name_of_your_test

Did your test pass? When writing a new test, it is always a good idea to also test failure conditions. Modify your YAML configuration file to make your test fail. You do not need to rebuild the bundle if you are only making changes to the YAML files. You can simply rerun your test after modifying the YAML file. Run your test in verbose mode to see the detailed output.

ctest -VV -R name_of_your_test

Did your test fail as expected? Don’t forget to change your YAML file back to the passing condition. You can add more tests to your YAML configuration file to make your new filter robust.

Thursday B: Adding a filter test to JEDI¶

Requirements¶

Introduction¶

Testing in JEDI¶

Step 1: Access Your AWS instance¶

Step 2: Review the new added filter¶

Step 3: Add YAML configuration file for the new test¶

Step 4: Register your test to CMakeLists.txt¶

Step 5: Run your new test¶

Table of Contents

Previous topic

Next topic

This Page