Tuesday B: Using UFO for bias correction and QC =============================================== 1. Introduction --------------- This practical session introduces the JEDI Unified Forward Operator (UFO) code and will teach you how to configure and run various bias correction and data quality control steps. The tutorial will begin with atmospheric motion vectors derived from infrared satellite data tracking of cloud motions. We will show a series of QC filter steps with these data. Then, we will show an example of bias correction of microwave satellite radiance data from SSMIS. Finally we will demonstrate quality control of infrared SEVIRI satellite data. This activity assumes that you have successfully completed Monday's activities. You should have a working build of the fv3-jedi bundle. You also should have access to the same academy node as before, either using the JupyterLab environment (recommended) or SSH. Access your AWS instance and enter the Singularity container ------------------------------------------------------------ Connect to your assigned compute node. You will use the same method as previous exercises. You already have the singularity container that contains the JEDI dependencies. Enter the container using: .. code:: bash cd ~/ singularity shell -e jedi-gnu-openmpi-dev_latest.sif Once in the container be sure also to remove limits the stack memory to prevent spurious failures as noted in yesterday's `introductory exercise `__. .. code:: bash ulimit -s unlimited ulimit -v unlimited 2. Review of YAML structure --------------------------- By now, you should have seen and experienced the structure of YAML files, so please refer to prior exercises of the Academy if editing of these files is not yet clear. See `this link `__ for more examples. 3. Download and explore the test data ------------------------------------- During the previous tutorial (#3), sample data were downloaded from our AWS data repository. So rather than re-download, we can explore the data again as a reminder. .. code:: bash cd ~/jedi/tutorials/tutorial_obs_data The ``tutorial_obs_data`` directory contains excerpts of data generated from previous HofX model runs. Our radiance data are from a November 1, 2020, 12Z model run, and our conventional data are from December 15, 2020, 00Z. There are five subdirectories here, ``crtm``, ``geoval``, ``obs``, ``aux_files``, and ``answers``. - The ``obs`` subdirectory contains radiance observations from various instruments, including satellite platforms of AMSU-A, ATMS and SSMIS. There is also a variety of non-radiance data that we refer to as "conventional data," including aircraft, surface observations, radiosondes, and wind data derived from satellite-detected cloud motion. These observation files are stored in IODA's internal file format. Observation files can range in size from a few kilobytes to many megabytes. Some files store only a few observations, while others may contain millions. - The ``geoval`` directory contains model state information that has been interpolated to the observation locations. GeoVaLs are "Geophysical Values at Locations." In an ordinary JEDI run, we generate our own GeoVaLs in memory by consulting the model, but to save time in this practical we prepopulate our data from a previous invocation of JEDI. - The ``crtm`` directory contains CRTM coefficient data used by the radiative transfer model. - The ``aux_files`` directory contains auxilliary files for satellite biases and lapse rate information. - The ``answers`` directory contains "hints" in case you get stuck when writing your YAML files. Our tutorial begins with the atmospheric motion vector (AMV) data, which are created by tracking cloud motions through time using satellites. These are frequently called "SatWinds" while another type of satellite-derived wind motion data uses a scatterometer instrument for detecting ocean wave movement are frequently called "ScatWinds." The Satwind observation file is ``obs/satwinds_obs_2020121500_m.nc`` and contains over 50K data points in a 6-h time window. This file is reduced from the full volume of global data by a factor of about 50. The corresponding GeoVaLs file is: ``geoval/satwinds_geoval_2020121500_m.nc``. In a later step of this tutorial, we will be using satellite radiance data in the microwave and infrared regions. The Special Sensor Microwave Imager (SSM/I) and the Special Sensor Microwave Imager Sounder (SSMIS) are satellite passive microwave radiometers. For more complete details, see `this SSMI link `__. The corresponding satellite observation and GeoVaLs files are .. code:: bash obs/ssmis_f17_obs_2020110112.nc4 obs/seviri_m11_obs_2020110112.nc4 geoval/ssmis_f17_geoval_2020110112.nc4 geoval/severi_m11_geoval_2020110112.nc4 4. Quality control of SatWinds data. ------------------------------------ In this step, we will write a YAML file to run a set of filters that decide which data are rejected and which are retained in the data assimilation. In general SatWinds data are already relatively high quality, because it is a "data product" derived from visible and infrared satellite detection of moving clouds. The producers, such as NOAA-NESDIS, Japan Meteorological Agency, and Europe's EU-MET have already applied many quality control procedures in contrast to some other measurements such as surface weather observations. Regardless, we will demonstrate some extra precautions needed to ensure the DA system does not try to number-crunch potentially bad data. Since we would like to avoid modifying our testing data, first create a new directory for our experiment. .. code:: bash mkdir -p ~/tutorial_4_experiments cd ~/tutorial_4_experiments Now, create a new YAML file, and name it ``satwinds_gfs_qc.yaml``. Insert the text block below into the new YAML file. In a YAML file, indentation is important, so please ensure that your file looks like this example. Also please ensure that your indents use spaces instead of tabs. If you are using the ``vi`` editor, it may be helpful to open the editor and immediately type ``:set paste`` so the indentation shown below should be kept the same. .. code:: yaml window begin: 2020-12-14T20:30:00Z window end: 2020-12-15T03:30:00Z observations: - obs space: name: satwinds_QC obsdatain: obsfile: /home/ubuntu/jedi/tutorials/tutorial_obs_data/obs/satwind_obs_2020121500_m.nc obsdataout: obsfile: /home/ubuntu/tutorial_4_experiments/satwind_obs_2020121500_m_out.nc simulated variables: [eastward_wind, northward_wind] geovals: filename: /home/ubuntu/jedi/tutorials/tutorial_obs_data/geoval/satwind_geoval_2020121500_m.nc obs filters: # # Before filtering any observations, there are 5247 missing U,V wind components out of the total 51007 obs, # passedBenchmark: 91520 = 51007 x 2 (wind components) - 5247 missing of each component. # # Observation Range Sanity Check: either wind component or velocity exceeds 135 m/s - filter: Bounds Check filter variables: - name: eastward_wind - name: northward_wind minvalue: -135 maxvalue: 135 action: name: reject - filter: Bounds Check filter variables: - name: eastward_wind - name: northward_wind test variables: - name: Velocity@ObsFunction maxvalue: 135.0 action: name: reject # # No observations exceed this threshold, but it is good as an absolute sanity check. # # Reject winds that are within 100 hPa of the surface. - filter: Difference Check filter variables: - name: eastward_wind - name: northward_wind reference: surface_pressure@GeoVaLs value: air_pressure@MetaData maxvalue: -10000 # # The filter above rejects 4335 observed SatWinds. # # Reject winds that could be in the stratosphere (no clouds to track motion). - filter: Difference Check filter variables: - name: eastward_wind - name: northward_wind reference: TropopauseEstimate@ObsFunction value: air_pressure@MetaData minvalue: -3000 # 30 hPa above tropopause level, negative p-diff action: name: reject # # The filter above rejects only 21 observed SatWinds. # # Reject when (O-B) difference of wind direction is more than 50 degrees, but only winds > 3 m/s. - filter: Bounds Check filter variables: - name: eastward_wind - name: northward_wind test variables: - name: WindDirAngleDiff@ObsFunction options: minimum_uv: 3.0 test_hofx: GsiHofX maxvalue: 50.0 action: name: reject # # The filter above rejects 3054 observed SatWinds. # # Perform special log-normal vector difference (LNVD), reject if larger than 3. - filter: Bounds Check filter variables: - name: eastward_wind - name: northward_wind test variables: - name: SatWindsLNVDCheck@ObsFunction options: testHofX: GsiHofX maxvalue: 3 action: name: reject # # The filter above does not reject any of the observations. # # EUMET (242) and JMA (243) vis, reject when pressure less than 700 mb. - filter: Bounds Check filter variables: - name: eastward_wind - name: northward_wind test variables: - name: air_pressure@MetaData minvalue: 70000 where: - variable: name: eastward_wind@ObsType is_in: 242, 243 action: name: reject # # The filter above reject 237 observations. # # When running test_ObsFilters.x, We must tell the application how many observations # remain after running all filters in each obs space. passedBenchmark: 76458 The different keys and groupings in the YAML file have meaning. - The first two lines, ``window begin`` and ``window end``, tell IODA the bounds of your assimilation window. All observations **prior to and including** the begin time or **after** the end time are dropped, i.e., (begin, end]. - The ``observations`` line and subsequent ``obs space`` start a new block in which input and output files are named. In contrast to the previous tutorial, we are not running an obs operator, HofX. Instead we will utilize the existing GeoVaLs file in which vertical columns of model data at observation points are already present. The advantage here is that we can develop UFO codes for filtering and QC of data in development mode to test various QC steps without the cost overhead of repeatedly running the HofX application. - The ``obs filters`` section contains a lengthy set of filter steps, some are quite simple like ``Bounds Check`` and ``Difference Check``, while a few require very specialized code called ``ObsFunctions`` to compute additional variables needed to test data quality. One example is the ``Difference Check`` to reject any SatWinds data that could possibly be in the stratosphere. This requires the special ObsFunction called ``TropopauseEstimate`` to find the approximate pressure of the tropopause in order to reject winds that might be more than a specified pressure lower (height higher) than the tropopause pressure (height). The filter found last in the YAML shows a powerful example of a ``where`` clause in which we are specifically targeting SatWinds from JMA or EU-Met to reject any data higher in the atmosphere than 700 hPa through the use of ``ObsType`` which is a 3-digit value associated with each instrument type that NCEP-EMC uses to identify many forms of data. It must be noted that the order of filters in the YAML does not matter. For parallel computing reasons, the filters listed are not sequential and currently one filter does not depend on another. However, since a chain of filters can sometimes be very helpful, a future release of JEDI-UFO will include ability for users to list a dependency chain within the YAML. 6. Run the test application --------------------------- The test application is named ``test_ObsFilters.x``. It takes one command-line argument: the path to your YAML file. You could run the application directly, but you are processing many SatWinds observations. These can be parallelized by running within an MPI environment. You can execute the program by typing this: .. code:: bash mpiexec -n 4 ~/jedi/build/bin/test_ObsFilters.x ~/tutorial_4_experiments/satwinds_gfs_qc.yaml On the console you will notice a large amount of output. Eventually, the application should complete and return success, which is displayed on the second to final line with ``status = 0`` displayed. If any errors are indicated that would be a status not equal zero, please ask for help to see what went wrong. Usually, there is a bad file path or a typo in the YAML. You may need to scroll the output above many diagnostic print statements, but you can find out how many observations were either missing in the input data or filtered out by this YAML file by different filter groups. Try to locate the lines shown here: .. code:: bash QC satwinds_QC eastward_wind: 5247 missing values. QC satwinds_QC eastward_wind: 3276 out of bounds. QC satwinds_QC eastward_wind: 4255 rejected by difference check. QC satwinds_QC eastward_wind: 38229 passed out of 51007 observations. QC satwinds_QC northward_wind: 5247 missing values. QC satwinds_QC northward_wind: 3276 out of bounds. QC satwinds_QC northward_wind: 4255 rejected by difference check. QC satwinds_QC northward_wind: 38229 passed out of 51007 observations. Wind data are a bit unique compared to other observations, because you see a set of print statement for each wind component individually, which is why they look like duplicate prints. The prints are also aggregated for each filter type such as ``Bounds Check`` which was used 5 times in the YAML. Review the comments in the YAML below each filter to see the exact number of obs that would be eliminated if you had run that filter singularly in the YAML file. Since we are running the ``test_ObsFilters.x`` application, we must supply the application with the final number of expected observations that successfully pass through the QC steps. So the last line of the YAML contains the key ``passedBenchmark`` to be considered a successful test. When running the JEDI software in regular mode of data assimilation, we will not know in advance the number of observations expected to pass, but this application would be substituted by another application in the regular execution of JEDI. In our example dataset, roughly 25% of the starting observations are filtered by this QC yaml file. 7. Plotting the results ----------------------- We can plot the observed wind components (eastward\_wind, northward\_wind) by running a python plotting script once for each variable. In addition, we can plot the results of our QC filter steps using the JEDI variable ``EffectiveQC`` which is a series of values to indicate which filter eliminated any observations. The source code file: ``ufo/src/ufo/filters/QCflags.h`` contains the integer value mapping of the filters shown in the image here: |QCflags in UFO| Execute the plotting code for east/north winds and EffectiveQC using these commands: .. code:: bash python ~/jedi/tutorials/tutorial_obs_data/script/plot_from_iodav2_hofx.py --window_begin 2020121421 --nprocs 4 \ --hofxfiles ~/tutorial_4_experiments/satwind_obs_2020121500_m_out_NPROC.nc \ --colmin -45 --colmax 45 --variable ObsValue/eastward_wind python ~/jedi/tutorials/tutorial_obs_data/script/plot_from_iodav2_hofx.py --window_begin 2020121421 --nprocs 4 \ --hofxfiles ~/tutorial_4_experiments/satwind_obs_2020121500_m_out_NPROC.nc \ --colmin -45 --colmax 45 --variable ObsValue/northward_wind python ~/jedi/tutorials/tutorial_obs_data/script/plot_from_iodav2_hofx.py --window_begin 2020121421 --nprocs 4 \ --hofxfiles ~/tutorial_4_experiments/satwind_obs_2020121500_m_out_NPROC.nc \ --colmin -6 --colmax 12 --variable EffectiveQC/eastward_wind The graphics should look like the following example of eastward\_wind observed values and the corresponding EffectiveQC |Observed east wind velocity| and |Wind Effective QC after UFO filters| The colorbars on the wind components are set to match between +-45 m/s for both wind components using the ``colmin`` and ``colmax`` command-line options for the observed wind values; otherwise the default plotting behavior assigns the colorbars based on the mean and standard deviation of the data. For the ``EffectiveQC`` plots, you can ignore the units label, because the color bar represents the integer of QC-flags shown above. The dots in blue shade all passed the QC, while the dots in green and yellow shades indicate QC flags that reject the data. 8. Create a YAML file to run the bias correction step for SSMIS satellite data. ------------------------------------------------------------------------------- We are now going to shift datasets entirely to show how UFO bias correction works with satellite radiance data from SSMIS. We are going to write the YAML that specifies a number of **predictors** that modify the brightness temperature data in various channels to account for biases. Examples of the predictors are the sensor scan position and whether the satellite is in **ascending** or **descending** pass. There is a lecture tomorrow morning that clearly shows that some satellite channels are affected in systematic ways that can easily be adjusted by correcting for the bias. The changes you will make in this practical will illustrate how the data appear before and after the bias correction. We should be certain that we are in the appropriate directory, so please execute: .. code:: bash cd ~/tutorial_4_experiments Create a new YAML file, and name it ``ssmis_gfs_HofX_bc.yaml``. Insert this text into the new YAML file: .. code:: yaml window begin: 2020-11-01T09:00:00Z window end: 2020-11-01T15:00:00Z observations: - obs operator: name: CRTM Absorbers: [H2O,O3,CO2] obs options: Sensor_ID: &Sensor_ID ssmis_f17 EndianType: little_endian CoefficientPath: /home/ubuntu/jedi/tutorials/tutorial_obs_data/crtm/ obs space: name: ssmis_f17 obsdatain: obsfile: /home/ubuntu/jedi/tutorials/tutorial_obs_data/obs/ssmis_f17_obs_2020110112.nc4 obsdataout: obsfile: /home/ubuntu/tutorial_4_experiments/ssmis_f17_hofx_2020110112.nc4 simulated variables: [brightness_temperature] channels: &channels 1-24 geovals: filename: /home/ubuntu/jedi/tutorials/tutorial_obs_data/geoval/ssmis_f17_geoval_2020110112.nc4 vector ref: GsiHofXBc tolerance: 1.e-5 obs bias: input file: /home/ubuntu/jedi/tutorials/tutorial_obs_data/aux_files/satbias_ssmis_f17.nc4 variational bc: predictors: - name: constant - name: cloud_liquid_water options: satellite: SSMIS ch19h: 12 ch19v: 13 ch22v: 14 ch37h: 15 ch37v: 16 ch91v: 17 ch91h: 18 - name: cosine_of_latitude_times_orbit_node - name: sine_of_latitude - name: lapse_rate options: order: 2 tlapse: &ssmis_f17_tlap /home/ubuntu/jedi/tutorials/tutorial_obs_data/aux_files/ssmis_f17_tlapmean.txt - name: lapse_rate options: tlapse: *ssmis_f17_tlap - name: emissivity - name: scan_angle options: var_name: scan_position order: 4 - name: scan_angle options: var_name: scan_position order: 3 - name: scan_angle options: var_name: scan_position order: 2 - name: scan_angle options: var_name: scan_position In a YAML file, indentation is important, so please ensure that your file looks like this example. Also please ensure that your indents use spaces instead of tabs. If you are using the ``vi`` editor, it may be helpful | to open the editor and immediately type ``:set paste`` so the indentation shown above should be kept the same. The different keys and groupings in the YAML file have meaning. - The first two lines, ``window begin`` and ``window end``, tell IODA the bounds of your assimilation window. - The ``observations`` line denotes that we are specifying an observation operator for the application to run. For this first example, we are only attempting to run a single observation operator, CRTM, as we used in a prior UFO practical. When CRTM performs its calculations, it will assume that the atmosphere has three absorbing gases, water vapor, ozone, and carbon dioxide. Water and ice clouds may both exist. - The ``obs options`` section provides additional information to properly run CRTM. Each instrument needs various ancillary data files that contain information about the sensor's channels, polarizations, spectral response funcitons, and so on. - The ``obs space`` section describes the input data that we are using with the operator. The observation data file is specified using the ``obsfile`` key in the ``obsdatain`` section. The results of the operator can optionally be written to a file. This occurs when an ``obsdataout`` section appears in the YAML. The syntax of ``obsdatain`` and ``obsdataout`` are identical. - The ``simulated variables`` and ``channels`` sections tell UFO that you want to simulate brightness temperatures for instrument channels 1-24. - The ``geovals`` section provides interpolated model values at the observed locations. This is a "shortcut" for the JEDI system to avoid reading full model backgrounds, and this is very useful when developing a new operator or when incrementally implementing bias correction and quality control filters. For the purposes of this practical exercise (i.e. to keep runtimes short), we provide geovals files. - The next two lines (``vector ref`` and ``tolerance``) allow us to specify a final "check" in our test application to verify that our simulated results match those of another system. In this case, we are matching against GSI's H(x) operator and want to ensure that our calculations match theirs. If the reference check is not specified, then no check is performed. - The ``obs bias`` section contains a variety of bias predictor names (to invoke functions) that are needed to calculate the bias correction to the microwave brightness temperature data. References Dee, D.P., 2004: Variational bias correction of radiance data in the ECMWF system. In Proc. ECMWF Workshop on Assimilation of High Spectral Resolution Sounders in NWP. ECMWF, Reading, UK, 97-112. Zhu, Y., Derber, J., Collard, A., Dee, D., Treadon, R., Gayno, G., Jung, J.A., 2014. Enhanced radiance bias correction in the National Centers for Environmental Prediction’s Gridpoint Statistical Interpolation data assimilation system. Quarterly Journal of the Royal Meteorological Society 140, 1479–1492. https://doi.org/10.1002/qj.2233. 9. Run the test application --------------------------- In a prior step, we only needed to run QC filters and not use any ObsOperator because we used pre-calculated GeoVaLs files. In this step, we will still run CRTM to simulate the satellite radiances, so we will use test application is named ``test_ObsOperator.x``. It takes one command- line argument: the path to your YAML file. You could run the application directly, but you are processing many SSMIS observations. These can be parallelized by running within an MPI environment. You can execute the program by typing this: .. code:: bash mpiexec -n 4 ~/jedi/build/bin/test_ObsOperator.x ~/tutorial_4_experiments/ssmis_gfs_HofX_bc.yaml On the console you will notice a large amount of output. Eventually, the application should complete and return success, which is displayed on the second to final line with ``status = 0`` displayed. If any errors are indicated that would be a status not equal zero, please ask for help to see what went wrong. Usually, there is a bad file path or a typo in the YAML. 10. Check the results --------------------- You may have noticed from the YAML defined and used in the previous sections that there was an ``obsdataout`` section in it. That section specifies an ``obsfile`` template name to save the output files of the run. So, let's change the current directory to the one where those files are supposed to be saved and check them. On the console, you can change directory and list the files there: You are expected to see a list of files similar to the following: .. code:: bash ssmis_gfs_HofX_bc.yaml ssmis_f17_hofx_2020110112_0000.nc4 ssmis_f17_hofx_2020110112_0001.nc4 ssmis_f17_hofx_2020110112_0002.nc4 ssmis_f17_hofx_2020110112_0003.nc4 If you recall, the ``obsfile`` template was defined as ``/home/ubuntu/tutorial_4_experiments/ssmis_f17_hofx_2020110112.nc4`` for this ``SSMIS`` case. The name of the files that you are seeing in your console follow that template, but you have four files following that template with an underscore and a set of numbers appended to its name (e.g., ``_0000``). This is because you've run your application using four processor elements and the program distributes the input file among these four processor elements. Let's generate a figure showing some results from our run. To do this, we need to run the following command: .. code:: bash python ~/jedi/tutorials/tutorial_obs_data/script/plot_from_iodav2_hofx.py \ --window_begin 2020110109 --nprocs 4 \ --hofxfiles /home/ubuntu/tutorial_4_experiments/ssmis_f17_hofx_2020110112_NPROC.nc4 \ --omb True \ --variable hofx/brightness_temperature_24 --colmin -2 --colmax 2 The above command will invoke the plotting script passing a list of arguments, described as below: - ``--window_begin 2020110109``: the timestamp of the beginning of the window (following the YYYYMMDDHH template) - ``--nprocs 4``: number of processor elements used to run the application (same number of HofX files) - ``--hofxfiles ../ssmis_f17_hofx_2020110112_NPROC.nc4``: the template for HofX file names (with ``_NPROC`` appended to it) - ``--omb True``: we wish to plot the Obs minus background (O - B) data - ``--variable hofx/brightness_temperature_24``: the variable to be plotted (in this case the ``hofx`` of ``brightness_temperature`` for channel 24) The command above is expected to generate a figure (``brightness_temperature_24_omb.png``) showing the spatial distribution of simulated brightness temperature observations minus background (from model using HofX) from ``SSMIS`` for channel 24 using all bias predictors to bias correct the observations. |SSMIS channel 24 O-B with bias correction| In the next section, we are going to remove all the sections in the YAML related to scan position to show the impact of skipping the bias correction for scan geometry. First, let's be sure to save the first graphic we created in the example so far, so execute the follow command: .. code:: bash /bin/mv brightness_temperature_24_omb-hofx.png brightness_temperature_24_omb_step10.png 11. Remove the orbit node (ascending/descending) bias predictors ---------------------------------------------------------------- Edit the YAML file that we created in step#4 (``ssmis_gfs_HofX_bc.yaml``) and **remove** entirely the two lines shown below. This will eliminate the bias predictors related to whether the satellite is passing in ascending or descending modes, which have a known bias signal. .. code:: yaml - name: cosine_of_latitude_times_orbit_node - name: sine_of_latitude Now we are going to re-run the same HofX and bias-correction HofX forward operator, but but skipping the two bias predictors. Recall the same execution command is this: .. code:: bash mpiexec -n 4 ~/jedi/build/bin/test_ObsOperator.x ~/tutorial_4_experiments/ssmis_gfs_HofX_bc.yaml Due to removing two bias predictors, the number of observations that pass through the step changes, and we did not edit the number so an error message with non-zero status will occur. The step ran fine, so we will just ignore the error message. To visualize the result, we can re-create the same plot as the last time, but using the newly generated output (that overwrites the previous files) and shows what happens when we remove the bias predictor related to orbit node. Re-run the plotting code again using: .. code:: bash python ~/jedi/tutorials/tutorial_obs_data/script/plot_from_iodav2_hofx.py \ --window_begin 2020110109 --nprocs 4 \ --hofxfiles /home/ubuntu/tutorial_4_experiments/ssmis_f17_hofx_2020110112_NPROC.nc4 \ --omb True \ --variable hofx/brightness_temperature_24 --colmin -2 --colmax 2 Now you can visually see the obs - background difference with and without accounting for bias correction of the satellite orbit node by comparing the two figures: ``brightness_temperature_24_omb_step10.png`` and ``brightness_temperature_24_omb.png``. |SSMIS channel 24 O-B without orbit node bias correction| 12. Exercise: Add quality control for SEVIRI data and see its effect -------------------------------------------------------------------- For one final step, we will add quality control for SEVIRI data to eliminate any observations with local zenith angle greater than 50 degree. This is done due to the large slant angle through the atmosphere; only smaller zeninth angles are appropriate to ingest. Create a new YAML file to quality control the SEVIRIR satellite data, using the filename ``seviri_gfs_HofX_qc.yaml``. Insert the following block of yaml into the new file. .. code:: yaml window begin: 2020-11-01T09:00:00Z window end: 2020-11-01T15:00:00Z observations: - obs operator: name: CRTM Absorbers: [H2O,O3,CO2] obs options: Sensor_ID: &Sensor_ID seviri_m11 EndianType: little_endian CoefficientPath: /home/ubuntu/jedi/tutorials/tutorial_obs_data/crtm/ obs space: name: seviri_m11_qc obsdatain: obsfile: /home/ubuntu/jedi/tutorials/tutorial_obs_data/obs/seviri_m11_obs_2020110112.nc4 obsdataout: obsfile: /home/ubuntu/tutorial_4_experiments/seviri_m11_obs_2020110112_out.nc simulated variables: [brightness_temperature] channels: &all_channels 4-11 geovals: filename: /home/ubuntu/jedi/tutorials/tutorial_obs_data/geoval/seviri_m11_geoval_2020110112.nc4 obs filters: # # QC for local zenith angle ; only use data with less than 50 degree sensor zenith angle due to slant path potential problems. - filter: Domain Check filter variables: - name: brightness_temperature channels: *all_channels where: - variable: name: sensor_zenith_angle@MetaData maxvalue: 50.0 passedBenchmark: 16444 Now we can run the QC steps using the ``test_ObsFilters.x`` application like we did for QC of Satwinds back in an earlier step using the following command line: .. code:: bash mpiexec -n 4 ~/jedi/build/bin/test_ObsFilters.x ~/tutorial_4_experiments/seviri_gfs_HofX_qc.yaml This should result in four new output files (one file per core) for the SEVIRI output similar to a prior step for SSMIS output. Also, similar to the SatWinds QC example earlier, we can create a plot of the EffectiveQC performed by UFO using the command line: .. code:: bash python ~/jedi/tutorials/tutorial_obs_data/script/plot_from_iodav2_hofx.py --window_begin 2020110109 --nprocs 4 \ --hofxfiles ~/tutorial_4_experiments/seviri_m11_obs_2020110112_out_NPROC.nc \ --colmin 0 --colmax 19 --variable EffectiveQC/brightness_temperature_5 And, reaching the finish line for this tutorial, the result of the plot should look like the following image: |SEVIRI channel 5 EffectiveQC| .. |QCflags in UFO| image:: ./images/UFO_qcflag_list.png .. |Observed east wind velocity| image:: ./images/eastward_wind_ObsValue.png .. |Wind Effective QC after UFO filters| image:: ./images/eastward_wind_EffectiveQC.png .. |SSMIS channel 24 O-B with bias correction| image:: ./images/radiance/ssmis_BT24_OMB_withBC.png .. |SSMIS channel 24 O-B without orbit node bias correction| image:: ./images/radiance/ssmis_BT24_OMB_withoutBC.png .. |SEVIRI channel 5 EffectiveQC| image:: ./images/radiance/seviri_BT5_EffectiveQC.png