Monday B: observations operators / configuration¶
In this tutorial, we will build on the first steps taken this morning and learn more about the JEDI configurations and YAML files and how to modify them. We will learn how to run JEDI Unified Forward Operators (aka H(x) or HofX) using YAML files as configuration.
In real-world applications, x is the state vector provided by the forecast model, and H(x) is an operator that uses the geolocation (latitude, longitude, height) and the nature of the measurements (wind, brightness temperature, etc.) to simulate the observations from the model fields represented by x. In the first part of this practice, we will use the observations and a gridded FV3 model forecast to compute HofX. Since the model data are gridded and the observations are data points, an interpolation of the model fields from the grid to the locations of the observations needs to be performed.
In the second part of this practice, we will see that, in JEDI, we can skip running a model and vastly shorten the development cycle when working on UFO functions. We use for that purpose auxiliary data named GeoVals. The GeoVals files contain the model fields interpolated at observation locations. In short, the interpolation has already been performed beforehand and the results stored in a file. GeoVals files, therefore, are paired with the corresponding observation file. Practically, GeoVaLs are vertical profiles of requested model variables at observation x-y-t location. The forward operator defines which variables it needs from the model to compute H(x).
Examples of GeoVals: radiances: vertical profiles of t, q, ozone, pressure; surface variables: wind, SST, land properties, etc.
radiosondes/aircrafts: vertical profiles of pressure (to do vertical interpolation), t, u, v, q
sea ice concentration retrieval: sea ice concentrations for different ice thickness categories
SST retrieval: SST (observation operator becomes an identity in this case)
Now that you have finished the first activity, you have built and tested fv3-bundle and you’re ready to use it. So, if you are not there already, log in to your AWS node and re-enter the container.
In the first part of this practice, we will see how H(x) is produced in the context of UFO using a model forecast file. In the second part, we will see how H(x) is produced in the context of UFO using a GeoVals file.
Preparation¶
Step 1: Revive your environment¶
Before anything can be run we need to revive the parts of the environment that are not preserved when the instance is stopped and restarted.
Begin by entering the container again:
cd $HOME
singularity shell -e jedi-gnu-openmpi-dev_latest.sif
Your prompt should now look something like:
Singularity>
Once in the container be sure also to remove limits the stack memory to prevent spurious failures:
ulimit -s unlimited
ulimit -v unlimited
Step 2: Install the diagnostics and visualization scripts¶
In order to visualize the HofX output, we need graphics functions. Those functions are provided as a Python package: fv3-jedi-tools
We will need to install this package for viewing the results of our applications after we run them. These will be used in other activities as well, throughout the Academy. Be sure to run these commands inside the container to ensure you have a consistent python environment:
cd ~/jedi
git clone -b 1.1.0 https://github.com/jcsda-internal/fv3-jedi-tools
cd fv3-jedi-tools
/usr/local/miniconda3/bin/pip install --user -e .
Even if FV3-JEDI-TOOLS was installed previously the path still needs to be set in each session:
export PATH=$HOME/.local/bin:$PATH
Run Hofx with a FV3 forecast file¶
Step 3: Download the data¶
These data will be the basis for this tutorial and will introduce some of the concepts in the tutorials covering FV3-JEDI later in the week. Begin by downloading and extracting these data to your tutorials directory:
mkdir -p $HOME/jedi/tutorials
cd $HOME/jedi/tutorials
wget https://fv3-jedi-public.s3.amazonaws.com/Academy/1.1.0/hofx_tutorial.tar.gz
tar -xzf hofx_tutorial.tar.gz
This creates the directory hofx_tutorial. This directory contains everything needed to run HofX with the model forecast and observation files. You can list the directory with the command:
ls $HOME/jedi/tutorials/hofx_tutorial
You will see three subdirectories:
Data: auxiliary data files needed to run the HofX application (CRTM coefficients).
config: Configuration files in YAML format for HofX and the visualization functions.
input: Observations and forecast files (background) needed as input of HofX.
run_hofx.sh: Bash script to run HofX and generate graphics.
Step 4: Run the fv3 based application with fv3 forecast¶
cd $HOME/jedi/tutorials/hofx_tutorial
export JEDI_BUILD_DIR="$HOME/jedi/build"
./run_hofx.sh Atms_n20
./run_hofx.sh Amsua_n19
The script runs the FV3 nomodel hofx application using the background and observation files and produces a plot.
Plots are written out in jedi/tutorials/hofx_tutorial/output/plots
To visualize the image, navigate to the jedi/tutorials/hofx_tutorial/output/plots/Atms_n20 or jedi/tutorials/hofx_tutorial/output/plots/Amsua_n19 directories with the JupyterLab navigation tool and double-click on the image files brightness_temperature-channel9_hofx_20201001_030000.png or brightness_temperature-channel4_hofx_20201001_030000.png.
If you are not using the JupyterLab interface, you can use feh to view the images.
Inside the config directory, note that there are YAML configuration files for a number of observation types in addition to the ATMS and AMSU-A examples that you just ran.
The YAML files with “.jedi.” in their names contain configuration for the hofx application
while the files with “.plot.” in their names contain configuration for the plotting tool.
Take a look inside the application (“.jedi.”) YAML files and note the pairing of the obs space and obs operator specifications.
Try running the run_hofx.sh script without any arguments.
You should get a list of what observation types you can experiment with.
./run_hofx.sh
produces
Usage:
./run.bash <instrument-name>
Where <instrument-name> is one of these:
Aircraft
Amsua_n19
Atms_n20
GnssroBnd
Radiosonde
Satwinds
Select a few of these observation types (e.g., Radiosonde, GnssroBnd) and run the application on those.
./run_hofx.sh Radiosonde
./run_hofx.sh GnssroBnd
Then view the plots in the corresponding output directories.
Run Hofx with a GeoVals file¶
Step 5: Download the data¶
These data will be the basis for this tutorial and the other tutorials covering UFO over the week. Begin by downloading and extracting these data to your tutorials directory:
mkdir -p $HOME/jedi/tutorials
cd $HOME/jedi/tutorials
wget https://fv3-jedi-public.s3.amazonaws.com/Academy/1.1.0/ufo_hofx_tutorial.tar.gz
tar -xzf ufo_hofx_tutorial.tar.gz
This creates the directory ufo_hofx_tutorial. This directory contains everything needed to run HofX with geovals. You can list the directory with the command:
ls $HOME/jedi/tutorials/ufo_hofx_tutorial
You will see three subdirectories:
Data: input data: observations and GeoVals files and auxiliary files (CRTM coefficients).
config: Configuration files in YAML format for HofX and the visualization functions.
run_hofx.sh: Bash script to run HofX and generate graphics.
There’s no input directory this time, because the GeoVals fils are under in the Data directory with the observations files.
Step 6: Run the ufo based application with geovals¶
cd $HOME/jedi/tutorials/ufo_hofx_tutorial
export JEDI_BUILD_DIR="$HOME/jedi/build"
Try running the script without an argument to see the observation types it supports.
./run_hofx.sh
which should produce:
Usage:
./run.bash <instrument-name>
Where <instrument-name> is one of these:
aircraft
amsua_n19
mhs_metop-b
sondes
Take a look in the config directory.
Note that there are YAML files for only the amsua_n19 and sondes cases.
Take a look in these YAML files and note the absense of the model related sections (geometry and state), and the addition of the geovals section.
Now try the examples that are provided.
./run_hofx.sh sondes
./run_hofx.sh amsua_n19
This runs a UFO hofx test application using the geovals and observations files and produces a plot.
Plots are written out in jedi/tutorials/ufo_hofx_tutorial/output/plots
To visualize the image, navigate to the jedi/tutorials/ufo_hofx_tutorial/output/plots/sondes or jedi/tutorials/ufo_hofx_tutorial/output/plots/amsua_n19 directories with the JupyterLab navigation tool and double-click on the image files air_temperature_hofx_20201001_030000.png or brightness_temperature-channel4_hofx_20201001_030000.png
If you are not using the JupyterLab interface, you can use feh to view the images.
Step 7: Add the missing YAML configuration for Aircraft¶
The aircraft observation type is very similar to the radiosonde observation type. Both of these types carry instrument packages for measuring temperature, for example, and record lat, lon, datetime and pressure level. Therefore, they both can use the same observation operator which performs a vertical interpolation.
Because of their similarity, copy the sondes YAML files to their corresponding aircraft files.
cd config
cp sondes.hofx.jedi.yaml aircraft.hofx.jedi.yaml
cp sondes.hofx.plot.yaml aircraft.hofx.plot.yaml
Edit these files and replace “sondes” with “aircraft”. In the “.jedi.” YAML file, replace the name “Radiosonde” with “Aircraft”.
Now try running the application on the new aircraft configuration.
cd ~/jedi/tutorials/ufo_hofx_tutorial
./run_hofx.sh aircraft
Note in the output from the application whether the tests pass.
You may need to adjust the tolerance specification in the application YAML to get the tests to pass.
Step 8: Add the missing YAML configuration for MHS MetOp-B¶
Similar to step 7, you can add the missing YAML configuration for the mhs_metop-b observation type. In this case, the configuration for this observation type is very similar to the amsua_n19 observation type. Both of these types can be simulated using the same observation operator, the Community Radiative Transfer Model (CRTM), named inside UFO as CRTM.
Because of their similarity, copy the amsua_n19 YAML files to their corresponding mhs_metop-b files.
cd config
cp amsua_n19.hofx.jedi.yaml mhs_metop-b.hofx.jedi.yaml
cp amsua_n19.hofx.plot.yaml mhs_metop-b.hofx.plot.yaml
Edit these files and replace “amsua_n19” with “mhs_metop-b”.
In the mhs_metop-b.hofx.jedi.yaml file, replace the name “AMSUA NOAA19” with “MHS METOP-B” and remove the two cloud related YAML keys (Clouds & Cloud_Fraction) by deleting the following lines from the YAML file:
Clouds: [Water, Ice]
Cloud_Fraction: 1.0
In the same file (mhs_metop-b.hofx.jedi.yaml), you need also to adjust the number of channels for MHS MetOp-B. This can be done replacing the key channels: 1-15 (for AMSUA N19) with channels: 1-5 (for MHS MetOp-B).
Now try running the application on the new MHS MetOp-B configuration.
cd ~/jedi/tutorials/ufo_hofx_tutorial
./run_hofx.sh mhs_metop-b