################################################
Monday B: observations operators / configuration
################################################

In this tutorial, we will build on the first steps taken this morning and learn more about the JEDI configurations and YAML files and how to modify them. We will learn how to run JEDI Unified Forward Operators (aka H(x) or HofX) using YAML files as configuration.

In real-world applications, :code:`x` is the state vector provided by the forecast model, and :code:`H(x)` is an operator that uses the geolocation (latitude, longitude, height) and the nature of the measurements (wind, brightness temperature, etc.) to simulate the observations from the model fields represented by :code:`x`. In the first part of this practice, we will use the observations and a gridded FV3 model forecast to compute HofX. Since the model data are gridded and the observations are data points, an interpolation of the model fields from the grid to the locations of the observations needs to be performed.

In the second part of this practice, we will see that, in JEDI, we can skip running a model and vastly shorten the development cycle when working on UFO functions. We use for that purpose auxiliary data named GeoVals. The GeoVals files contain the model fields interpolated at observation locations. In short, the interpolation has already been performed beforehand and the results stored in a file. GeoVals files, therefore, are paired with the corresponding observation file. Practically, GeoVaLs are vertical profiles of requested model variables at observation x-y-t location. The forward operator defines which variables it needs from the model to compute :code:`H(x)`.

Examples of GeoVals:
radiances: vertical profiles of t, q, ozone, pressure; surface variables: wind, SST, land properties, etc.

radiosondes/aircrafts: vertical profiles of pressure (to do vertical interpolation), t, u, v, q

sea ice concentration retrieval:  sea ice concentrations for different ice thickness categories

SST retrieval: SST (observation operator becomes an identity in this case)

Now that you have finished the first activity, you have built and tested fv3-bundle and you're ready to use it. So, if you are not there already, log in to your AWS node and re-enter the container.

In the first part of this practice, we will see how H(x) is produced in the context of UFO using a model forecast file. In the second part, we will see how H(x) is produced in the context of UFO using a GeoVals file.

.. _ufo_hofx_development_preparation:

Preparation
***********

.. _ufo_hofx_development_revive:

Step 1: Revive your environment
-------------------------------

Before anything can be run we need to revive the parts of the environment that are not preserved
when the instance is stopped and restarted.

Begin by entering the container again:

.. code:: shell

   cd $HOME
   singularity shell -e jedi-gnu-openmpi-dev_latest.sif

Your prompt should now look something like:

.. code:: shell

   Singularity>

Once in the container be sure also to remove limits the stack memory to prevent spurious failures:

.. code:: shell

   ulimit -s unlimited
   ulimit -v unlimited

.. _ufo_hofx_development_install:

Step 2: Install the diagnostics and visualization scripts
---------------------------------------------------------

In order to visualize the HofX output, we need graphics functions. Those functions are provided as a Python package: fv3-jedi-tools

We will need to install this package for viewing the results of our applications after we run them. These will be used in other activities as well, throughout the Academy. Be sure to run these commands inside the container to ensure you have a consistent python environment:

.. code:: shell

   cd ~/jedi
   git clone -b 1.1.0 https://github.com/jcsda-internal/fv3-jedi-tools
   cd fv3-jedi-tools
   /usr/local/miniconda3/bin/pip install --user -e .

Even if FV3-JEDI-TOOLS was installed previously the path still needs to be set in each session:

.. code:: shell

   export PATH=$HOME/.local/bin:$PATH


.. _ufo_hofx_development_fv3-forecast:

Run Hofx with a FV3 forecast file
*********************************

.. _ufo_hofx_development_fv3_download:

Step 3: Download the data
-------------------------

These data will be the basis for this tutorial and will introduce some of the concepts in the tutorials covering FV3-JEDI later in the week.
Begin by downloading and extracting these data to your tutorials directory:

.. code:: shell

   mkdir -p $HOME/jedi/tutorials
   cd $HOME/jedi/tutorials
   wget https://fv3-jedi-public.s3.amazonaws.com/Academy/1.1.0/hofx_tutorial.tar.gz
   tar -xzf hofx_tutorial.tar.gz


This creates the directory hofx_tutorial. This directory contains everything needed to run HofX with the model forecast and observation files. You can list the directory with the command:

.. code:: shell

   ls $HOME/jedi/tutorials/hofx_tutorial

You will see three subdirectories:

:code:`Data`: auxiliary data files needed to run the HofX application (CRTM coefficients).

:code:`config`: Configuration files in YAML format for HofX and the visualization functions.

:code:`input`: Observations and forecast files (background) needed as input of HofX.

:code:`run_hofx.sh`: Bash script to run HofX and generate graphics.

.. _ufo_hofx_development_run_fv3:

Step 4: Run the fv3 based application with fv3 forecast
-------------------------------------------------------

.. code:: shell

   cd $HOME/jedi/tutorials/hofx_tutorial
   export JEDI_BUILD_DIR="$HOME/jedi/build"

.. code:: shell

   ./run_hofx.sh Atms_n20

.. code:: shell

   ./run_hofx.sh Amsua_n19

The script runs the FV3 nomodel hofx application using the background and observation files and produces a plot.
Plots are written out in :code:`jedi/tutorials/hofx_tutorial/output/plots`

To visualize the image, navigate to the :code:`jedi/tutorials/hofx_tutorial/output/plots/Atms_n20` or :code:`jedi/tutorials/hofx_tutorial/output/plots/Amsua_n19` directories with the JupyterLab navigation tool and double-click on the image files :code:`brightness_temperature-channel9_hofx_20201001_030000.png` or :code:`brightness_temperature-channel4_hofx_20201001_030000.png`.

If you are not using the JupyterLab interface, you can use :code:`feh` to view the images.

Inside the :code:`config` directory, note that there are YAML configuration files for a number of observation types in addition to the ATMS and AMSU-A examples that you just ran.
The YAML files with ".jedi." in their names contain configuration for the hofx application
while the files with ".plot." in their names contain configuration for the plotting tool.
Take a look inside the application (".jedi.") YAML files and note the pairing of the :code:`obs space` and :code:`obs operator` specifications.

Try running the :code:`run_hofx.sh` script without any arguments.
You should get a list of what observation types you can experiment with.

.. code:: shell

    ./run_hofx.sh

produces

.. code:: shell

  Usage:
  ./run.bash <instrument-name>
  Where <instrument-name> is one of these:
  Aircraft
  Amsua_n19
  Atms_n20
  GnssroBnd
  Radiosonde
  Satwinds

Select a few of these observation types (e.g., Radiosonde, GnssroBnd) and run the application on those.

.. code:: shell

  ./run_hofx.sh Radiosonde
  ./run_hofx.sh GnssroBnd

Then view the plots in the corresponding output directories.

.. _ufo_hofx_development_geovals:

Run Hofx with a GeoVals file
****************************

.. _ufo_hofx_development_geovals_download:

Step 5: Download the data
-------------------------

These data will be the basis for this tutorial and the other tutorials covering UFO over the week.
Begin by downloading and extracting these data to your tutorials directory:

.. code:: shell

   mkdir -p $HOME/jedi/tutorials
   cd $HOME/jedi/tutorials
   wget https://fv3-jedi-public.s3.amazonaws.com/Academy/1.1.0/ufo_hofx_tutorial.tar.gz
   tar -xzf ufo_hofx_tutorial.tar.gz


This creates the directory ufo_hofx_tutorial. This directory contains everything needed to run HofX with geovals. You can list the directory with the command:

.. code:: shell

   ls $HOME/jedi/tutorials/ufo_hofx_tutorial

You will see three subdirectories:

:code:`Data`: input data: observations and GeoVals files and auxiliary files (CRTM coefficients).

:code:`config`: Configuration files in YAML format for HofX and the visualization functions.

:code:`run_hofx.sh`: Bash script to run HofX and generate graphics.

There's no :code:`input` directory this time, because the GeoVals fils are under in the :code:`Data` directory with the observations files.

.. _ufo_hofx_development_run_ufo:

Step 6: Run the ufo based application with geovals
--------------------------------------------------

.. code:: shell

   cd $HOME/jedi/tutorials/ufo_hofx_tutorial
   export JEDI_BUILD_DIR="$HOME/jedi/build"

Try running the script without an argument to see the observation types it supports.

.. code:: shell

   ./run_hofx.sh

which should produce:

.. code:: shell

  Usage: 
  ./run.bash <instrument-name>
  Where <instrument-name> is one of these:
  aircraft
  amsua_n19
  mhs_metop-b
  sondes

Take a look in the :code:`config` directory.
Note that there are YAML files for only the amsua_n19 and sondes cases.
Take a look in these YAML files and note the absense of the model related sections (:code:`geometry` and :code:`state`), and the addition of the :code:`geovals` section.

Now try the examples that are provided.

.. code:: shell

   ./run_hofx.sh sondes

.. code:: shell

   ./run_hofx.sh amsua_n19


This runs a UFO hofx test application using the geovals and observations files and produces a plot.
Plots are written out in :code:`jedi/tutorials/ufo_hofx_tutorial/output/plots`

To visualize the image, navigate to the :code:`jedi/tutorials/ufo_hofx_tutorial/output/plots/sondes` or :code:`jedi/tutorials/ufo_hofx_tutorial/output/plots/amsua_n19` directories with the JupyterLab navigation tool and double-click on the image files :code:`air_temperature_hofx_20201001_030000.png` or :code:`brightness_temperature-channel4_hofx_20201001_030000.png` 

If you are not using the JupyterLab interface, you can use :code:`feh` to view the images.

.. _ufo_hofx_development_add_aircraft:

Step 7: Add the missing YAML configuration for Aircraft
-------------------------------------------------------

The aircraft observation type is very similar to the radiosonde observation type.
Both of these types carry instrument packages for measuring temperature, for example, and record lat, lon, datetime and pressure level.
Therefore, they both can use the same observation operator which performs a vertical interpolation.

Because of their similarity, copy the sondes YAML files to their corresponding aircraft files.

.. code:: shell

  cd config
  cp sondes.hofx.jedi.yaml aircraft.hofx.jedi.yaml
  cp sondes.hofx.plot.yaml aircraft.hofx.plot.yaml

Edit these files and replace "sondes" with "aircraft".
In the ".jedi." YAML file, replace the name "Radiosonde" with "Aircraft".

Now try running the application on the new aircraft configuration.

.. code:: shell

   cd ~/jedi/tutorials/ufo_hofx_tutorial
   ./run_hofx.sh aircraft

Note in the output from the application whether the tests pass.
You may need to adjust the :code:`tolerance` specification in the application YAML to get the tests to pass.

.. _ufo_hofx_development_add_metopb:

Step 8: Add the missing YAML configuration for MHS MetOp-B
----------------------------------------------------------

Similar to step 7, you can add the missing YAML configuration for the :code:`mhs_metop-b` observation type. In this case, the configuration for this observation type is very similar to the :code:`amsua_n19` observation type. Both of these types can be simulated using the same observation operator, the Community Radiative Transfer Model (CRTM), named inside UFO as :code:`CRTM`.

Because of their similarity, copy the :code:`amsua_n19` YAML files to their corresponding :code:`mhs_metop-b` files.

.. code:: shell

  cd config
  cp amsua_n19.hofx.jedi.yaml mhs_metop-b.hofx.jedi.yaml
  cp amsua_n19.hofx.plot.yaml mhs_metop-b.hofx.plot.yaml

Edit these files and replace "amsua_n19" with "mhs_metop-b".

In the :code:`mhs_metop-b.hofx.jedi.yaml` file, replace the name "AMSUA NOAA19" with "MHS METOP-B" and remove the two cloud related YAML keys (:code:`Clouds` & :code:`Cloud_Fraction`) by deleting the following lines from the YAML file:

.. code:: shell

    Clouds: [Water, Ice]
    Cloud_Fraction: 1.0

In the same file (:code:`mhs_metop-b.hofx.jedi.yaml`), you need also to adjust the number of channels for MHS MetOp-B. This can be done replacing the key :code:`channels: 1-15` (for AMSUA N19) with :code:`channels: 1-5` (for MHS MetOp-B).

Now try running the application on the new MHS MetOp-B configuration.

.. code:: shell

   cd ~/jedi/tutorials/ufo_hofx_tutorial
   ./run_hofx.sh mhs_metop-b