Monday B: Running an Application, Git Flow, and ZenHub

In this activity you will:

  • Run a JEDI application inside the container
  • Generate and view a 3DVar increment and other diagnostics
  • Modify a JEDI configuration file to alter program execution
  • create a feature branch
  • issue a pull request
  • organize and track your work with ZenHub

Step 1: Run a JEDI Application

In the previous activity we ran JEDI applications through the ctest test suite. Here we will run JEDI applications directly, independent of ctest. We will start with a 3D Variational Data Assimilation application (3DVar) and then move on to more complex applications later in the week.

The first step is to collect the run script and associated input files in a convenient location. To do so, run the following commands outside the container (because rsync isn’t installed in the container):

cd
mkdir academy
rsync -av ~/jedi/build/academy/ academy/   # rsync preserves symlinks

Take a look at the files you just copied over. The run script defines a workflow that is needed to run a variational data assimilation application with fv3-jedi and the B-Matrix Unstructured Mesh Package (BUMP). First BUMP is used to compute the correlation statistics and localization for the background error covariance matrix (B-Matrix). Then the variational application is run, and a seperate application computes the increment for visualization and analysis. Each of these applications runs with 6 MPI tasks (the minimum for fv3) and each takes only two arguments, namely a (yaml) configuration file and a filename for storing the text output messages (i.e. the log).

The conf directory contains jedi configuration files in yaml format that govern the execution of the application, including the specification of input data files, control flags, and parameter values. If you look inside, you’ll see references to where the input data files are. For example, the ~/jedi/src/fv3-bundle/fv3-jedi/test/Data/fv3files contains namelist and other configuration files for the FV3 model and the ~/jedi/src/fv3-bundle/fv3-jedi/test/Data/inputs/gfs_c12 directory contains model backgrounds and ensemble states that are used to define the grid, initialize forecasts, and compute the B-Matrix. The c12 refers to the horizontal resolution, signifying 12 by 12 grid points on each of the 6 faces of the cubed sphere grid, or 864 horizontal grid points total. This is, of course, much lower resolution than operational forecasts but it is sufficient to run efficiently for a tutorial!

If you peruse the config files further, you may see references to the data/obs and ~/jedi/build/fv3-jedi/test/Data/obs directory, which contains links to the observation files that will be assimilated. Another source of input data is the ~/jedi/build/fv3-jedi/test/Data/crtm directory, which contains coefficients for the Community Radiative Transfer Model (CRTM) that are used to compute simulated satellite radiance observations from model states (i..e. the forward operator).

We again encourage you to explore these various directories to get a feel for how the input to jedi applications is provided.

To run a hybrid 3D variational data assimilation application, first enter the Singularity container so you have all the runtime libraries you need. Then just execute the run script, specifying hyb-3dvar as the application you wish to run:

cd
singularity shell -e jedi-gnu-openmpi-dev_latest.sif
cd academy
./run.bash hyb-3dvar

The output of each this experiment can now be found in the run-hyb-3dvar directory. You may wish to take a few moments to survey the types of output files that are produced.

Step 2: Plot convergence, H(x), and innovation statistics

The academy branch of the fv3-jedi repository includes python scripts that can be used to vizualize the output of the 3DVar application you ran in Step 2. They can be found in the in the test/diagnostics subdirectory of the repository but they are copied over to the ~/jedi/build/bin directory upon compilation.

You may choose to run these diagnostic visualization scripts either inside the Singularity container or outside. Either way, you should make sure they are in your path to make running them easier.

The preferred method is to run the python scripts outside the container, perhaps in a separate terminal window from your Singularity window. This is the easiest to set up. To do this, you just have to modify your path as follows to add the executables from the JEDI build directory:

# for generating plots outside the Singularity container
export PATH=$HOME/jedi/build/bin:$PATH

Or, if you’d rather run the python scripts from inside the container, there are a few more things you need to add to your path (these make cartopy and other locally-installed python packages available within the container). You also need to let Singularty know how to handle your X window configuration

# you may need to set this before you enter the container, if it's not set already
 export DISPLAY=localhost:10.0

# then, from within the container, enter:
 export PATH=$HOME/jedi/build/bin:/home/ubuntu/miniconda3/bin:/home/ubuntu/miniconda3/condabin:/home/ubuntu/.local/bin:$PATH
 export DISPLAY=localhost:10.0

Now, to see the usage of any of the python scripts, call the script with the -h or --help option:

convergence.py --help

You see that the convergence script expects a JEDI assimilation log file as input, along with an optional specification of the type of norm. To run it, just follow the instructions:

cd ~/academy/run-hyb-3dvar
convergence.py -l logs/hyb-3dvar.log

This should generate a plot in the same directory where you ran the script. You can view it by navigating the JupyterLab menu on the left to that directory and selecting the image file. Now try some of the other options for --jnorm.

hofx.py -f hofx/amsua_metop-a_hyb-3dvar-gfs_2018041500_m_NPROC.nc4

Again, you can view the result by selecting the file from the menu and you can explore different options. Note that the default value for the variable is brightness_temperature_9@ObsValue, which is the input observations. To see the actual H(x) that is produced, set the variable name to be brightness_temperature_9@hofx2. Compare the two plots.

Now try plotting out innovation statitics in the same manner, using the script innovation.py.

Step 4: Plot the increment

As mentioned above, the application in the run.bash script generates an increment that can be used for visualization. Like many of the other outputs, this is rendered as a netcdf file and it can be visualized with one of the jedi-fv3 diagnostic scripts:

field_level_plot.py --input increment/hyb-3dvar-gfs.20180415_000000.nc4 --field T --field_lines DELP --field_barbs ua,va

Experiment with different plots. Plot out different variables and levels. Hint: run ncdump on the increment file to see what variables are available.

Step 5: Change the Configuration

We encourage you to edit the hyb-3dvar configuration files in your local academy/conf directory and see how that effects the resulting increments (save the 4dvar files for later in the week).

Here are a few possible activities - we encourage you to come up with your own:

  • change the variable list in one or more of the observations that are assimilated. For example, you can remove eastward_wind and northward_wind from the aircraft and/or radiosonde observations, leaving only temperature.
  • remove one of the observation types entirely, such as aircraft or GNSSRO refractivity measurements (hint: you may wish to review the basic yaml syntax to see how components of a particular yaml item are defined).
  • change the localization length scales for bump (hint: rh and rv correspond to horizonal and vertical length scales respectively, in units of meters)
  • save your modifications in a config file with a different name and use that one instead

After each change remember to run the run.bash script again to generate new output.

Step 6: Make a feature branch

So far we haven’t modified the JEDI code at all; we have just been working in our own ~/academy space. Now we will explore how developers can contribute to the JEDI code base. The changes we make in this activity will be trivial but they are intended to illustrate the process that should be followed when JEDI users and developers are prepared to make more substantial contributions. Throughout we will highlight the Git Flow paradigm emphasized in the lectures.

First navigate to the ~/jedi/src/fv3-bundle/fv3-jedi directory and create a new feature branch. If you are working inside the Singularity container, you can use the git flow extension to do this. But, first you’ll have to initialize it as follows:

cd ~/jedi/src/fv3-bundle/fv3-jedi
git flow init

A prompt will ask you what branch you would like to use for bringing forth production releases. Enter master. At the next prompt, asking which branch to use for the integration of the next release, enter develop. Then at the remaining prompts you can simply press enter, choosing the default values.

Now you can create a new feature branch as follows:

git flow feature start <mybranch>  # give it a unique name - not 'mybranch'!

You do not have to use the git flow extension - you can create your feature branch with standard git commands. Just make sure you follow the Git-Flow naming conventions discussed in the lectures. And, make sure you give your branch a unique name - maybe based on your group number.

Now pick your favorite modification to the configuration files from Step 5. To minimize conflicts with everybody editing the same file, you should give your modified yaml file a unique name and then copy it to the ~/jedi/src/fv3_bundle/fv3-jedi/academy/conf directory. Now you have changed the code base. You can see this is the case by running:

git status

Now commit your changes to your feature branch, entering an informative message. You can then push it to GitHub with the following command:

git flow feature publish <mybranch>

Note - you should not use your personal GitHub credentials (if you have an account) because it is unlikely that you have push/write access to the jedi-ac01/fv3-jedi repository.

Step 7: Make a Pull Request and track it with ZenHub

Normally when you make a code change, you would add a test to verify that it works as intended. We will skip that step in this activity, though we will return to writing unit tests later in the week.

For now we’ll say that we’re ready for our branch to be merged into the develop branch. You’ll have to do this on GitHub, not the command line. Recall that you can retrieve your GitHub user name and password by entering the following on the command line:

echo $MYGROUP $MYPW

So, open a web browser, log in to GitHub, and navigate to the jedi-ac01/fv3-jedi repository.

Find your feature branch in the drop-down menu and issue a pull request. Be sure to Ask other groups to review your pull request. At a minimum, ask the two groups adjacent to yours. So, if you are group 7, for example, you would ask for a code review from Groups 6 and 8. Remember there are 20 groups so group 20 would ask for reviews from Groups 19 and 1 while Group 1 would ask for reviews from Groups 20 and 2.

However, we are not done. Our pull request is not complete until we have provided all the necessary information about the contribution. For this we need ZenHub.

If you use Chrome or Firefox, the easiest way to access ZenHub is by installing the browser extension, which you can do here. If you use a different browser, you can access the ZenHub boards through the ZenHub website (sign in using your the GitHub credentials for your Group).

Navigate to the ZenHub project board for the jedi-ac01/fv3-jedi repository. You will see a number of issues and you will find your pull request there. Note that each pull request in GitHub is an issue in GitHub and each issue in GitHub appears as an issue on the ZenHub project board. Click on your issue to bring up an interactive window.

Now fill in the information in the window. When you are reviewing other pull requests, do not approve them until you have made sure that the authors have provided all of the following information:

  • Issue title
  • Informative description
  • At least two code reviewers
  • A label
  • An estimate of effort
  • An Epic identifying where this change fits into the JCSDA AOP (Note that JEDI6: Academies is a legitimate Epic)

If you have any questions about how to fill any of these items out, consult you lecture notes or ask a JEDI master. Also, as a reviewer (and an author), you should make sure all merge conflicts have been resolved.

You’re not done with this step until you have submitted your own pull request and reviewed at least two others. You do not have to be explicitly invited to review a pull request - you can view, comment on, and review any of the pull requests in the repository. Don’t hesitate to comment on the code changes that others have made. Each pull request may go through several changes before it is approved.

As pull requests are approved, a JEDI master will merge them into develop.

Step 8: Create a ZenHub Issue near and dear to your heart

The estimate of effort for all the pull requests submitted in Step 7 is likely to be small. After all, we made all these changes in the last hour or two. But you did not participate in the JEDI Academy to make a trivial change in an Academy configuration file. What do you want to achieve with JEDI? How do you want to enhance its capabilities? How do you want to contribute? How much effort will this take? Whom will be involved? How does it fit into the JCSDA AOP?

In this step, we encourage you to create a new ZenHub issue that describes the change you’d like to make. Make sure you fill in all the fields itemized above, with the exception of reviewers, since this is not a pull request: title, description, labels, estimate, epic. In place of reviewers, you can assign one or more people to be responsible for carrying it out, starting with yourself (you’ll need to know the GitHub user names of others if you wish to assign them). Here you can use personal GitHub user names if you wish (as opposed to Academy group accounts). Since there are multiple members of each group, feel free to create multiple issues. This is not only an exercise to help you become familiar with ZenHub; it is also useful information for the JEDI core team. We’d like to hear what you have in mind.

Finally, before calling it a day, have a look at some of the issues your fellow padawans have created. Feel free to comment on them. You may even wish to collaborate.

Contact Mark Miesch (miesch@ucar.edu) or any other members of the JEDI core team if you have any problems or questions.