Monday: Building and Running JEDI on AWS

Steps 1-4: Access AWS

Each group has their own compute node running and patiently waiting for them on Amazon Web Services (AWS). Each AWS node is provisioned with the software you need to build, run, and otherwise work with the JEDI code.

In particular, all week we will be working with the fv3-bundle. This is a packaged CMake build system that includes all the GitHub repositories needed to run the Finite Volume Cubed Sphere (FV3) code within JEDI, including OOPS, IODA, UFO, CRTM, and FV3-JEDI itself.

Since accessing your AWS node is something that you will do every day, the instructions on how to do so are posted on the main page for this Academy.

A few comments are in order about the build process (Step 4), repeated here for reference:

cd ~/jedi/build
rm -rf *
ecbuild --build=Release ../src/fv3-bundle
make update
make -j4

First, note that the build is done in a separate directory from the source code. This is important; it avoids conflicts and allows you to easily clean the build directory when you want to do a fresh build.

Second, the ECMWF package ecbuild is used to configure the build before running make. ecbuild consists of a collection of CMake macros and modules that facilitate the build process by finding and linking the appropriate compilers and libraries.

Third, note that the directory passed to ecbuild is a bundle, in this case fv3-bundle. Have a look at the contents of that directory, ../src/fv3-bundle. Most of the subdirectories there are individual git repositories that ecbuild has retrieved from GitHub during previous builds. We have included these here to avoid the need for downloading large data files but if you wish, you can see how ecbuild works by deleting one of the directories, such as ufo and then seeing how ecbuild will do a fresh clone if it is not there (hint: we recommend that you do not re-clone crtm, fv3-jedi or ioda because these have many files on git-LFS that take a while to download).

You can look in the file ../src/fv3-bundle/CMakeLists.txt to see the list of repositories that are involved in building fv3-bundle. Look for statements such as this that add repositories from GithHub, specifying the branch.

ecbuild_bundle( PROJECT oops GIT "https://github.com/JCSDA/oops.git" BRANCH feature/academy UPDATE )

If a repository is not there, ecbuild will clone it. Then, it will check out the specified branch. The UPDATE argument to ecbuild_bundle tells ecbuild to pull the latest changes from GitHub before compiling. This is done by executing the make update command in the build sequence above.

After you have successfully built and tested fv3-bundle, please return to this page and proceed to Step 5.

Step 5: Run an Application

We have prepared a 3DVar application for you to run. To execute the code, enter these commands

cd ~/academy
./run.bash hyb-3dvar 2>&1 | tee run.log

Now take a moment to investigate what you just did. Look through the run.bash file to see what commands were executed. Look through the run.log file to see the output. Note in particular the State and Increment print sections, the Cost function minimization, and the timiing statistics.

Step 6: Visualize the increment

To visualize the increment we just generated with our 3D Var application, we will use a software product developed by NASA called Panoply.

Panoply is installed on your AWS node but it is a GUI that runs slowly across the internet. So, we recommend that you run panoply from your local workstation.

As you recall from Step 2, the aws_init script that was provided to you is used to initialize your AWS connection, including setting the AWSIP environment variable to proper value for your group. It also defines other useful aliases. So, to download the the increment from AWS to your workstation enter

cd <path-to-aws-int>
source aws_init.csh # or .sh if using bash
get-increment

This should download a NetCDF file with a name something like: hyb-3dvar-inc.20180415_000000z.nc4.

Now fire up Panoply by running this script:

panoply.sh

In the finder screen, navigate to the directory that contains the increment file, select it, and then select Open. Select Temperature from the list of data sets. Then, in the upper left corner, select Create Plot. A dialog box will come up: you can leave all the settings at their default values and select Create.

Now you should be able to see the temperature increment. Note the color table on the bottom and the gray outlines that mark the edges of the cubed sphere. This shows a level at the top of the domain (level 1) by default. To see a more representative level lower down in the atmosphere, go to the Vertical level menu item at the bottom and select level 50.

Now play around a bit with Panoply. Explore it’s options. Try plotting out zonal averages instead of a map. Navigate to the Map tabe and try a different projection. Go back to the original window and create a new plot with a different field. Try comibine plots.

Step 7: Build and Run JEDI in the Singularity Container

Thus far we have been working with the JEDI environment modules, which are a convenient way to provide the JEDI software dependencies on HPC and cloud platforms. However, we also provide software containers that include all of the JEDI dependencies. These are a useful way to get up and running with JEDI quickly on any platform.

In this exercise we will use the Singulary container. Alternatively, the JEDI team also provides a Charliecloud container for use on systems where Singularity is not available.

To obtain the JEDI Singularity container from Singularity Hub and enter it, execute the following commands:

singularity pull shub://JCSDA/singularity
singularity shell -e singularity_latest.sif

The first command will take a few minutes so please be patient. After executing the second command, you will be “inside the container” - an isolated user environment that is distinct from the host environment.

Look around a bit. Note that you can still see the files in your home directory. And, if you enter whoami you see that you are still you - namely the user ubuntu. Everything may look similar. That is, until you enter a command like this:

ls /usr/local/lib

This will give a very different result depending on whether you are inside or outside the container. From inside the container, you will see all the JEDI dependencies pre-installed and ready to use.

Now try to build and run JEDI from within the container. In this case, the software versions inside and outside the container (in the environment module) are the same so you could use the same build directory. But, this is not always the case so it is good practice to use a different build directory (or wipe the other one clean).

So, to build JEDI you can enter these commands.

cd ~/jedi/build_sin
ecbuild --build=Release ../src/fv3-bundle
make -j4

Note that we access the same source code both inside and outside the container so there is no need to clone or pull the repositories again.

Now you can run ctest from the build directory and/or you can run the application again by entering the following:

cd ~/academy
./run-sin.bash hyb-3dvar 2>&1 | tee run-sin.log

Feel free to compare the run-sin.bash script for Singularity and the run.bash that we used to run the application outside the container. The main difference is that the Singularity container does not use modules.

Step 8: Explore the JEDI GitHub Repos

From your AWS account, enter this:

echo $MYGROUP $MYPW

This is your group’s username and password for GitHub. In a web browser, navigate to https://github.com and use these credentials to log in. As an Academy padawan you have read access to many of the JCSDA repositories. You can get acquainted with oops, ufo, ioda, crtm, and fv3-jedi or others that pique your interest, such as ioda-converters. Find a JEDI master near you to ask questions. If we have time, we will show you how to install the ZenHub extension to see the project boards.

Contact Mark Miesch (miesch@ucar.edu) or any other members of the JEDI core team if you have any problems or questions.