Monday A: Getting Started with AWS, Singularity, and JEDI

In this activity you will:

  • Learn how to access AWS through the academy JupyterLab

  • Get to know the JEDI development container

  • Build JEDI fv3-bundle from source

  • Run the unit tests for fv3-bundle

  • Explore the JEDI source code and directory structure

Step 1: Access your AWS Instance

As a padawan in this JEDI academy, you already have a compute node on the Amazon cloud waiting for you to use it. To start this activity, a JEDI master will materialize in your virtual group and give each of you an ip address and a password. You will use these throughout the week to access your AWS node.

When you receive your ip address and password, you should proceed to log into your compute node as described here. These AWS access instruction are set apart from the activity instructions because you will repeat them every day when you do the activities.

Step 2: Explore the JupyterLab Interface

Take a few moments to familiarize yourself with the web interface provided by JupyterLab. Select the terminal tab in the main window to access the linux command line. Find an image file in the directory tree and see what happens when you select it.

Go to the python console window (likely labeled Console 1) and do a calculation: estimate how many seconds you have left of the Academy (and rejoice in the result!). Hint: <shift>-<enter> executes a particular cell in the Jupyter notebook; see the Run menu for more options. Open a new ssh terminal by selecting the artist’s palette on the left and scrolling down to Terminal-New Terminal. Switch to a dark background for your terminal window if you wish.

Step 3: Download and enter the JEDI Container

In order to build JEDI, we will need some of the software packages it depends on. These include build tools such as CMake and ecbuild, IO utilities such as NetCDF, and linear algebra libraries such as LAPACK and Eigen. We’ll also need C++ and Fortran compilers and an MPI implementation. For the activities in this academy, we will acquire these dependencies by means of a software development container. For an overview of what software containers are and why we use them, see the JEDI documentation

Various JEDI containers exist with different plaforms (Singularity, Charliecloud, and Docker), different compilers (gnu, clang, and intel), and different MPI implementations (Openmpi, mpich, Intel MPI). For the Academy we’ll be using the gnu-openmpi Singularity container, which you can obtain by executing the following commands:

cd $HOME
singularity pull library://jcsda/public/jedi-gnu-openmpi-dev

If the pull was successful, you should see a new file in your current directory with the name jedi-gnu-openmpi-dev_latest.sif. If you wish, you can verify that the container came from JCSDA by entering:

singularity verify jedi-gnu-openmpi-dev_latest.sif

Now you can enter the container with the following command:

singularity shell -e jedi-gnu-openmpi-dev_latest.sif

To exit the container at any time (not now!), simply enter


It is worth noting here that this JEDI singularity container is public - as long as you have access to the Singularity software (available on many HPC systems and available to download on to your laptop), you can download it. You do not need to be on AWS.

Step 4: Get to know the Container

When you ran the singularity shell command at the end of Step 1, you entered a new world, or at least a new computing environment. Take a moment to explore it.

First, notice that you are in the same directory as before:

echo $PWD

So, things may look the same, though your command line prompt has likely changed. And, you can see that your username is the same as before and your home directory has not changed:

echo $HOME
cd ~

You are still the same person. And, more importantly from a system administrator’s perspective, you still have the same access permissions that you did outside of the container. You can still see all the files in your home directory. And, you can still edit them and create new files (give it a try). But things have indeed changed. Enter this:

lsb_release --all

This tells you that you are now running an ubuntu 18.04 operating system, regardless of what host computer you are on and what operating system it has. Furthermore, take a look at some of the system directories such as:

ls /usr/local/lib

There you will see a host of JEDI dependencies, such as netcdf, lapack, and eckit, that may not be installed on your host system. Thus, singularity provides its own version of system directories such as /usr but shares other directories with the host system, such as $HOME. If you’re familiar with any of these libraries, you can run some commands, for example:

nc-config --all

Step 5: Build fv3-bundle

JEDI packages are organized into bundles. Each bundle identifies the different GitHub repositories that are needed to run the applications and orchestrates how all of these repositories are built and linked together.

In this tutorial we will build fv3-bundle. We will put the code in a directory coming off your home directory called jedi:

mkdir -p $HOME/jedi
cd $HOME/jedi
git clone

This should create a new directory called $HOME/jedi/fv3-bundle.

To see what code repositories will be built, cd to the fv3-bundle directory and view the file CMakeLists.txt. Look for the lines that begin with ecbuild-bundle.

ecbuild is a collection of CMake utilities that forms the basis of the JEDI build system. The ecbuild-bundle() function calls specify different GitHub repositories and integrate them into the building of the bundle, in order of dependency.

You will see references there to core JEDI repositories like OOPS, SABER, IODA, and UFO. You will also see references to repositories used to construct observation operators, such as JCSDA’s Community Radiative Transfer Model (CRTM). And, finally, you will see references to GitHub repositories that contain code needed to build the FV3-GFS and FV3-GEOS models and integrate them with JEDI. These include the linearized FV3 model used for 4D Variational DA, and the FV3-JEDI repository that provides the interface between JEDI and models based on the FV3 dynamical core.

Now, an important tip is: never build a bundle from the main bundle directory. In our example this means the top-level $HOME/jedi/fv3-bundle directory. Building from this directory would cause cmake to create new files that conflict with the original source code.

So, we will create a new build directory and run ecbuild from there:

mkdir -p $HOME/jedi/build
cd $HOME/jedi/build
ecbuild --build=Release ../fv3-bundle

The --build=Release option builds an optimized version of the code so our applications will run a bit faster than if we were to omit it. The only required argument of ecbuild is the directory where the bundle is.

We have not yet compiled the code; we have merely set the stage. To appreciate part of what these commands have done, take a quick look at the bundle directory:

ls ../fv3-bundle

Do you notice anything different? The bundle directory now includes directories that contain the code repositories that were specified by all those ecbuild-bundle calls in the CMakeLists.txt file as described above (apart from a few that are optional): oops, saber, ioda, ufo, fv3-jedi etc. If you wish, you can look in those directories and find the source code.

So, one of the things that ecbuild does is to check to see if the repositories are there. If they are not, it will retreive (clone) them from GitHub. Running the make update command makes this explicit:

make update

Here ecbuild more clearly tells you which repositories it is pulling from GitHub and which branches. Running make update ensures that you get the latest versions of the various branches that are on GitHub. Though this is not necessary for tagged releases (which do not change), it is a good habit to get into if you seek to contribute to the JEDI source code.

All that remains is to actually compile the code (be sure to cd back to the build directory to run this):

make -j4

The -j4 option tells make to do a parallel build with 4 parallel processes.

While JEDI is building, you can proceed to Step 6.

Step 6: Explore the JEDI code

The JEDI code is organized into multiple git repositories, each with its own web interface on GitHub. You may recognize some of the repositority names from today’s lectures - names like oops, ufo, ioda, saber, and fv3-jedi. If you don’t recognize these yet, you will by the end of the week.

Now explore some of the repositories themselves by navigating the directory tree with the menu on the left. Most have a src directory where the code is held as well as a test directory that mimics the structure of the src directory to test every class, function, module, and subroutine. An exception is oops which, as the highest-level organizational component is organized a bit differently. Here the QG and Lorenz 95 toy models have their own source and test directories (oops/qg/test and oops/l95/test respectively). Navigate to the oops/src/oops/interface directory to behold some of the generic C++ templates that set JEDI apart from other DA systems.

Note that when you select files of different types (C++, python, etc), the JupyterLab interface will bring them up in a new window, often with appropriate formatting.

If you wish, you can also explore the JEDI repositories on GitHub and the documentation for many of the components in our online users manual.

Step 7: Run the tests

Running the tests gives you an appreciation for how thoroughly the JEDI code is tested. Here we will only run the Tier 1 tests - more computationally extensive higher-tier tests are run regularly with varying frequency. These thoroughly test all the applications, functions, methods, class constructors, and other JEDI components. As emphasized in our working principles, no code is added to JEDI unless there is a test to make sure that it is working and that it continues to work as the code evolves.

A common source of spurious test failure is memory faults due to an insufficient memory stack size, which can lead to segmentation faults. To avoid this, run the following commands before running the JEDI ctests:

ulimit -s unlimited
ulimit -v unlimited

Now, to run the test suite, enter the following:

cd $HOME/jedi/build

When the tests complete, you can view the test log as follows (starting from the ~/jedi/build directory):

cd Testing/Temporary
vi LastTest.log


If you selected files from the JupyterLab menu then this creates hidden files that can cause failures in the coding norms tests, for example oops_coding_norms. You can ignore these if you wish. Or, if you want the tests to pass again you can go to the directory in question and remove the jupyter notebook checkpoint files:

rm -rf `find -type d -name .ipynb_checkpoints`

For further tips on working with ecbuild and ctest see the JEDI Documentation on building and testing.