Monday A: Getting Started with AWS, Singularity, and JEDI¶
In this activity you will:
Learn how to access AWS through the academy JupyterLab
Get to know the JEDI development container
Build JEDI fv3-bundle from source
Run the unit tests for fv3-bundle
Explore the JEDI source code and directory structure
Step 1: Access your AWS Instance¶
As a padawan in this JEDI academy, you already have a compute node on the Amazon cloud waiting for you to use it. To start this activity, a JEDI master will materialize in your virtual group and give each of you an ip address and a password. You will use these throughout the week to access your AWS node.
When you receive your ip address and password, you should proceed to log into your compute node as described here. These AWS access instruction are set apart from the activity instructions because you will repeat them every day when you do the activities.
Step 2: Explore the JupyterLab Interface¶
Take a few moments to familiarize yourself with the web interface provided by JupyterLab. Select the terminal tab in the main window to access the linux command line. Find an image file in the directory tree and see what happens when you select it.
Go to the python console window (likely labeled Console 1) and do a calculation: estimate how many seconds you have left of the Academy (and rejoice in the result!). Hint:
<enter> executes a particular cell in the Jupyter notebook; see the Run menu for more options. Open a new ssh terminal by selecting the artist’s palette on the left and scrolling down to
New Terminal. Switch to a dark background for your terminal window if you wish.
Step 3: Download and enter the JEDI Container¶
In order to build JEDI, we will need some of the software packages it depends on. These include build tools such as
ecbuild, IO utilities such as NetCDF, and linear algebra libraries such as LAPACK and Eigen. We’ll also need C++ and Fortran compilers and an MPI implementation. For the activities in this academy, we will acquire these dependencies by means of a software development container. For an overview of what software containers are and why we use them, see the JEDI documentation
Various JEDI containers exist with different plaforms (Singularity, Charliecloud, and Docker), different compilers (gnu, clang, and intel), and different MPI implementations (Openmpi, mpich, Intel MPI). For the Academy we’ll be using the gnu-openmpi Singularity container, which you can obtain by executing the following commands:
cd $HOME singularity pull library://jcsda/public/jedi-gnu-openmpi-dev
If the pull was successful, you should see a new file in your current directory with the name
jedi-gnu-openmpi-dev_latest.sif. If you wish, you can verify that the container came from JCSDA by entering:
singularity verify jedi-gnu-openmpi-dev_latest.sif
Now you can enter the container with the following command:
singularity shell -e jedi-gnu-openmpi-dev_latest.sif
To exit the container at any time (not now!), simply enter
It is worth noting here that this JEDI singularity container is public - as long as you have access to the Singularity software (available on many HPC systems and available to download on to your laptop), you can download it. You do not need to be on AWS.
Step 4: Get to know the Container¶
When you ran the
singularity shell command at the end of Step 1, you entered a new world, or at least a new computing environment. Take a moment to explore it.
First, notice that you are in the same directory as before:
So, things may look the same, though your command line prompt has likely changed. And, you can see that your username is the same as before and your home directory has not changed:
whoami echo $HOME cd ~ ls
You are still the same person. And, more importantly from a system administrator’s perspective, you still have the same access permissions that you did outside of the container. You can still see all the files in your home directory. And, you can still edit them and create new files (give it a try). But things have indeed changed. Enter this:
This tells you that you are now running an ubuntu 18.04 operating system, regardless of what host computer you are on and what operating system it has. Furthermore, take a look at some of the system directories such as:
There you will see a host of JEDI dependencies, such as netcdf, lapack, and eckit, that may not be installed on your host system. Thus, singularity provides its own version of system directories such as
/usr but shares other directories with the host system, such as
$HOME. If you’re familiar with any of these libraries, you can run some commands, for example:
Step 5: Build fv3-bundle¶
JEDI packages are organized into bundles. Each bundle identifies the different GitHub repositories that are needed to run the applications and orchestrates how all of these repositories are built and linked together.
In this tutorial we will build
fv3-bundle. We will put the code in a directory coming off your home directory called
mkdir -p $HOME/jedi cd $HOME/jedi git clone https://github.com/jcsda-academy/fv3-bundle.git
This should create a new directory called
To see what code repositories will be built,
cd to the
fv3-bundle directory and view the file
CMakeLists.txt. Look for the lines that begin with
ecbuild is a collection of CMake utilities that forms the basis of the JEDI build system. The
ecbuild-bundle() function calls specify different GitHub repositories and integrate them into the building of the bundle, in order of dependency.
You will see references there to core JEDI repositories like OOPS, SABER, IODA, and UFO. You will also see references to repositories used to construct observation operators, such as JCSDA’s Community Radiative Transfer Model (CRTM). And, finally, you will see references to GitHub repositories that contain code needed to build the FV3-GFS and FV3-GEOS models and integrate them with JEDI. These include the linearized FV3 model used for 4D Variational DA, and the FV3-JEDI repository that provides the interface between JEDI and models based on the FV3 dynamical core.
Now, an important tip is: never build a bundle from the main bundle directory. In our example this means the top-level
$HOME/jedi/fv3-bundle directory. Building from this directory would cause cmake to create new files that conflict with the original source code.
So, we will create a new build directory and run ecbuild from there:
mkdir -p $HOME/jedi/build cd $HOME/jedi/build ecbuild --build=Release ../fv3-bundle
--build=Release option builds an optimized version of the code so our applications will run a bit faster than if we were to omit it. The only required argument of
ecbuild is the directory where the bundle is.
We have not yet compiled the code; we have merely set the stage. To appreciate part of what these commands have done, take a quick look at the bundle directory:
Do you notice anything different? The bundle directory now includes directories that contain the code repositories that were specified by all those
ecbuild-bundle calls in the
CMakeLists.txt file as described above (apart from a few that are optional):
fv3-jedi etc. If you wish, you can look in those directories and find the source code.
So, one of the things that
ecbuild does is to check to see if the repositories are there. If they are not, it will retreive (clone) them from GitHub. Running the
make update command makes this explicit:
ecbuild more clearly tells you which repositories it is pulling from GitHub and which branches. Running
make update ensures that you get the latest versions of the various branches that are on GitHub. Though this is not necessary for tagged releases (which do not change), it is a good habit to get into if you seek to contribute to the JEDI source code.
All that remains is to actually compile the code (be sure to
cd back to the build directory to run this):
-j4 option tells make to do a parallel build with 4 parallel processes.
While JEDI is building, you can proceed to Step 6.
Step 6: Explore the JEDI code¶
The JEDI code is organized into multiple git repositories, each with its own web interface on GitHub. You may recognize some of the repositority names from today’s lectures - names like
fv3-jedi. If you don’t recognize these yet, you will by the end of the week.
Now explore some of the repositories themselves by navigating the directory tree with the menu on the left. Most have a
src directory where the code is held as well as a
test directory that mimics the structure of the
src directory to test every class, function, module, and subroutine. An exception is
oops which, as the highest-level organizational component is organized a bit differently. Here the QG and Lorenz 95 toy models have their own source and test directories (
oops/l95/test respectively). Navigate to the
oops/src/oops/interface directory to behold some of the generic C++ templates that set JEDI apart from other DA systems.
Note that when you select files of different types (C++, python, etc), the JupyterLab interface will bring them up in a new window, often with appropriate formatting.
Step 7: Run the tests¶
Running the tests gives you an appreciation for how thoroughly the JEDI code is tested. Here we will only run the Tier 1 tests - more computationally extensive higher-tier tests are run regularly with varying frequency. These thoroughly test all the applications, functions, methods, class constructors, and other JEDI components. As emphasized in our working principles, no code is added to JEDI unless there is a test to make sure that it is working and that it continues to work as the code evolves.
A common source of spurious test failure is memory faults due to an insufficient memory stack size, which can lead to segmentation faults. To avoid this, run the following commands before running the JEDI ctests:
ulimit -s unlimited ulimit -v unlimited
Now, to run the test suite, enter the following:
cd $HOME/jedi/build ctest
When the tests complete, you can view the test log as follows (starting from the
cd Testing/Temporary vi LastTest.log
If you selected files from the JupyterLab menu then this creates hidden files that can cause failures in the coding norms tests, for example
oops_coding_norms. You can ignore these if you wish. Or, if you want the tests to pass again you can go to the directory in question and remove the jupyter notebook checkpoint files:
rm -rf `find -type d -name .ipynb_checkpoints`