Monday A: Getting Started with AWS, Singularity, and JEDI¶
In this activity you will:
- split into groups and be assigned a group number that you will use for the rest of the week
- learn how to access AWS through the academy Jupyterlab
- learn how use a JEDI development container
- build JEDI from source
- run the unit tests for fv3-bundle
- explore the JEDI source code and directory structure
Step 1: Access your AWS Instance¶
A JEDI master will direct you to form 20 groups of 2-3 people each. Each group has their own cloud computing instance on which you will build and run JEDI. Once you have a group number, you can log into your compute node as described here. These AWS access instruction are set apart from these activity instructions because you will repeat them every afternoon when you do the activities.
Step 2: Explore the JupyterLab Interface¶
Take a few moments to familiarize yourself with the web interface provided by JupyterLab. Select the terminal tab in the main window to access the linux command line. Navigate the directory tree with the menu on the left. Find an example in the jedi tree of the following file types and see what happens when you select it
- an image file
- a python file (
*.py) - a yaml file (
*.yml) - a text file (
*.txt) - a C++ or Fortran file
Go to the python console window (likely labeled Console 1) and do a calculation: estimate how many seconds you have left of the Academy (and rejoice in the result!). Hint: <shift>-<enter> executes a particular cell in the Jupyter notebook; see the Run menu for more options. Open a new ssh terminal by selecting the artist’s palette on the left and scrolling down to Terminal-New Terminal. Switch to a dark background for your terminal window if you wish.
Step 3: Explore the JEDI source code¶
You may have noticed during Step 2 that the JEDI source code is already there, in the directory ~/jedi/src/fv3-bundle. The JEDI code is organized into multiple git repositories, each with its own web interface on GitHub. You may recognize some of the repositority names from today’s lectures - names like oops, ufo, ioda, saber, and fv3-jedi. If you don’t recognize these yet, you will by the end of the week.
The name of the fv3-bundle directory is significant. JEDI bundles are convenient ways for users to build all of the JEDI components that are needed for a particular application. As its name suggests, fv3-bundle includes the source code for all the repositories require to run the FV3 model within JEDI, with the accompanying data assimilation capabilities of core JEDI repositories including the Object Oriented Prediction System (OOPS), the Interface for Observational Data Assimilation (IODA), the Unified Forward Operator (UFO) and the System-Agnostic Background Error Representation (SABER). To see how this list of repositories is specified, navigate to the ~/jedi/src/fv3-bundle directory and view the file CMakelists.txt.
Now explore some of the repositories themselves. Most have a src directory where the code is held as well as a test directory that mimics the structure of the src directory to test every class, function, module, and subroutine. An exception is oops which, as the highest-level organizational component is organized a bit differently. Here the QG and Lorentz 95 toy models have their own source and test directories (oops/qg/test and oops/l95/test respectively). Navigate to the oops/src/oops/interface directory to behold some of the generic C++ templates that set JEDI apart from other DA systems.
If you wish, you can also explore the JEDI repositories on GitHub.
As part of the setup of your AWS/JupyterLab compute instance, your GitHub credentials were also configured. This allows ecbuild to pull from GitHub when you run make update in Step 6 below.
Each group has its own GitHub user name and password. To see what they are, just enter this from the linux command line:
echo $MYGROUP
echo $MYPW
You can open a separate browser window and copy and paste these credentials to log into Github <https://github.com>. Go ahead and give it a try. As an Academy padawan, you should then be able to explore many of the JEDI repositories on GitHub.
Step 4: Download and enter the JEDI Container¶
You’re not quite ready yet to build JEDI. Though the source code is already there, the JEDI dependencies are not. In order to build JEDI, you’ll need software packages such as ecbuild, netcdf, and Eigen, as well as MPI. These dependencies are provided in a software container. For an overview of what software containers are and why we use them, see the JEDI documentation.
Various JEDI containers exist with different plaforms (Singularity, Charliecloud, and Docker), different compilers (gnu, clang, and intel), and different MPI implementations (Openmpi, mpich, Intel MPI). For the Academy we’ll be using the gnu-openmpi Singularity container, which you can obtain by executing the following commands:
cd $HOME
singularity pull library://jcsda/public/jedi-gnu-openmpi-dev
If the pull was successful, you should see a new file in your current directory with the name jedi-gnu-openmpi-dev_latest.sif. If you wish, you can verify that the container came from JCSDA by entering:
singularity verify jedi-gnu-openmpi-dev_latest.sif
Now you can enter the container with the following command:
singularity shell -e jedi-gnu-openmpi-dev_latest.sif
To exit the container at any time, simply enter
exit
It is worth noting here that this JEDI singularity container is public - as long as you have access to singularity, you can download it. You do not need to be on AWS.
Step 5: Get to know the Container¶
When you ran the singularity shell command at the end of Step 1, you entered a new world, or at least a new computing environment. Take a moment to explore it.
First, notice that you are in the same directory as before:
echo $PWD
So, things may look the same, though your command line prompt has likely changed. And, you can see that your username is the same as before and your home directory has not changed:
whoami
echo $HOME
cd ~
ls
You are still the same person. And, more importantly from a system administrator’s perspective, you still have the same access permissions that you did outside of the container. You can still see all the files in your home directory. And, you can still edit them and create new files (give it a try). But things have indeed changed. Enter this:
lsb_release --all
This tells you that you are now running an ubuntu 18.04 operating system, regardless of what host computer you are on and what operating system it has. Furthermore, take a look at some of the system directories such as:
ls /usr/local/lib
There you will see a host of JEDI dependencies, such as netcdf, lapack, and eckit, that may not be installed on your host system. Thus, singularity provides its own version of system directories such as /usr but shares other directories with the host system, such as $HOME. If you’re familiar with any of these libraries, you can run some commands, for example:
nc-config --all
Step 6: Build and Test JEDI¶
Now you’re ready to build JEDI. To do so, go to the build directory and run the following commands (make sure you do this from inside the container or else you’re likely to get a message that ecbuild is not found):
cd ~/jedi/build
ecbuild ../src/fv3-bundle
make update
The ecbuild command uses CMake to comfigure the Makefiles for each of the repositories. Furthermore, if any of the repositories do not exist yet on your system, ecbuild will clone them from GitHub. If they do exist, the make update command will pull the latest commits from GitHub to ensure that the code is up to date. Now you’re ready to actually compile fv3-bundle:
make -j4
It will take some time for the code to compile. When it is finished, you can run the suite of unit tests for fv3-bundle by entering:
ctest
Both the compilation and the unit testing will take some time. While you’re waiting you can open a new ssh or browser window and return to Step 3 to contine exploring the JEDI source code.
For further tips on working with ecbuild and ctest see the JEDI Documentation on building and testing.
When the tests complete, you can view the test log as follows (starting from the ~/jedi/build directory):
cd Testing/Temporary
vi LastTest.log
Note
If you selected files from the JupyterLab menu then this creates hidden files that can cause failures in the coding norms tests, for example oops_coding_norms. You can ignore these if you wish. Or, if you want the tests to pass again you can go to the directory in question and remove the jupyter notebook checkpoint files:
rm -rf `find -type d -name .ipynb_checkpoints`
Contact Mark Miesch (miesch@ucar.edu) or any other members of the JEDI core team if you have any problems or questions.