########################################### Monday: Building and Running JEDI on AWS ########################################### Steps 1-4: Access AWS ---------------------- Each group has their own compute node running and patiently waiting for them on Amazon Web Services (AWS). Each AWS node is provisioned with the software you need to build, run, and otherwise work with the JEDI code. In particular, all week we will be working with the **fv3-bundle**. This is a packaged `CMake `_ build system that includes all the GitHub repositories needed to run the Finite Volume Cubed Sphere (FV3) code within JEDI, including **OOPS**, **IODA**, **UFO**, **CRTM**, and **FV3-JEDI** itself. Since accessing your AWS node is something that you will do every day, the instructions on how to do so are posted on the main page for this Academy. .. pull-quote:: So, please :doc:`Follow steps 1-4 on the "Accessing AWS" page to log in to your AWS instance, set up your environment, build, and test fv3-bundle <../aws>` (for the first time!) A few comments are in order about the build process (Step 4), repeated here for reference: .. code:: bash cd ~/jedi/build rm -rf * ecbuild --build=Release ../src/fv3-bundle make update make -j4 First, note that the build is done in a separate directory from the source code. This is important; it avoids conflicts and allows you to easily clean the build directory when you want to do a fresh build. Second, the ECMWF package :code:`ecbuild` is used to configure the build before running make. ecbuild consists of a collection of `CMake `_ macros and modules that facilitate the build process by finding and linking the appropriate compilers and libraries. Third, note that the directory passed to ecbuild is a bundle, in this case :code:`fv3-bundle`. Have a look at the contents of that directory, :code:`../src/fv3-bundle`. Most of the subdirectories there are individual :code:`git` repositories that :code:`ecbuild` has retrieved from GitHub during previous builds. We have included these here to avoid the need for downloading large data files but if you wish, you can see how ecbuild works by deleting one of the directories, such as :code:`ufo` and then seeing how :code:`ecbuild` will do a fresh clone if it is not there (hint: we recommend that you do not re-clone :code:`crtm`, :code:`fv3-jedi` or :code:`ioda` because these have many files on git-LFS that take a while to download). You can look in the file :code:`../src/fv3-bundle/CMakeLists.txt` to see the list of repositories that are involved in building fv3-bundle. Look for statements such as this that add repositories from GithHub, specifying the branch. .. code:: bash ecbuild_bundle( PROJECT oops GIT "https://github.com/JCSDA/oops.git" BRANCH feature/academy UPDATE ) If a repository is not there, :code:`ecbuild` will clone it. Then, it will check out the specified branch. The :code:`UPDATE` argument to :code:`ecbuild_bundle` tells ecbuild to pull the latest changes from GitHub before compiling. This is done by executing the :code:`make update` command in the build sequence above. After you have successfully built and tested fv3-bundle, please return to this page and proceed to Step 5. Step 5: Run an Application ----------------------------- We have prepared a 3DVar application for you to run. To execute the code, enter these commands .. code:: bash cd ~/academy ./run.bash hyb-3dvar 2>&1 | tee run.log Now take a moment to investigate what you just did. Look through the :code:`run.bash` file to see what commands were executed. Look through the :code:`run.log` file to see the output. Note in particular the State and Increment print sections, the Cost function minimization, and the timiing statistics. Step 6: Visualize the increment ----------------------------------- To visualize the increment we just generated with our 3D Var application, we will use a software product developed by NASA called `Panoply `_. Panoply is installed on your AWS node but it is a GUI that runs slowly across the internet. So, we recommend that you run panoply from your local workstation. As you recall from Step 2, the :code:`aws_init` script that was provided to you is used to initialize your AWS connection, including setting the :code:`AWSIP` environment variable to proper value for your group. It also defines other useful aliases. So, to download the the increment from AWS to your workstation enter .. code:: bash cd source aws_init.csh # or .sh if using bash get-increment This should download a NetCDF file with a name something like: :code:`hyb-3dvar-inc.20180415_000000z.nc4`. Now fire up Panoply by running this script: .. code:: bash panoply.sh In the finder screen, navigate to the directory that contains the increment file, select it, and then select **Open**. Select **Temperature** from the list of data sets. Then, in the upper left corner, select **Create Plot**. A dialog box will come up: you can leave all the settings at their default values and select **Create**. Now you should be able to see the temperature increment. Note the color table on the bottom and the gray outlines that mark the edges of the cubed sphere. This shows a level at the top of the domain (level 1) by default. To see a more representative level lower down in the atmosphere, go to the **Vertical level** menu item at the bottom and select level **50**. Now play around a bit with Panoply. Explore it's options. Try plotting out zonal averages instead of a map. Navigate to the Map tabe and try a different projection. Go back to the original window and create a new plot with a different field. Try comibine plots. Step 7: Build and Run JEDI in the Singularity Container ---------------------------------------------------------- Thus far we have been working with the JEDI environment modules, which are a convenient way to provide the JEDI software dependencies on HPC and cloud platforms. However, we also provide software containers that include all of the JEDI dependencies. These are a useful way to get up and running with JEDI quickly on any platform. In this exercise we will use the `Singulary container `_. Alternatively, the JEDI team also provides a `Charliecloud container `_ for use on systems where Singularity is not available. To obtain the JEDI Singularity container from Singularity Hub and enter it, execute the following commands: .. code:: bash singularity pull shub://JCSDA/singularity singularity shell -e singularity_latest.sif The first command will take a few minutes so please be patient. After executing the second command, you will be "inside the container" - an isolated user environment that is distinct from the host environment. Look around a bit. Note that you can still see the files in your home directory. And, if you enter :code:`whoami` you see that you are still you - namely the user :code:`ubuntu`. Everything may look similar. That is, until you enter a command like this: .. code:: bash ls /usr/local/lib This will give a very different result depending on whether you are inside or outside the container. From inside the container, you will see all the JEDI dependencies pre-installed and ready to use. Now try to build and run JEDI from within the container. In this case, the software versions inside and outside the container (in the environment module) are the same so you *could* use the same build directory. But, this is not always the case so it is good practice to use a different build directory (or wipe the other one clean). So, to build JEDI you can enter these commands. .. code:: bash cd ~/jedi/build_sin ecbuild --build=Release ../src/fv3-bundle make -j4 Note that we access the same source code both inside and outside the container so there is no need to clone or pull the repositories again. Now you can run :code:`ctest` from the build directory and/or you can run the application again by entering the following: .. code:: bash cd ~/academy ./run-sin.bash hyb-3dvar 2>&1 | tee run-sin.log Feel free to compare the :code:`run-sin.bash` script for Singularity and the :code:`run.bash` that we used to run the application outside the container. The main difference is that the Singularity container does not use modules. Step 8: Explore the JEDI GitHub Repos --------------------------------------- From your AWS account, enter this: .. code:: bash echo $MYGROUP $MYPW This is your group's username and password for GitHub. In a web browser, navigate to `https://github.com `_ and use these credentials to log in. As an Academy `padawan `_ you have read access to many of the JCSDA repositories. You can get acquainted with **oops**, **ufo**, **ioda**, **crtm**, and **fv3-jedi** or others that pique your interest, such as **ioda-converters**. Find a JEDI master near you to ask questions. If we have time, we will show you how to install the ZenHub extension to see the project boards. .... Contact Mark Miesch (miesch@ucar.edu) or any other members of the JEDI core team if you have any problems or questions.