######################################################### Monday A: Getting Started with AWS, Singularity, and JEDI ######################################################### .. _day1a-goals: In this activity you will: - split into groups and be assigned a group number that you will use for the rest of the week - learn how to access AWS through the academy Jupyterlab - learn how use a JEDI development container - build JEDI from source - run the unit tests for fv3-bundle - explore the JEDI source code and directory structure Step 1: Access your AWS Instance -------------------------------- A JEDI master will direct you to form 20 groups of 2-3 people each. Each group has their own cloud computing instance on which you will build and run JEDI. Once you have a group number, you can log into your compute node :doc:`as described here <../aws>`. These AWS access instruction are set apart from these activity instructions because you will repeat them every afternoon when you do the activities. Step 2: Explore the JupyterLab Interface ---------------------------------------- Take a few moments to familiarize yourself with the web interface provided by JupyterLab. Select the terminal tab in the main window to access the linux command line. Navigate the directory tree with the menu on the left. Find an example in the jedi tree of the following file types and see what happens when you select it - an image file - a python file (``*.py``) - a yaml file (``*.yml``) - a text file (``*.txt``) - a C++ or Fortran file Go to the python console window (likely labeled **Console 1**) and do a calculation: estimate how many seconds you have left of the Academy (and rejoice in the result!). *Hint:* ````-```` *executes a particular cell in the Jupyter notebook; see the Run menu for more options*. Open a new ssh terminal by selecting the artist's palette on the left and scrolling down to ``Terminal``-``New Terminal``. Switch to a dark background for your terminal window if you wish. Step 3: Explore the JEDI source code ------------------------------------ You may have noticed during Step 2 that the JEDI source code is already there, in the directory ``~/jedi/src/fv3-bundle``. The JEDI code is organized into multiple git repositories, each with its own web interface on GitHub. You may recognize some of the repositority names from today's lectures - names like ``oops``, ``ufo``, ``ioda``, ``saber``, and ``fv3-jedi``. If you don't recognize these yet, you will by the end of the week. The name of the ``fv3-bundle`` directory is significant. **JEDI bundles** are convenient ways for users to build all of the JEDI components that are needed for a particular application. As its name suggests, ``fv3-bundle`` includes the source code for all the repositories require to run the `FV3 model `_ within JEDI, with the accompanying data assimilation capabilities of core JEDI repositories including the Object Oriented Prediction System (OOPS), the Interface for Observational Data Assimilation (IODA), the Unified Forward Operator (UFO) and the System-Agnostic Background Error Representation (SABER). To see how this list of repositories is specified, navigate to the ``~/jedi/src/fv3-bundle`` directory and view the file ``CMakelists.txt``. Now explore some of the repositories themselves. Most have a ``src`` directory where the code is held as well as a ``test`` directory that mimics the structure of the ``src`` directory to test every class, function, module, and subroutine. An exception is ``oops`` which, as the highest-level organizational component is organized a bit differently. Here the QG and Lorentz 95 toy models have their own source and test directories (``oops/qg/test`` and ``oops/l95/test`` respectively). Navigate to the ``oops/src/oops/interface`` directory to behold some of the generic C++ templates that set JEDI apart from other DA systems. If you wish, you can also explore the JEDI repositories on GitHub. As part of the setup of your AWS/JupyterLab compute instance, your GitHub credentials were also configured. This allows ecbuild to pull from GitHub when you run ``make update`` in Step 6 below. Each group has its own GitHub user name and password. To see what they are, just enter this from the linux command line: .. code:: echo $MYGROUP echo $MYPW You can open a separate browser window and copy and paste these credentials to log into `Github `. Go ahead and give it a try. As an Academy *padawan*, you should then be able to explore many of the JEDI repositories on GitHub. Step 4: Download and enter the JEDI Container --------------------------------------------- You're not quite ready yet to build JEDI. Though the source code is already there, the JEDI dependencies are not. In order to build JEDI, you'll need software packages such as ``ecbuild``, ``netcdf``, and ``Eigen``, as well as MPI. These dependencies are provided in a software **container**. For an overview of what software containers are and why we use them, see the `JEDI documentation `_. Various JEDI containers exist with different plaforms (Singularity, Charliecloud, and Docker), different compilers (gnu, clang, and intel), and different MPI implementations (Openmpi, mpich, Intel MPI). For the Academy we'll be using the gnu-openmpi Singularity container, which you can obtain by executing the following commands: .. code:: cd $HOME singularity pull library://jcsda/public/jedi-gnu-openmpi-dev If the pull was successful, you should see a new file in your current directory with the name ``jedi-gnu-openmpi-dev_latest.sif``. If you wish, you can verify that the container came from JCSDA by entering: .. code:: singularity verify jedi-gnu-openmpi-dev_latest.sif Now you can *enter the container* with the following command: .. code:: singularity shell -e jedi-gnu-openmpi-dev_latest.sif To exit the container at any time, simply enter .. code:: exit It is worth noting here that this JEDI singularity container is public - as long as you have access to singularity, you can download it. You do not need to be on AWS. Step 5: Get to know the Container --------------------------------- When you ran the ``singularity shell`` command at the end of Step 1, you entered a new world, or at least a new computing environment. Take a moment to explore it. First, notice that you are in the same directory as before: .. code:: echo $PWD So, things may look the same, though your command line prompt has likely changed. And, you can see that your username is the same as before and your home directory has not changed: .. code:: whoami echo $HOME cd ~ ls You are still the same person. And, more importantly from a system administrator's perspective, you still have the same access permissions that you did outside of the container. You can still see all the files in your home directory. And, you can still edit them and create new files (give it a try). But things have indeed changed. Enter this: .. code:: lsb_release --all This tells you that you are now running an ubuntu 18.04 operating system, regardless of what host computer you are on and what operating system it has. Furthermore, take a look at some of the system directories such as: .. code:: ls /usr/local/lib There you will see a host of JEDI dependencies, such as netcdf, lapack, and eckit, that may not be installed on your host system. Thus, singularity provides its own version of system directories such as ``/usr`` but shares other directories with the host system, such as ``$HOME``. If you're familiar with any of these libraries, you can run some commands, for example: .. code:: nc-config --all Step 6: Build and Test JEDI --------------------------- Now you're ready to build JEDI. To do so, go to the build directory and run the following commands (make sure you do this from *inside* the container or else you're likely to get a message that ``ecbuild`` is not found): .. code:: cd ~/jedi/build ecbuild ../src/fv3-bundle make update The ``ecbuild`` command uses ``CMake`` to comfigure the Makefiles for each of the repositories. Furthermore, if any of the repositories do not exist yet on your system, ``ecbuild`` will clone them from GitHub. If they do exist, the ``make update`` command will pull the latest commits from GitHub to ensure that the code is up to date. Now you're ready to actually compile ``fv3-bundle``: .. code:: make -j4 It will take some time for the code to compile. When it is finished, you can run the suite of unit tests for ``fv3-bundle`` by entering: .. code:: ctest Both the compilation and the unit testing will take some time. While you're waiting you can open a new ssh or browser window and return to Step 3 to contine exploring the JEDI source code. For further tips on working with ``ecbuild`` and ``ctest`` see the JEDI Documentation on `building `_ and `testing `_. When the tests complete, you can view the test log as follows (starting from the ``~/jedi/build`` directory): .. code:: cd Testing/Temporary vi LastTest.log .. Note:: If you selected files from the JupyterLab menu then this creates hidden files that can cause failures in the coding norms tests, for example ``oops_coding_norms``. You can ignore these if you wish. Or, if you want the tests to pass again you can go to the directory in question and remove the jupyter notebook checkpoint files: .. code:: rm -rf `find -type d -name .ipynb_checkpoints` .... Contact Mark Miesch (miesch@ucar.edu) or any other members of the JEDI core team if you have any problems or questions.