Acessing Amazon Web Services (AWS)¶
Step 1: Acquire AWS Access Files¶
A JEDI Master (Mark) will provide you with three files:
academy-virginia.pemaws_init.shaws_init.csh
The first contains an ssh key that is necessary to log into AWS instances. The other two files are shell scripts that you will use to set up your AWS access. Both have an identical function and you will use one or the other depending on whether you use the bash shell or some variant of the C shell (e,g. csh or tcsh).
The first file, academy-virginia.pem will be used throughout the week. However, you will need a new aws_init initialization script at the start of each day. The reason for this is because we wish to use our AWS resources efficiently. At the end of each day, we will halt our fleet of AWS Elastic Compute Cloud (EC2) instances because we won’t be using them overnight. The disk content will be saved (so you can pick up where you left off the previous day), but the servers will be shut down. When we restart them the next morning they will have different ip addresses. The new script that will be provided each day will contain the ip addresses that are valid for that day.
In order for the ssh key file to work, you will have to make sure that you disable read permission for anyone other than the user (if you don’t do this, ssh may throw an error):
chmod 400 academy-virginia.pem
We will have 20 groups, with 2 padawans per group. Each group has their own computer in the computer lab, their own AWS compute node, and their own GitHub account. The group numbers are assigned in the table below.
It is important to know your group number
Making a mistake here will mean that you will be working in the work space of another group.
| Group | Machine | GitHub user name |
|---|---|---|
| 1 | 112 | jedi-ac01 |
| 2 | 114 | jedi-ac02 |
| 3 | 116 | jedi-ac03 |
| 4 | 118 | jedi-ac04 |
| 5 | 120 | jedi-ac05 |
| 6 | 122 | jedi-ac06 |
| 7 | 124 | jedi-ac07 |
| 8 | 126 | jedi-ac08 |
| 9 | 128 | jedi-ac09 |
| 10 | 130 | jedi-ac10 |
| 11 | 132 | jedi-ac11 |
| 12 | 134 | jedi-ac12 |
| 13 | 136 | jedi-ac13 |
| 14 | 138 | jedi-ac14 |
| 15 | 140 | jedi-ac15 |
| 16 | 142 | jedi-ac16 |
| 17 | 144 | jedi-ac17 |
| 18 | 146 | jedi-ac18 |
| 19 | 148 | jedi-ac19 |
| 20 | 150 | jedi-ac20 |
Step 2: Log in to AWS¶
If you use bash, enter this from a terminal command line:
source aws_init.sh # bash
Or, alternatively, if you use tcsh or csh, enter
source aws_init.csh # tcsh, csh
The script will prompt you for your group number; enter it as requested and hit return.
Each of these scripts does two things. First, it sets the environment variable AWSIP to be equal to the ip address of your group’s EC2 instance (compute node). Second, it defines the alias ssh-aws, which you can use to log in to your group’s EC2 instance. So, to log into your group’s EC2 instance, just enter:
ssh-aws
You will likely get a warning asking if you are sure you want to continue connecting. Enter yes to add this ip address to ssh’s list of known hosts.
Step 3: Initialize Your AWS Environment¶
After you successfully log in to your group’s EC2 instance, source the startup script in the home directory - be prepared to enter your group number at the prompt:
source startup.sh
The most important function of this script is to set up the GitHub login credentials for your group so you can access the JEDI code repositories without being prompted for a user name and password. Be sure you enter the correct group number (you can run it again if necessary) because this will allow you to participate in team exercises on GitHub such as pushes, pull requests, and code reviews. The script also sets up a few other environment variables that you will need to compile and run JEDI.
Next, you will need to load the jedi module. This contains the compilers, the MPI library, and the other software dependencies that you will need to build and run JEDI. The JEDI test and application nodes have a number of modules available but the academy node only has a version of the code compiled with version 7.4 of the gnu compiler suite (gcc, g++, gfortran) and version 3.2.1 of OpenMPI. To load this module, enter:
module load jedi/gnu-openmpi
This is really a meta-module that loads a number of JEDI dependencies. To see a list of the software you now have access to, enter
module list
Step 4: Build and Run JEDI¶
Now you are ready to build JEDI. The source code for the FV3 bundle is in the subdirectory jedi/src/fv3-bundle but you will compile and run it in the build directory jedi/build. To do so, enter the following commands
cd ~/jedi/build
rm -rf *
ecbuild --build=Release ../src/fv3-bundle
make update
make -j4
Each step may take a few minutes so please be patient. The rm -rf * step is not necessary the first time you build the code but it is good practice to clean the build directory before you start a new build. The --build=Release option is not necessary - it may be omitted - but if there, it tells ecbuild to optimize the code so it runs more efficiently. The make update step will pull the latest code changes from the GitHub repositories.
Now you are ready to run JEDI. You can start by running the test suite as follows:
ctest
You can also run other applications as described elsewhere.
Contact Mark Miesch (miesch@ucar.edu) if you have any problems or questions.