Thursday: Working with the JEDI Development Methodology¶
Introduction:¶
Today we will take a look at the JEDI development methodology, which is based on an agile development flow called “git-flow”. Click here for more information about git-flow. Today’s talks on Compatibility tools and Unit testing introduced some of the concepts, tools and methods we will use in this exercise. We will walk through one iteration of the git-flow process going from the submission of an issue, modification of source code, build and test, and the submission of a pull request (which would ultimately result in your code changes being merged into the code base).
The goal of this exercise is to learn about the software development process that we use with JEDI (as opposed to learning how to introduce new features into the code). Therefore, today we will just make small changes to YAML configuration files that you got familiar with in yesterday’s activity so that the focus can be on the process. We will be switching from a “running applications” mode to “developing software” so we will work in the source and build directories instead of the academy directory that you have been using up to this point.
In order to allow you to issue a pull request, the house elves have updated all of your repositories to point to a fork of the fv3-jedi repository (which is where we will make the changes). In order to check that the elves did the job correctly:
cd jedi/src/fv3-bundle/fv3-jedi
git remote -v
You should see a response that looks like this:
origin https://github.com/jedi-ac01/fv3-jedi.git (fetch)
origin https://github.com/jedi-ac01/fv3-jedi.git (push)
If you have “jcsda” in the path in place of “jedi-ac01”, ask for help to get your repository updated.
For this exercise every group will submit a pull request, and a partner group will review the pull request. Let’s pair up the groups by the following table:
| Partner Groups | |
|---|---|
| A | B |
| 01 | 02 |
| 03 | 04 |
| 05 | 06 |
| 07 | 08 |
| 09 | 10 |
| 11 | 12 |
| 13 | 14 |
| 15 | 16 |
| 17 | 18 |
| 19 | 20 |
For this exercise it is recommended that you open up two shell window on the AWS node, and also have the firefox browser open on the classroom workstation.
Step 1: Try out a build and test¶
In one of the shell windows, run the following:
cd jedi/build # this is the "build" directory
make update # make sure we are in sync with the github repos
make -j4 # build the system (compile code)
ctest # run all the unit tests
Experiment with ctest.
ctest -N # list out all tests without running them
ctest -V -R fv3 # run all of the FV3 tests (subset of tess)
ctest -V -R fv3jedi_hyb3dvar_gfs # run the hybrid 3dvar gfs test (single test)
Step 2: Submit an issue¶
Open up the firefox browser, and log into github.com as was done in Monday’s activity. On your class workstation run:
echo $MYGROUP $MYPW
to get your user and password for github.
Once signed into github, go to the link: https://github.com/jedi-ac01/fv3-jedi.git. Create an issue for your parter group to resolve by doing the following:
- Click on the ZenHub tab near the top in the center
- Click on the New Issue button near the top on the right
- Fill in the Issue Title, and optionally the comment section
- Click on Assignees and select your partner group
- You can shortcut finding your partner group by typing “jedi-ac” in the search box
- This should transform the list into the academy logins and you can then select your partner group’s login
- A check mark should appear next to the selected login
- Close the selection pop-up
- Click on the Submit New Issue button toward the bottom on the right
- Close the issue pop-up window
The view should return to the ZenHub board and you can see the issue appear in the New Issues column.
Step 3: Create a feature branch¶
In the other shell window you opened up:
- Initialize git-flow
cd jedi/src/fv3-bundle/fv3-jedi # this is the "source" directory
git flow init -d
- Start a feature branch
git flow feature start acXX # fill in the XX with your group number
edit ../CMakeLists.txt # this is the CMakeLists.txt file in fv3-bundle
# Change the branch in the ecbuild_bundle command for fv3-jedi from develop to feature/acXX
- Check to make sure you are on your feature branch
git branch -v
# You should get a response that marks feature/acXX with an asterisk
It is advisable to run the “git branch -v” command occasionally as you go through this exercise to ensure you didn’t accidentally get switched to another branch (such as the develop branch).
Step 4: Make a YAML configuration change¶
Pick one of the changes you made in yesterday’s activity (perhaps a new filter) and put that into the source code in you local clone of the fv3-jedi repository. These instructions are to be run in the window that is cd’d into the jedi/src/fv3-bundle/fv3-jedi directory (which is your local clone).
- Select the change you want and put it into the source YAML file
edit test/testinput/hyb-3dvar_gfs.yaml
# replace 'edit' above with your favorite editor program
# This is an example for the hybrid 3dvar flow, you may want to
# change one of the other YAML files. The YAML files in the test/testinput
# directory should match the names of the YAML files from yesterday's
# activity in the conf directory.
# make the desired change and write the new YAML file
Step 5: Test your change¶
In the build directory (window in the jedi/build directory)
make -j4 # build
ctest # run all of the unit tests
ctest -V -R fv3jedi_hyb3dvar_gfs # OR run the test associated with the
# change you made
The FV3 test associated with the YAML file you changed may fail because of the change you made. The normal procedure would be to verify that the change is expected and valid, and update the reference files in the test that are being used to compare with the test output (which would make the test pass again). However, for the purpose of this activity just get to the point where the run completes without errors and any remaining test failure is due to minor differences such as numbers changing slightly or the log file messages changing.
Step 6: Commit and publish your change¶
Back in the window in the source directory (jedi/src/fv3-bundle/fv3-jedi), run the following:
git status # this should show the file you changed (in red)
git diff # this should show the lines you changed in the file
# once satisfied that git status and diff are consistent with what
# you changed, run the next set of commands
git add test/testinput # stages your change for the commit
git status # your file that you changed should appear in
# green now
git commit
# An edit session will start. You must add a comment in the first
# line for the commit to be taken. Do that and then save the file
# and quit the editor.
git flow feature publish acXX # this pushes your commit to the githup repo
Step 7: Create and submit a pull request¶
You should be able to view your feature branch on github now. Go to the browser and click on the Code tab. Then click on the Branch button (left side down a little ways from the top) and select your feature branch from the pull-down list. The Branch button text should change to indicate your feature branch is selected.
Over on the right side just below the row that the Branch button is in, click on the Compare button. This should open a new window that shows the differences between your feature branch and a “base branch”.
Warning
What’s supposed to happen is that “develop” should appear as the base branch, but the JCSDA repo develop branch is being used instead of the jedi-ac01 repo develop branch. This can be rectified by clicking on the “base repository” button (the one on the far left) and selecting “jedi-ac01/fv3-jedi” as the base repository. You may also need to click on the resulting “base” button and change the branch to “develop” as well.
You should see a window now that shows your commit, and the differences between your feature branch and the develop branch. If you are satisfied with the differences being shown, then click on the “Create Pull Request” button and a new window shows up. Fill in the Title window and optionally the comment window.
Click on the Reviewers button (to the right of the comment window) and assign your partner group as the reviewer. A long pull-down window will appear and you can shortcut by typing “jedi-ac” in the search window. Click on your partner group’s login name and a check mark should appear next to that name. Then close the pull-down window and your partner group should appear as the reviewer under the Reviewers button.
Scroll down underneath the comment window and click on the “Connect with an issue” button. A pull down window with a list of the ZenHub issues should appear. Select the issue that was assigned to you. This causes the issue to automatically track along with the progress of the pull request. When the pull request is under review, the issue moves to the Review column on the ZenHub board and when the pull request is merged the issue moves to the Closed column.
When satisfied with the pull request information you filled in, click on the “Create Pull Request” button. A pull request window will appear with all of the information you filled in. You should see the number by the Pull Requests tab increment up by one.
Go to the ZenHub board and notice how your issue and pull request are tied together.
Step 8: Review your partner group’s pull request¶
Go to the browser and click on the Pull Requests tab (near the top and third tab from the right).
Select your partner group’s pull request and a window with their information will appear. Click on the Files Changed button (near the top, rightmost tab). A window with the file differences appears. Look this over, make an assessment of the changes, then click on the Review Changes tab (near the top on the right). Three options appear with short explanations:
- Comment
- Approve
- Request changes
Select the appropriate button and then click on the Submit Review button.
Optional steps: Add another change to your pull request¶
You can repeat steps 4-6 and this will add new changes to your pull request. This is what is done if a reviewer requests changes and you implement new code to satisfy that request.
Try adding another change to the YAML configuration. After publishing (git flow feature publish command is run) your new commit will appear in the list shown in your pull request (under the Conversation tab), and the resulting differences will be shown under the Files Changed tab.
After publishing your new changes, you can ask your reviewer to start over by “dismissing” the initial review. Go to the pull request under the Conversation tab and you should see a symbol next to your reviewer’s name that looks like two arrows chasing each other in a circle. Click on this symbol and the initial review will be dismissed and the reviewer can then start over.
Contact Stephen Herbener (stephenh@ucar.edu) or any other members of the JEDI core team if you have any problems or questions.