UFSCOMP Test Procedure

Test Procedure

This document aims to give brief description about the test procudure of UFSCOMP development.

Developer Responsiblity

1. Developer changes the code in her/his sandbox
2. Pushes all changes to the remote repository using seperate branches
   • Runs test suite and compare with beaseline
3. Submits PR with all hashes/tags linked with PR
   • Put name of UFSCOMP branch to PR
   • Put hashes of each component to the PR description if branch is not used
4. Put baseline directory into PR description and add extra information such as
   • Bit-to-bit
   • Round-off
   • namelist changes (as diff output)
   • etc.

Tester Responsiblity

1. Create same working environment with PR

   • Clone UFSCOMP

     git clone https://github.com/ESCOMP/UFSCOMP.git
     

   • Checkout UFSCOMP branch or use hash informations that are given in PR description.

     cd UFSCOMP
     git checkout app_cesm-cmeps
     

   • Checkout Externals

     ./manage_externals/checkout_externals
     

   • Merge PR with current tag/hash of cime

     cd cime
     git checkout nuopc-cmeps
     git merge [name of branch that points current PR]
     

2. Run test suite

   • CESM

     cd cime/scripts
     # Cheyenne:
     qcmd -l walltime=4:00:00 -- ./create_test --xml-testlist ../src/drivers/nuopc/cime_config/testdefs/testlist_drv.xml --xml-machine cheyenne --xml-category nuopc --compare may1intel19 --baseline-root /glade/p/cesmdata/cseg/nuopc_baselines
     

     Note: don't forget to point correct baseline for comparison

   • FV3GFS

     Currently, there is no test suite for FV3GFS application but creating new test and running it can be used to test PR

     cd cime/scripts
     # Cheyenne:
     ./create_newcase --compset UFS_S2S --res C384_t025 --case ufs.s2s.c384_t025.jan --driver nuopc --run-unsupported
     cd ufs.s2s.c384_t025.jan/
     ./case.setup
     ./xmlchange DOUT_S=FALSE
     ./xmlchange STOP_N=1
     ./xmlchange RUN_REFDATE=2012-01-01
     ./xmlchange RUN_STARTDATE=2012-01-01
     ./xmlchange JOB_WALLCLOCK_TIME=00:30:00
     qcmd -- ./case.build
     ./case.submit	  
     

     • Testing NUOPC and sub-set of MCT tests
     • Testing SMS / ERS / Threading tests (after threading works)
     • Testing both optimized and debug versions of both CESM and FV3GFS applications (it requires to introduce new indicator such as _D)

3. If it is necessary, create baseline under CESMDATA root

   • Tester generates the baseline
   • Only keep last three baseline

4. If tests fail

   • comment issue in PR
   • let developer fix the issue

5. If tests pass

   • merge PR (currently just for CIME)

6. Modify External and merge with UFSCOMP app-cesm-cmeps

   • tag it
   • update PR with new UFSCOMP app-cesm-cmeps tag or hash
   • update PR and add new basline information, if it is necessary