Running operational requirements tests

What are the Operational Requirements Tests (ORT)?

• A suite of tests
• Each test addresses a single requirement that a new implementation should satisfy:
  • Thread (THR) reproducibility: varying the number of threads produces the same results
  • MPI process (MPI) reproducibility: varying the number of MPI tasks produces the same results
  • Domain decomposition (DCP) reproducibility: varying the tile layout of FV3 produces the same results
  • Restart (RST) reproducibility: restarting produces the same results
  • 64/32 (BIT): changing to double/single precision can be compiled and simulation can be run to completion
  • Debug (DBG): can be compiled and simulation can be run to completion in debug mode
  • Diagnostic reproducibility test (FHZ)

Why ORTs?

• The purpose of running ORTs is to satisfy the operational implementation requirement.
• Currently, regression tests conduct unit tests only for 'control' test
• Bugs that break the unit tests and are found later during the model implementation stage could be very difficult to debug
  • Identify problems earlier in the development process
• Developers ensure their implementations pass unit tests before requesting a Pull Reqeust

Procedure to Run ORTs

For the most up-to-date information on running ORTs, see the WM User's Guide ORT documentation.

• Developers finish code changes and do their own tests
• Pick a test case under ufs-weather-model/tests/tests that is relevant to their code changes
• Modify or add a test case if no test exists that can test their code changes
• $ cd ufs-weather-model/tests
• $ ./opnReqTest -n <test-name>
• Check and commit the unit test report/logfile to the developer's branch
• Make a pull request

Examples (using fv3_ccpp_control as <test-name>)

• To run all tests: $ ./opnReqTest -n fv3_ccpp_control
• To run THR test only: $ ./opnReqTest -n fv3_ccpp_control -c thr
• To run MPI and DCP tests only: $ ./opnReqTest -n fv3_ccpp_control -c mpi,dcp
• To run DBG and RST tests only: $ ./opnReqTest -n fv3_ccpp_control -c dbg,rst

Command Line Interface

• $ ./opnReqTest -h: displays usage of shell script
• $ ./opnReqTest -n <test-name>: runs all unit tests for test-name
• $ ./opnReqTest -n <test-name> -c <test-case>: runs a specific unit test case (thr, mpi, dcp, rst, bit, dbg, fhz)
• The full set of option flags is:
  • -a: specify HPC to use for batch job
  • -n: specify
  • -c: specify ; defaults to all test-cases: thr,mpi,dcp,rst,bit,dbg,fhz; comma-separated list of any combination of std,thr,mpi,dcp,rst,bit,dbg,fhz
  • -b: test bit reproducibility; compare against baseline
  • -d: test debug reproducibility; compare against baseline
  • -s: test reproducibility for std; compare against baseline
  • -x: skip compilation
  • -z: skip running (i.e., only compiles)
  • -k: keep run directory
  • -e: use ecFlow workflow manager

Input Files

• utest.bld
  • Specify required build options in this format: <test-name> | <build_opt>
  • <build_opt> can be WW3=Y, CCPP=Y, etc.
• <test-name>, <INPUT_NML>, <FV3_RUN>: all similar to those in regression tests

Output Files

• UnitTests_<machine>.<compiler>.log: unit test logfile (commit)
• Compile_ut_<machine>.<compiler>.log: compile logfile (do not commit)
• ufs-weather-model/log_ut_<machine>.<compiler> directory (do not commit)

File Locations Summary

• utest: ufs-weather-model/tests
• utest.bld: ufs-weather-model/tests
• <test-name>: ufs-weather-model/tests/tests
• UnitTests_<machine>.<compiler>.log: ufs-weather-model/tests
• Compile_ut_<machine>.<compiler>.log: ufs-weather-model/tests
• log_ut_<machine>.<compiler> directory: ufs-weather-model/tests
• Run directory
  • On Hera: /scratch1/NCEPDEV/stmp2/$USER/FV3_UT/ut_*
  • On Orion: /work/noaa/stmp/$USER/stmp/$USER/FV3_UT/ut_*
  • On Dell P3: /gpfs/dell2/ptmp/$USER/FV3_UT/ut_*

Details on Unit Test Configuration

• THR: Number of threads = 2, Tasks per node /= 2
• MPI: JNPES /= 2, WRITE_GROUP = 2, WRTTASK_PER_GROUP = 12
• DCP: Swap INPES and JNPES
• RST: use restart files at middle of the run (SHOUR+FHMAX/2) and compare results at end of the run (SHOUR+FHMAX)
• BIT: able to compile and run to completion with different precision
• DBG: able to compile and run to completion in debug mode