Expectations for Developers

Testing Expectations

• Run the full RT suite, preferably on Ursa, Hercules, or Derecho (unless changes are specific to a different machine). Developers often choose to add the -e or -r options, too, to expedite testing:

  ./rt.sh -a <account> -e
  

• Commit and push the updated test_changes.list file.
• Commit and push the log file (RegressionTest_<machine>.log). In the case of baseline changes, this log file will show failures, but this is fine as long as the failures are reasonable (e.g., a comparison failure because a test has baseline updates).
• Exceptions: Regression testing is not required for text-only changes, such as documentation updates, or for updates to the CI, which do not touch the model code.

Important

Do NOT run with the -c option when producing the log and test_changes.list file that you plan to upload. Code managers need to see which tests (if any) will fail using the current baselines. Using the -c option will produce passes for baseline-changing tests, and the tests will not be added to test_changes.list.

Code managers will recreate baselines when necessary as part of the PR merge process. Developers are welcome to test with -c and run against their newly created baselines using -m as part of their own testing process to ensure reproducibility, but the logs produced from these steps should not be the ones uploaded to the PR.

Review Expectations

• Before we can schedule a PR for testing, all subcomponent PRs need to be approved.