Abstract Out Reproducibility Tests #3887
Description
@juliasloan25 has mentioned that ClimaCoupler currently lacks reproducibility test infrastructure, which would be useful for seeing how PRs in ClimaAtmos and other packages change our coupled simulations. There are some tests that compare simulations against reference solutions, but these errors are so large that they exhibit minimal changes even when the outputs of ClimaAtmos are significantly modified. Although it is helpful to have tests against reference solutions, reproducibility tests are more useful when the reference errors are large. In addition to ClimaCoupler, ClimaLand and other packages would also benefit from reproducibility tests.
In the process of abstracting out the reproducibility tests into a separate package, it would be good to make several updates:
- Generalize the infrastructure to work with other packages
- Make a generic API that can be called from any CI pipeline with minimal setup
- Add documentation and unit tests
- Only have CI jobs fail when errors are physically significant
- Allow users to specify parameters for rescaling errors to determine if they are physically significant
- Insignificant errors (e.g., roundoff errors from changing order of operations) should only be logged
- Consider moving all reproducibility tests into a single job per pipeline
- This would make our CI results less noisy, with jobs that run simulations only failing when they crash
- However, it would force users to spend time looking for artifacts of jobs that break reproducibility tests