Recipe Test Workflow (RTW) prototype #2786
Description
Introduction
- This issue is a summary issue for the development of the Recipe Test Workflow (RTW) prototype
- The requirements for the RTW are available at Recipe test workflow proposal #2723
- Please get in touch with one of the current developers / reviewers (@ehogan, @alistairsellar, @mo-gill and @chrisbillowsMO) if you would like to collaborate / contribute (thanks to previous contributors @KatherineTomkins, @mo-tgeddes and @Jon-Lillis!) 👍
How we are working
- We created a main feature branch for the prototype (called recipe_test_workflow_prototype), and sub-feature branches are used for individual development tasks (see the discussion we had with V)
- We created a
Recipe Test Workflow (RTW)label, which we add to every issue and PR - We open an issue for each individual development task, stating
Part of #2786in the first comment of the issue - We create a draft PR and link to the issue for the individual development task as normal
- We ensure the RTW checks pass before requesting a reviewer
- Documentation exists but is currently separate from the main documentation (it is located in the
recipe_test_workflowdirectory) and:- we hope to add the documentation to main documentation before the main feature branch is merged
- we update the documentation for each individual development task, as appropriate
Completed Issues / PRs
| Task | Issue | PR | Assignee | Merge date |
|---|---|---|---|---|
| Create a feature branch in the ESMValTool repository, add initial commit | N/A | #2787 | @mo-tgeddes | 2022-09-02 |
| Update configure task to use the correct paths | #2814 | #2815 | @KatherineTomkins | 2022-09-29 |
Update the install_cold task to install the latest version of ESMValTool and ESMValCore |
#2821 | #2803 | @KatherineTomkins | 2022-10-03 |
| Add compare task to RTW | #2819 | #2820 | @mo-tgeddes | 2022-10-04 |
| Add workflow checks to RTW | #2857 | #2859 | @mo-tgeddes | 2022-10-17 |
| Add two groups of two test recipes | #2816 | #2822 | @KatherineTomkins | 2022-10-20 |
| Upgrade the RTW to work with ESMValTool v2.8.0 | #3168 | #3169 | @ehogan | 2023-06-26 |
| Implement RTW review comments | #3245 | #2863 | @mo-tgeddes | 2023-06-28 |
| Separate install cold and clone task for RTW | #2870 | #2871 | @mo-tgeddes | 2023-06-28 |
| Update the RTW to work at the MO | #3385 | #3386 | @ehogan | 2023-10-18 |
| Add JASMIN as an additional site to the RTW | #3060 | #3302 | @ehogan | 2023-10-20 |
| Simplify configure task in recipe test workflow | #3398 | #3394 | @alistairsellar | 2023-10-26 |
| Use Docker containers in the RTW | #3246 | #3399 | @alistairsellar | 2023-12-12 |
| Run RTW prototype daily with local KGO | #3493 | #3556 | @alistairsellar | 2024-03-27 |
| Add imagehash package as an ESMValTool dependency | #3436 | #3557 | @alistairsellar | 2024-04-03 |
| Set resources for RTW recipes individually | #3561 | #3562 | @alistairsellar | 2024-04-09 |
| RTW should support recipes in subdirectories | #3566 | #3567 | @alistairsellar | 2024-05-09 |
| Update Cylc version in lock file for RTW | #3593 | #3596 | @mo-gill | 2024-05-20 |
Store RTW files in share directory in installed workflow |
#3434 | #3600 | @mo-gill | 2024-05-21 |
Add recipe recipe_consecdrydays to RTW |
#3608 | #3609 | @mo-gill | 2024-05-22 |
| Validate RTW user config file | #3392 | #3615 | @chrisbillowsMO | 2024-05-24 |
Remove imagehash install from RTW |
#3621 | #3622 | @mo-gill | 2024-05-29 |
| Add Rose and Cylc to testing dependencies for RTW | #3623 | #3625 | @ehogan | 2024-05-30 |
Add recipe_ocean_multimap to RTW |
#3626 | #3627 | @chrisbillowsMO | 2024-06-10 |
| Replace the MO-specific URL in the RTW with a public URL | #3654 | #3657 | @mo-gill | 2024-06-20 |
Include verbose output from compare.py in RTW |
#3669 | #3671 | @ehogan | 2024-06-21 |
| Add code owners for the RTW | #3667 | #3668 | @ehogan | 2024-06-21 |
Add rose macro -V command to rtw-tests.yml |
#3632 | #3649 | @chrisbillowsMO | 2024-06-24 |
| Add how to add a recipe documentation | #3074 | #3614 | @mo-gill | 2024-06-25 |
Add the recipe_ocean_amoc.yml recipe to the RTW |
#3663 | #3664 | @ehogan | 2024-06-25 |
Review rose-app.conf files in RTW |
#3571 | #3601 | @mo-gill | 2024-06-25 |
| Merge the RTW documentation with the ESMValTool documentation | #3253 | #3684 | @ehogan | 2024-06-26 |
| Final sanity check and reorganisation of RTW | #3685 | #3707 | @ehogan | 2024-07-02 |
| Remove Rose and Cylc from the ESMValTool environment | #3723 | #3816 | @ehogan | 2024-11-20 |
Work to be done after the prototype is merged
| Task | Issue | PR | Assignee |
|---|---|---|---|
| Add remaining recipes to RTW | |||
| Add a weekly tests cycle to RTW | #3570 | ||
| Find compare.py more robustly within container | #3437 | (see ESMValGroup/ESMValCore#2381) | |
| compare.py should fail cleanly when imagehash not available | #3433 | ||
| Investigate the use of containers in the RTW at the MO | #3579 | ||
| Get RTW to check for changes before running tests | #2872 | ||
| Add a mechanism for notification emails / opening new issues via Github API | |||
| Decide how to manage and store KGOs | |||
| Decide whether to resolve issues with recipes that are not reproducible | |||
| A question about fonts for R diagnostics | #3396 | ||
| Flag failing recipes and don't re-run until they have been changed or the repo has been updated |
Addresses the user stories:
As a release manager
I want to know if changes to main in ESMValTool or Core have caused the recipes to fail to run
So that the API can be fixed before a release
As a release manager
I want to know if changes to ESMValTool or Core have changed the output the recipes
So that I can make fixes or changes before a release
As a recipe owner
I want to know if changes to ESMValTool or Core have changed the output for my recipe
So that I can make fixes or changes before a release
As a member of the ESMValTool or Core development community
I want to run the recipe run workflow efficiently
So that costs are not excessive
As a member of the ESMValTool or Core development community
I want to know about tools available for regression testing
So that I am aware of my responsibilities as a developer
As a member of the ESMValTool or Core development community
I want to know if changes to main in ESMValTool or Core have caused the recipes to fail to run
So that I can fix issues in the API