Skip to content

Consolidate setting stream in creation of request file and variables list#380

Merged
NParsonsMO merged 3 commits intomainfrom
213-stream-in-request-file-and-variables-list
Mar 10, 2026
Merged

Consolidate setting stream in creation of request file and variables list#380
NParsonsMO merged 3 commits intomainfrom
213-stream-in-request-file-and-variables-list

Conversation

@NParsonsMO
Copy link
Collaborator

@NParsonsMO NParsonsMO commented Feb 23, 2026

Closes #213 .

PR creation checklist for the developer

  • The <issue_number> above ☝️ has been replaced with the issue number.
  • main has been selected as the base branch.
  • The feature branch name follows the format <issue_number>_<short_description_of_feature>.
  • The text of the PR title exactly matches with the text (not including the issue number) of the issue title.
  • Appropriate reviewers have been added to the PR (once it is ready for review).
  • The PR has been assigned to the developer(s).
  • The same labels as on the issue (except for the good first issue label) have been added to the PR.
  • The Climate Model Evaluation Workflow (CMEW) project has been added to the PR.
  • The appropriate milestone has been added to the PR.

Definition of Done for the developer

  • The change in this PR addresses the above issue / all acceptance criteria have been met.
  • The change in this PR follows the requirements in the wiki: Developer Guide (including copyrights).
  • The GitHub Actions workflow checks pass.
  • The tests run locally and pass (Note: the tests are not run by the GitHub Actions workflow, see wiki: Run the tests locally).
  • Updating the Rose metadata (select one of the following):
    • Rose metadata related to the change has been added or updated.
    • The change does not require Rose metadata to be added or updated.
  • Rendering the Rose metadata (select one of the following):
    • The Rose GUI shows the change as expected.
    • The change in this PR does not affect the Rose GUI.
  • Updating the tests (select one of the following):
    • Tests related to the change have been added or updated.
    • The change does not require tests to be added or updated.
  • Updating the user documentation (i.e. everything in the doc directory, including the Quick Start section; select one of the following):
    • The user documentation related to the change has been updated appropriately.
    • The change in this PR does not require the user documentation to be updated.
  • Rendering the user documentation (wiki: Build the documentation locally provides instructions; select one of the following):
    • The HTML pages show the change as expected.
    • The change in this PR does not affect the HTML pages.
  • Updating the API documentation (e.g. docstrings in Python modules; select one of the following):
    • The API documentation related to the change has been updated appropriately.
    • The change in this PR does not affect the API documentation.

PR creation checklist for the reviewer

  • The <issue_number> above ☝️ has been replaced with the issue number.
  • main has been selected as the base branch.
  • The feature branch name follows the format <issue_number>_<short_description_of_feature>.
  • The text of the PR title exactly matches with the text (not including the issue number) of the issue title.
  • Appropriate reviewers have been added to the PR (once it is ready for review).
  • The PR has been assigned to the developer(s).
  • The same labels as on the issue (except for the good first issue label) have been added to the PR.
  • The Climate Model Evaluation Workflow (CMEW) project has been added to the PR.
  • The appropriate milestone has been added to the PR.

Definition of Done for the reviewer

  • The change in this PR addresses the above issue / all acceptance criteria have been met.
  • The change in this PR follows the requirements in the wiki: Developer Guide (including copyrights).
  • The GitHub Actions workflow checks pass.
  • The tests run locally and pass (Note: the tests are not run by the GitHub Actions workflow, see wiki: Run the tests locally).
  • Updating the Rose metadata (select one of the following):
    • Rose metadata related to the change has been added or updated.
    • The change does not require Rose metadata to be added or updated.
  • Rendering the Rose metadata (select one of the following):
    • The Rose GUI shows the change as expected.
    • The change in this PR does not affect the Rose GUI.
  • Updating the tests (select one of the following):
    • Tests related to the change have been added or updated.
    • The change does not require tests to be added or updated.
  • Updating the user documentation (i.e. everything in the doc directory, including the Quick Start section; select one of the following):
    • The user documentation related to the change has been updated appropriately.
    • The change in this PR does not require the user documentation to be updated.
  • Rendering the user documentation (wiki: Build the documentation locally provides instructions; select one of the following):
    • The HTML pages show the change as expected.
    • The change in this PR does not affect the HTML pages.
  • Updating the API documentation (e.g. docstrings in Python modules; select one of the following):
    • The API documentation related to the change has been updated appropriately.
    • The change in this PR does not affect the API documentation.

Important

  • Remember to re-check the Definition of Done after making changes in response to a review.
  • The developer merges the PR.
  • Remember to use the format #<pull_request_number>: <pull_request_title> when writing the merge commit message for the pull request, so the pull request number is immediately visible on GitHub, regardless of the length of the pull request title.

@NParsonsMO NParsonsMO self-assigned this Feb 23, 2026
@NParsonsMO
Copy link
Collaborator Author

In the spirit of "lean" development, this just overwrites two hardcoded instances of the string "apm" and replaces them with references to a single environment variable.

I've put this variable into flow.cylc, as that's where everything else seems to go. I guess also, it might be easier to find in future than in the rose-app.conf[env] section?

In future we are going to want to have multiple streams, which this doesn't deliver.

One way to do this (in future) would be to create an interim variables.txt file without any streams, then use CDDS's "guess streams" feature to create a final variables.txt file.

@NParsonsMO NParsonsMO added standardise Anything related to CDDS configure Anything related to configuration technical debt Technical debt in CMEW climate model monitor Needed for the climate model monitor workflow labels Feb 23, 2026
@NParsonsMO NParsonsMO marked this pull request as ready for review February 23, 2026 13:54
@mo-nikosbaltas mo-nikosbaltas self-requested a review March 4, 2026 12:31
ROSE_TASK_APP = configure_standardise
START_YEAR = {{ START_YEAR }}
NUMBER_OF_YEARS = {{ NUMBER_OF_YEARS }}
STREAM_ID = "apm"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hard-coding STREAM_ID here is not acceptable unless this is not meant to be configurable. Best practice (if it is configurable by end-user) is to have a template variable in rose-suite.conf (STREAM_ID="apm") and then in flow.cylc should access it like: STREAM_ID = {{ STREAM_ID }}. If this is a site constant then add it in site/metoffice.cylc.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought that this PR was just to only hard code it in the one place (it had previously been two), while we awaited a decision on the shape of the future solution.

Not all variables will have the same stream, so I don't think it's user configurable, I think it's something where we will either need to create a lookup table, or create the request in two stages (CDDS has a guess stream function)..

I don't know which we're doing. Or if perhaps there is a different solution.

Copy link
Collaborator Author

@NParsonsMO NParsonsMO Mar 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@alistairsellar Does this meet the intended acceptance criteria? I think I focussed on "duplication",
image
But the line above the acceptance criteria implies that you had intended this to be a full solution to the problem of each variable potentially having a different stream.
image

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hrrm, sorry, the issue description was quite ambiguous because it had that implementation point, which contradicted the point above it: "(In the future the required streams will be inferred from the variables to be extracted and the MIP table). I think that the implementation point was only relevant to the bit in brackets.

My understanding is that the acceptance criteria are right, so I've tidied up the rest to clarify. This issue should not make the stream configurable. We probably don't ever want this to be user configurable, since the future is to infer streams from variables.

Copy link
Collaborator

@mo-nikosbaltas mo-nikosbaltas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Following updates on the issue #213 of this PR, ACs are satisfied. Tests run successfully so good to go.

@zmaalick zmaalick self-requested a review March 5, 2026 12:09
@NParsonsMO NParsonsMO merged commit 0a9cdaa into main Mar 10, 2026
3 checks passed
@NParsonsMO NParsonsMO deleted the 213-stream-in-request-file-and-variables-list branch March 10, 2026 12:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

climate model monitor Needed for the climate model monitor workflow configure Anything related to configuration standardise Anything related to CDDS technical debt Technical debt in CMEW

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Consolidate setting stream in creation of request file and variables list

4 participants