Add a low resolution test to mimic GFSv17 cycling as much as possible #3617
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
JessicaMeixner-NOAA
requested review from
TerrenceMcGuinness-NOAA,
WalterKolczynski-NOAA and
DavidHuber-NOAA
as code owners
JessicaMeixner-NOAA
requested review from
guillaumevernieres and
CatherineThomas-NOAA
guillaumevernieres
previously approved these changes
Apr 25, 2025
CatherineThomas-NOAA
requested changes
Apr 28, 2025
There was a problem hiding this comment.
I think we need to add an account line in the main yaml, like in PR #3411. Otherwise, my test succeeded on WCOSS2 and all the tasks looks correct.
@CatherineThomas-NOAA - Thanks for catching this. I've added the account. |
CatherineThomas-NOAA
previously approved these changes
Apr 28, 2025
There was a problem hiding this comment.
Thanks @JessicaMeixner-NOAA! The account update works in my test. Approve.
|
@CatherineThomas-NOAA Would this experiment, with appropriate changes to the ensemble, be a good basis for working on the reduced ensemble members? |
|
@AndrewEichmann-NOAA - Yes, with changes to |
KateFriedman-NOAA
previously approved these changes
Apr 28, 2025
|
Took a look at the test output provided by @JessicaMeixner-NOAA on Hera and noticed that the final cycle metp jobs (18z offset from end) aren't running because their dependencies are set for a non-existent cycle: Will fire off CI on WCOSS2, look for the same behavior, and see what fix is needed. |
KateFriedman-NOAA
added
the
CI-Wcoss2-Ready
|
@KateFriedman-NOAA - The C96C48_hybatmaerosnowDA.yaml test does not have the extra met+ jobs, although it also has DO_METP=YES as well. Not sure why this test would have the extra met+ jobs that are different from C96C48_hybatmaerosnowDA |
emcbot
added
CI-Wcoss2-Building
|
CI Tests set up to run in /lfs/h2/emc/ptmp/emc.global/PR/PR_3617/RUNTESTS on WCOSS |
aerorahul
added
CI-Hera-Ready
emcbot
removed
the
CI-Hera-Building
|
Found a fix that resolves the gfs_metp dependency issue by making the offset for part of the dependency adjust based on WCOSS2 (BACKUPSYS-C) global-workflow> git diff dev/workflow/rocoto/gfs_tasks.py
diff --git a/dev/workflow/rocoto/gfs_tasks.py b/dev/workflow/rocoto/gfs_tasks.py
index 89ac5e6e8..3b0a99dfd 100644
--- a/dev/workflow/rocoto/gfs_tasks.py
+++ b/dev/workflow/rocoto/gfs_tasks.py
@@ -1938,7 +1938,10 @@ class GFSTasks(Tasks):
dep_dict = {'type': 'cycleexist', 'condition': 'not', 'offset': offset}
deps2.append(rocoto.add_dependency(dep_dict))
- offset = timedelta_to_HMS(-to_timedelta(f'{6*lookback}H'))
+ edate_gfs = self._base['EDATE']
+ edate_metp = edate_gfs.replace(hour=18)
+ edate_metp_diff = edate_metp - edate_gfs
+ offset = timedelta_to_HMS(-to_timedelta(f'{edate_metp_diff}H'))
dep_dict = {'type': 'task', 'name': f'{self.run}_arch_vrfy', 'offset': offset}
deps2.append(rocoto.add_dependency(dep_dict))
deps.append(rocoto.create_dependency(dep_condition='and', dep=deps2))I updated the xml for the The extended CI case on WCOSS2 is still running after I booted the gempak jobs that hit their walltimes (reran fine within time) and it should finish in the next couple hours. |
|
@KateFriedman-NOAA thanks! please let me know if there's any code you want me to add to this branch, etc. And thanks for finding and fixing the bug! I just thought it was a feature ... oops!? |
emcbot
added
CI-Hera-Failed
|
Experiment C96C48mx500_S2SW_cyc_gfs FAILED on Hera in Build# 2 in |
emcbot
added
CI-Hera-Failed
|
CI Failed on Hera in Build# 2 |
@JessicaMeixner-NOAA yes please add this fix into your branch, thanks! |
|
WCOSS2 CI completed successfully with the gfs_metp dependency fix in place locally. |
KateFriedman-NOAA
added
CI-Wcoss2-Passed
JessicaMeixner-NOAA
dismissed stale reviews from KateFriedman-NOAA, aerorahul, and CatherineThomas-NOAA
via
1e8236f
|
@KateFriedman-NOAA I just pushed the fix from you. Can you please check to make sure it was correctly put into the code? |
@JessicaMeixner-NOAA looks good, thanks for adding that to your branch! The CI on Hera and Gaea stalled from the same issue on WCOSS2 (gfs_metp dependencies wrong). The fix tested locally within the CI on WCOSS2 confirmed the fix works. Going to declare the CI on Hera and Gaea for this PR as successful. |
KateFriedman-NOAA
approved these changes
Apr 30, 2025
There was a problem hiding this comment.
Re-approving. Thanks for adding the gfs_metp dependency fix into this PR @JessicaMeixner-NOAA !
KateFriedman-NOAA
added
CI-Hera-Passed
tsga
added a commit
to tsga/global-workflow
that referenced
this pull request
May 1, 2025
* develop: Update GSI hash and GSI fix version to resolve bugs (NOAA-EMC#3626) Add missing marine DA files to archiving (NOAA-EMC#3596) Add a low resolution test to mimic GFSv17 cycling as much as possible (NOAA-EMC#3617) Add the setting to use the reject list for station t/q observations in GSI based soil DA (NOAA-EMC#3599) GitLab CI Framework for schedule PR cases and ctests on multi hosts (NOAA-EMC#3603) Avoid parallel restart I/O on WCOSS2 (NOAA-EMC#3615) Enables user toggling of GDASApp g-w ctests (NOAA-EMC#3587) COM variable updates for prep and some external downstream jobs (NOAA-EMC#3608) Remove MOS from system (NOAA-EMC#3612) Updates to enable soil DA (NOAA-EMC#3452) Unexport SHELLOPTS when running htar (NOAA-EMC#3601) Fix check for netcdf wave restart (NOAA-EMC#3594) Call err_chk/err_exit for fatal errors in post JJobs/ex-scripts (NOAA-EMC#3571) Remove support for Jet and S4 (NOAA-EMC#3572) Hotfix in GitLab pipline for Nightly (env MACHINE breaks build on head node) (NOAA-EMC#3578) [hotfix] Missed a path during merging develop (NOAA-EMC#3577) Prepare for ops readiness - part 1 (NOAA-EMC#3557) Update UFS weather-model to 20250328 hash (NOAA-EMC#3528) Fix SFS fcst config (NOAA-EMC#3574) Use err_chk in GDAS j-jobs (NOAA-EMC#3570) Perform compute builds on Gaea head nodes (NOAA-EMC#3560) Add initial capability to produce JEDI-based observation space summary stat files (NOAA-EMC#3471) Spread epos over more nodes on Hera to increase allocated memory (NOAA-EMC#3567) Create separate gists when multiple files are published on GitHub (NOAA-EMC#3551) Use err_chk in GSI J-Jobs and scripts (NOAA-EMC#3549) Add unified jinja obs list to marine DA (NOAA-EMC#3530) Save snow and aerosol analysis increments (and logs and YAMLs) every cycle (NOAA-EMC#3537) Add Dependencies to SFS Cleanup Job (NOAA-EMC#3559) Updates archiving to reflect current naming of marine anl output (NOAA-EMC#3541) Temporarily disable compute builds on C6 (NOAA-EMC#3558) Update gdas.cd hash to resolve msu prod_util failure (NOAA-EMC#3556) COMIN/COMOUT updates for enkf chgres and downstream product jobs (NOAA-EMC#3518) Call err_chk in forecast scripts for fatal errors (NOAA-EMC#3515) Add Rocoto Jobs for the Missing Products of GEFS (NOAA-EMC#3466) Download subset fix data with python script (NOAA-EMC#3400) Check that partition should be set (NOAA-EMC#3543) Rename wave output and refactor some wave scripts to use MPMD, and fix some bugzillas along the way (NOAA-EMC#3517) Add support for dual batch partitions on AWS NOAA-EMC#3483 Update CI build and run directories for GitLab Nightlies on C6 and added GitLab support on Hera (NOAA-EMC#3536) Hotfix path for CI in Jenkins on Gaea C6 to it's world-share path (NOAA-EMC#3532) Create single ocean grib2 product file (NOAA-EMC#3529) Scheduled Nightly CI/CD Pipeline Script in GitLab on Gaea C6 (NOAA-EMC#3493) make sure cold starts are handled correctly when DOIAU=YES (issue NOAA-EMC#3516) (NOAA-EMC#3520) Add check for DO_AERO_FCST before copying fv_tracer files (NOAA-EMC#3485) Use jinja templates instead of `@VARNAME@` in config files (NOAA-EMC#3411) Replace "status" (or comparable) with "err" in preparation for moving to err_chk/err_exit (NOAA-EMC#3507) Error in Java launch script for CI (NOAA-EMC#3465) Delete DATAROOT when running generate_workflows.sh (NOAA-EMC#3504) Fix 3244 garbled change (NOAA-EMC#3492) Enable ensemble archiving via Globus (NOAA-EMC#3479) Update MSU FIX_DIR paths (NOAA-EMC#3488) Updates for AOWCDA and hybatmaerosnowDA cases on Gaea C6 (NOAA-EMC#3487) Update GOCART path for GDAS/GFS/GCAFS implementations (NOAA-EMC#3455) Make RUN Variables Explicit in `config.resources` (NOAA-EMC#3478) Remove unused key from enkfgdas_earc_vrfy (NOAA-EMC#3473) Bug fix to the failing early cycle marine DA ensemble re-centering (NOAA-EMC#3454) Make marine LETKF optional (NOAA-EMC#3462) When sourcing for RUN=enkf*, use CASE_ENS (NOAA-EMC#3475) Updates for Gaea: verif-global tag, tracker tag, Fit2Obs tag, and C768 analysis resources (NOAA-EMC#3463) Update gefswave glo_025 mesh file with new mask (NOAA-EMC#3457) Update MSU glopara paths to new role-global space (NOAA-EMC#3443) Enable CI testing on AWS (NOAA-EMC#3459) Enable Gaea C5 Jenkins CI (NOAA-EMC#3447) Job reference removal from WMO product names (NOAA-EMC#3460) Turn off aerosol prognostic radiative feedback for GDAS NOAA-EMC#2926 (NOAA-EMC#3445) Add DO_GEMPAK check to postsnd subtask (NOAA-EMC#3451) Add a force option to setup_xml to ignore unwritable directories (NOAA-EMC#3448) Remove the eomg job (NOAA-EMC#3331) Migration to role account for Jenkins on Orion (NOAA-EMC#3440) Eliminate `_gfs`, `_gdas`, etc, variables and add necessary if blocks (NOAA-EMC#3420) Update workflow staging for sfcanl tiles and waveinit (NOAA-EMC#3429) Improve messaging to display clear warning when missing snogrb file (NOAA-EMC#3317) JEDI-based ensemble recentering and analysis calculation (NOAA-EMC#3312) Enable HPSS archiving on C5/6 (NOAA-EMC#3437) Check if HOMEDIR STMP and PTMP are writable (NOAA-EMC#3430) Update UFS_Utils and GFS-utils hashes to update Gaea support and ocean/ice post products (NOAA-EMC#3433) Enable C1152 forecasts on gaea C6 (NOAA-EMC#3438) Migration to role account for Jenkins on Hercules (NOAA-EMC#3423) Remove Direct Linking to COM from DATA for `extractvars` Job (NOAA-EMC#3379) Enable HPSS via Globus on Hercules and Orion Remove job name from product files & update GEMPAK module. (NOAA-EMC#3415) `link` instead of `copy` in staging jobs (NOAA-EMC#3410) Migrate CI Jenkins to role account on Hera (NOAA-EMC#3414) Add rocotorc documentation when using scrontab (NOAA-EMC#3417) Update jgdas atmos verfozn and verfrad with COMIN/COMOUT prefix instead of COM (NOAA-EMC#3342) Add configuration for empirically-corrected ozone parameters (NOAA-EMC#3386) Enable global-workflow to run C768C384 GSI on Gaea-C6 (NOAA-EMC#3412) Move logical checks into if blocks (NOAA-EMC#3339) Adding Jenkins CI to GaeaC6 using role account (NOAA-EMC#3389) Enable GDASApp g-w CI cases to run on wcoss2 (NOAA-EMC#3399) CI/CD Test on Gaea C5- And update config.gaea under ci/platform (NOAA-EMC#3280) Enable cycling support for Gaea C6 (NOAA-EMC#3323) Update enkf archive jobs to use COMIN/COMOUT (NOAA-EMC#3393) Copy marine ensemble output observation diags and spread (NOAA-EMC#3407) Ci testing on aws 2 (NOAA-EMC#3408) Disable METplus jobs on Hera (NOAA-EMC#3403) Add the mean EnKF soil increment to the deterministic member (NOAA-EMC#3295) Add mpich/8.1.19 to the WCOSS2 LD_LIBRARY_PATH for GDASApp jobs (NOAA-EMC#3396) Change order of RUNs (NOAA-EMC#3335) CI testing on aws (NOAA-EMC#3391) Rename Gulf of Mexico in bufr station list in GFSv17 (NOAA-EMC#3384) Enabling AWS CI/testing (NOAA-EMC#3383) Update issue templates to use new issue type field (NOAA-EMC#3369) Replace WAVECUR_DID variable with "rtofs" (NOAA-EMC#3337) Allow for C1152 ATM-Aero cycled DA to run on WCOSS2 (NOAA-EMC#3309) Remove Direct Linking to COM from DATA for `wavepostsbs` Job (NOAA-EMC#3303) Update jgdas enkf update job with COMIN or COMOUT prefix instead of COM (NOAA-EMC#3333) Add capability to run diff resolutions for marine anl and background (NOAA-EMC#3238) Update high resolution tests and fix minor wave issues (NOAA-EMC#3289) Add sfs as valid system (NOAA-EMC#3243) Add missing arch_tars dependencies (NOAA-EMC#3319) Fix the empty aerosol DA aerostat tar file issue (NOAA-EMC#3332) Add missing file safeguard for IMS prep in snow analysis tasks (NOAA-EMC#3329) Fix memory unsetting on Gaea (NOAA-EMC#3325) Fix error log parsing in compute build CI (NOAA-EMC#3301) Remove marineanlvrfy task from global-workflow (NOAA-EMC#3314) Add `gfs_wavepostpnt` dependencies to gfs_cleanup (NOAA-EMC#3313) Increase the GDASApp build wallclock (NOAA-EMC#3298) Capture build fail in Jenkins pipeline when no error logs are produced (NOAA-EMC#3297) Add/update config files for Gaea and check existence before sourcing config files in generate_workflows.sh (NOAA-EMC#3286) Fix ocean restarts when cold starting with DOIAU=YES (NOAA-EMC#3278) Splitting up the archive task (NOAA-EMC#3242) CTests extended validation for C48_ATM and staged C48_S2SW for gfs_fcst and gfs_atmos (NOAA-EMC#3256) Add esnowanl to enkfgfs cycle (NOAA-EMC#3283) Add gfs cycles to C48mx500_3DVarAOWCDA (NOAA-EMC#3249) Add fetch job and update stage_ic to work with fetched ICs (NOAA-EMC#3141) Remove WAFS files and references from `develop` (NOAA-EMC#3263) fix intel stack version number on c5 (NOAA-EMC#3258) Update gsi_monitor and ufs_utils hashes to recent hashes for C5/C6 build and run (NOAA-EMC#3252) Enable DA cycling on gaea C5/C6 (NOAA-EMC#3255) Copy post-processed sea ice increment for diagnostics (NOAA-EMC#3235)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR has one minor bug fix for the stage IC job when and adds a low resolution test that has all components for cycling that is anticipated to be used in GFSv17.
Resolves #3441
Type of change
Change characteristics
How has this been tested?
This test was run on hera:
RUNTESTS=/scratch1/NCEPDEV/climate/Jessica.Meixner/addlowrestest/testlowres03
(As of posting this PR a few last jobs remained, but 1.5 cycles have completed successfully).
Update: 2.5 cycles succeeded.
Checklist