restart_fh from startTime

[NickSzapiro-NOAA]

Commit Queue Requirements:

• Fill out all sections of this template.
• All sub component pull requests have been reviewed by their code managers.
• Run the full Intel+GNU RT suite (compared to current baselines) on either Hera/Derecho/Hercules
• Commit 'test_changes.list' from previous step

---

Description:

Currently, restart_fh is based on ESMF_ClockGet(clock, currTime=mcurrtime)+ESMF_TimeIntervalSet(s=fh_s) where currTime in ESMF clock at first timestep has already been adjusted by fhrot in UFS driver:
https://github.com/ufs-community/ufs-weather-model/blob/develop/driver/UFS.F90#L369-L377

So, for say a control run and its restart offset by fhrot, restart_fh entries in model_configure need to differ on continuation to write restarts at the same "calendar date" or "forecast hour wrt start time."

For example, currently with restart_fh: 2 3 7 11 13 15 for cpld_control_gfsv17 and cpld_restart_gfsv17 restarts are written at:

|         control         |        restart          |
| 2021  3 22  8  0  0   0 | 2021  3 22 20  0  0   0 |
| 2021  3 22  9  0  0   0 | 2021  3 22 21  0  0   0 |
| 2021  3 22 13  0  0   0 | 2021  3 23  1  0  0   0 |
| 2021  3 22 17  0  0   0 | 2021  3 23  5  0  0   0 |
| 2021  3 22 19  0  0   0 | 2021  3 23  7  0  0   0 |
| 2021  3 22 21  0  0   0 | 2021  3 23  9  0  0   0 |

By referencing restart_fh from startTime instead, the same attribute matches for each.

Commit Message:

* UFSWM - restart_fh from startTime
  * CDEPS - restart_fh from startTime
  * CICE - restart_fh from startTime
  * CMEPS - restart_fh from startTime
  * MOM6 - restart_fh from startTime
  * WW3 - restart_fh from startTime

Priority:

• Normal

Git Tracking

UFSWM:

• Closes #Simplify flexible forecast hour options to write restarts #2767

Sub component Pull Requests:

• CDEPS:
• CICE:
• CMEPS:
• MOM6:
• WW3:

UFSWM Blocking Dependencies:

• None

Documentation:

• This PR requires a documentation update, and the WM User's Guide has been updated based on the changes in this PR.
• This PR requires a documentation update, and a WM issue has been opened to track the need for a documentation update; a person responsible for submitting the update has been assigned to the issue (link issue).
• No documentation update is required for this PR (please explain).

---

Changes

Regression Test Changes (Please commit test_changes.list):

• No Baseline Changes.

Input data Changes:

• None.

Library Changes/Upgrades:

• No Updates

---

Testing Log:

• RDHPCS
  • Hera
  • Orion
  • Hercules
  • GaeaC6
  • Derecho
• WCOSS2
  • Dogwood/Cactus
  • Acorn
• CI
• opnReqTest (complete task if unnecessary)

[NickSzapiro-NOAA]

@junwang-noaa @DeniseWorthen This restart_fh from startTime works as expected in cpld_control_gfsv17 and restart. If this looks ok, I can open subcomponent PRs

[JessicaMeixner-NOAA]

@NickSzapiro-NOAA - This will require updates to the global-workflow, which has gone through many updates to ensure it's generating the correct restart times for restart runs, with or without IAU, etc. Can you create an issue in global-workflow to make sure people know that updating to this version of the model will require updates or things will break there?

[junwang-noaa]

@NickSzapiro-NOAA This is great! Would you please create a UWM issue first and then a G-W issue that links to the UWM issue? Using startTime as the base, it can simplify the forecast computation with or without IAU. I will create an FV3 PR and you can link with that in your UFS WM PR, then we can use restart_fh to define restart_interval and restart_interval can be removed from the model_configure (e.g. restart_internval will not be used even it shows up in model_configure)

[junwang-noaa]

@NickSzapiro-NOAA I added the fv3atm changes in my branch: https://github.com/junwang-noaa/fv3atm/tree/uppfreq.
Also since we only use restart_fh, the RT test needs to be updated, e.g.:

[Jun.Wang@hfe02 ufs-weather-model]$ git diff tests/tests/cpld_control_gfsv17
diff --git a/tests/tests/cpld_control_gfsv17 b/tests/tests/cpld_control_gfsv17
index 4be042a2..d7859978 100644
--- a/tests/tests/cpld_control_gfsv17
+++ b/tests/tests/cpld_control_gfsv17
@@ -79,7 +79,7 @@ export CPLCHM=.false.
 export DNATS=0

 export RESTART_N=3
-export RESTART_INTERVAL="${RESTART_N} -1"
+export RESTART_FH="${RESTART_N} -1"
 export OUTPUT_FH='0 21 24'

[NickSzapiro-NOAA]

fyi @junwang-noaa , on hera many tests are timing out. I don't know if hera is just particularly slow the past couple days.

There are also diffs in sfcf024.nc and GFSPRS.GrbF24 for cpld_control_gfsv17_iau_intel and cpld_control_gfsv17_nowav_iau_intel, like:

source: /scratch1/NCEPDEV/nems/Nick.Szapiro/tasks/restart_fh/uwm_git/tests

[Nick.Szapiro@hfe10 tests]$ nccmp -d -S -q -f -B --Attribute=checksum --warn=format -g /scratch2/NAGAPE/epic/UFS-WM_RT/NEMSfv3gfs/develop-20250603/cpld_control_gfsv17_iau_intel/sfcf024.nc /scratch1/NCEPDEV/stmp2/Nick.Szapiro/FV3_RT/rt_559687/cpld_control_gfsv17_iau_intel/sfcf024.nc
Variable  Group Count      Sum   AbsSum         Min         Max       Range        Mean      StdDev
cprat_ave /     54275 0.251225 0.251225 2.62535e-31 0.000119731 0.000119731 4.62874e-06 9.09076e-06
prate_ave /     68331 0.462894 0.462894 7.90013e-20 0.000222671 0.000222671  6.7743e-06 1.38146e-05

[gspetro-NOAA]

@NickSzapiro-NOAA Do you have a status update on this PR?

[NickSzapiro-NOAA]

Hi @gspetro-NOAA. It seems there may still be some issue with prate_ave NOAA-EMC/GFS#1

For restart_fh, my impression was that global-workflow didn't want this in ATM until after GFS.v17 retrospectives start...so we don't take developer time changing/checking changing from restart_interval. Is that right @JessicaMeixner-NOAA @dpsarmie ?

[JessicaMeixner-NOAA]

Does this PR fix the prate_ave issue? If so, can you sync up all the branches and I'll start a test to confirm from the workflow, as we will need to fix prate_ave issue.

[NickSzapiro-NOAA]

No @JessicaMeixner-NOAA , this doesn't fix prate_ave. This is for components to use restart_fh (instead of restart_interval in ATM). It was blocked since prate_ave and cprat_ave changed answers and this is supposed to be bit-for-bit