UFSATM: Replace GFS_interstitial phys_reset and rad_reset routines with combined reset routine

[climbfuji]

Commit Queue Requirements:

• Fill out all sections of this template.
• All sub component pull requests have been reviewed by their code managers.
• Run the full Intel+GNU RT suite (compared to current baselines) on either Hera/Derecho/Hercules
• Commit 'test_changes.list' from previous step

---

Description:

Update submodule pointer for UFSATM for the changes in NOAA-EMC/ufsatm#1004 (Replace GFS_interstitial phys_reset and rad_reset routines with combined reset routine) and ufs-community/ccpp-physics#310 (Update GFS_debug (GFS_interstitialtoscreen)).

This currently changes the answers for the RRTMGP regression test because of NOAA-EMC/ufsatm#1005 (fixed in a separate, forthcoming PR).

Commit Message:

* UFSWM - Update submodule pointer for UFSATM and ccpp-physics
  * AQM - 
  * CDEPS - 
  * CICE - 
  * CMEPS - 
  * CMakeModules - 
  * FV3 - Replace GFS_interstitial phys_reset and rad_reset routines with combined reset routine
    * ccpp-physics - Update GFS_debug schemes: remove GFS_interstitialtoscreen_timestep_init and GFS_interstitialtoscreen_init
    * atmos_cubed_sphere - 
  * GOCART - 
  * HYCOM - 
  * MOM6 - 
  * NOAHMP - 
  * WW3 - 
  * fire_behavior
  * stochastic_physics - 

Priority:

• Normal

Git Tracking

• Related to RRTMGP uses an interstitial variable that is only valid in the physics group, not in the radiation group of the suite NOAA-EMC/ufsatm#1005

Sub component Pull Requests:

• AQM:
• CDEPS:
• CICE:
• CMEPS:
• CMakeModules:
• FV3: Replace GFS_interstitial phys_reset and rad_reset routines with combined reset routine NOAA-EMC/ufsatm#1004
  • ccpp-physics: Update GFS_debug (GFS_interstitialtoscreen) ccpp-physics#310
  • atmos_cubed_sphere:
• GOCART:
• HYCOM:
• MOM6:
• NOAHMP:
• WW3:
• fire_behavior:
• stochastic_physics:
• None

UFSWM Blocking Dependencies:

• None

Documentation:

• No documentation update is required for this PR.

---

Changes

Regression Test Changes (Please commit test_changes.list):

• PR Adds New Tests/Baselines.
• PR Updates/Changes Baselines: control_p8_rrtmgp intel only; needs new baseline because of RRTMGP uses an interstitial variable that is only valid in the physics group, not in the radiation group of the suite NOAA-EMC/ufsatm#1005, which will be fixed in UWM changes for CCPP framework update #2881
• No Baseline Changes.

Input data Changes:

• None.

Library Changes/Upgrades:

• No Updates

---

Testing Log:

• RDHPCS
  • Hera
  • Orion
  • Hercules
  • GaeaC6
  • Derecho
  • Ursa
• WCOSS2
  • Dogwood/Cactus
  • Acorn
• CI
• opnReqTest (complete task if unnecessary)

[climbfuji]

@dustinswales I am getting a lot of test failures for certain classes of tests, not just RRTMGP. They all seem to be regional runs. Is there a different physics scheme that is being used in these tests? We'll need to check if these also "misuse" an interstitial variable. Should be relatively easy to verify. Alternatively, I might have missed resetting a certain interstitial variable in the combined reset call ...

[Dom.Heinzeller@uecflow01 tests]$ ls -l | grep fail_test
-rw-r--r-- 1 Dom.Heinzeller gmtb        133 Sep  5 14:29 fail_test_control_p8_rrtmgp_intel
-rw-r--r-- 1 Dom.Heinzeller gmtb        130 Sep  5 15:21 fail_test_conus13km_2threads_gnu
-rw-r--r-- 1 Dom.Heinzeller gmtb        136 Sep  5 14:44 fail_test_conus13km_2threads_intel
-rw-r--r-- 1 Dom.Heinzeller gmtb        127 Sep  5 15:24 fail_test_conus13km_control_gnu
-rw-r--r-- 1 Dom.Heinzeller gmtb        133 Sep  5 14:59 fail_test_conus13km_control_intel
-rw-r--r-- 1 Dom.Heinzeller gmtb        148 Sep  5 15:25 fail_test_conus13km_debug_2threads_gnu
-rw-r--r-- 1 Dom.Heinzeller gmtb        154 Sep  5 15:02 fail_test_conus13km_debug_2threads_intel
-rw-r--r-- 1 Dom.Heinzeller gmtb        142 Sep  5 15:33 fail_test_conus13km_debug_decomp_gnu
-rw-r--r-- 1 Dom.Heinzeller gmtb         48 Sep  5 15:29 fail_test_conus13km_debug_decomp_intel
-rw-r--r-- 1 Dom.Heinzeller gmtb        121 Sep  5 15:31 fail_test_conus13km_debug_gnu
-rw-r--r-- 1 Dom.Heinzeller gmtb        127 Sep  5 15:14 fail_test_conus13km_debug_intel
-rw-r--r-- 1 Dom.Heinzeller gmtb        130 Sep  5 15:33 fail_test_conus13km_debug_qr_gnu
-rw-r--r-- 1 Dom.Heinzeller gmtb        136 Sep  5 15:13 fail_test_conus13km_debug_qr_intel
-rw-r--r-- 1 Dom.Heinzeller gmtb        124 Sep  5 15:24 fail_test_conus13km_decomp_gnu
-rw-r--r-- 1 Dom.Heinzeller gmtb        130 Sep  5 14:57 fail_test_conus13km_decomp_intel
-rw-r--r-- 1 Dom.Heinzeller gmtb        154 Sep  5 15:32 fail_test_conus13km_radar_tten_debug_gnu
-rw-r--r-- 1 Dom.Heinzeller gmtb        160 Sep  5 15:31 fail_test_conus13km_radar_tten_debug_intel
-rw-r--r-- 1 Dom.Heinzeller gmtb        145 Sep  5 15:13 fail_test_cpld_regional_atm_fbh_intel
-rw-r--r-- 1 Dom.Heinzeller gmtb        115 Sep  5 15:18 fail_test_hrrr_c3_debug_gnu
-rw-r--r-- 1 Dom.Heinzeller gmtb        121 Sep  5 14:43 fail_test_hrrr_c3_debug_intel
-rw-r--r-- 1 Dom.Heinzeller gmtb        175 Sep  5 15:22 fail_test_hrrr_control_2threads_dyn32_phy32_gnu
-rw-r--r-- 1 Dom.Heinzeller gmtb        181 Sep  5 14:45 fail_test_hrrr_control_2threads_dyn32_phy32_intel
-rw-r--r-- 1 Dom.Heinzeller gmtb        139 Sep  5 15:14 fail_test_hrrr_control_2threads_gnu
-rw-r--r-- 1 Dom.Heinzeller gmtb        145 Sep  5 14:35 fail_test_hrrr_control_2threads_intel
-rw-r--r-- 1 Dom.Heinzeller gmtb        166 Sep  5 15:21 fail_test_hrrr_control_debug_dyn32_phy32_gnu
-rw-r--r-- 1 Dom.Heinzeller gmtb        172 Sep  5 14:50 fail_test_hrrr_control_debug_dyn32_phy32_intel
-rw-r--r-- 1 Dom.Heinzeller gmtb        130 Sep  5 15:14 fail_test_hrrr_control_debug_gnu
-rw-r--r-- 1 Dom.Heinzeller gmtb        136 Sep  5 14:43 fail_test_hrrr_control_debug_intel
-rw-r--r-- 1 Dom.Heinzeller gmtb        169 Sep  5 15:23 fail_test_hrrr_control_decomp_dyn32_phy32_gnu
-rw-r--r-- 1 Dom.Heinzeller gmtb        175 Sep  5 14:48 fail_test_hrrr_control_decomp_dyn32_phy32_intel
-rw-r--r-- 1 Dom.Heinzeller gmtb        133 Sep  5 15:17 fail_test_hrrr_control_decomp_gnu
-rw-r--r-- 1 Dom.Heinzeller gmtb        139 Sep  5 14:35 fail_test_hrrr_control_decomp_intel
-rw-r--r-- 1 Dom.Heinzeller gmtb        148 Sep  5 15:23 fail_test_hrrr_control_dyn32_phy32_gnu
-rw-r--r-- 1 Dom.Heinzeller gmtb        154 Sep  5 14:46 fail_test_hrrr_control_dyn32_phy32_intel
-rw-r--r-- 1 Dom.Heinzeller gmtb        112 Sep  5 15:19 fail_test_hrrr_control_gnu
-rw-r--r-- 1 Dom.Heinzeller gmtb        118 Sep  5 14:34 fail_test_hrrr_control_intel
-rw-r--r-- 1 Dom.Heinzeller gmtb        127 Sep  5 15:16 fail_test_hrrr_control_noqr_gnu
-rw-r--r-- 1 Dom.Heinzeller gmtb        115 Sep  5 15:14 fail_test_hrrr_gf_debug_gnu
-rw-r--r-- 1 Dom.Heinzeller gmtb        121 Sep  5 14:43 fail_test_hrrr_gf_debug_intel
-rw-r--r-- 1 Dom.Heinzeller gmtb         42 Sep  5 14:55 fail_test_regional_control_intel
-rw-r--r-- 1 Dom.Heinzeller gmtb         41 Sep  5 14:56 fail_test_regional_decomp_intel
-rw-r--r-- 1 Dom.Heinzeller gmtb        199 Sep  5 14:45 fail_test_regional_spp_sppt_shum_skeb_dyn32_phy32_intel
-rw-r--r-- 1 Dom.Heinzeller gmtb        163 Sep  5 14:33 fail_test_regional_spp_sppt_shum_skeb_intel

[dustinswales]

@climbfuji The regional physics are quite different than the global.
But... GP adopted some of the subgridscale cloud business from the regional applications, so maybe this helps narrow down the misused variables. I'm looking into this now.

[climbfuji]

@climbfuji The regional physics are quite different than the global. But... GP adopted some of the subgridscale cloud business from the regional applications, so maybe this helps narrow down the misused variables. I'm looking into this now.

@dustinswales On Friday I was able to fix all but two regression tests (the error was my fault). One of the failures is the RRTMGP test (b4b differences, as expected, thus good), the other one might have been a fluke (will rerun).

[dustinswales]

@climbfuji Found it, dustinswales/fv3atm@004fb86
Should you include it here? Or should I add it to the framework hash update branch?

[climbfuji]

@climbfuji Found it, dustinswales/fv3atm@004fb86 Should you include it here? Or should I add it to the framework hash update branch?

Excellent! I think a separate PR would be great. I can pull it into my cleanup PRs for the interstitial afterwards, but at least it's recorded separately. Thanks!

[climbfuji]

@climbfuji The regional physics are quite different than the global. But... GP adopted some of the subgridscale cloud business from the regional applications, so maybe this helps narrow down the misused variables. I'm looking into this now.

@dustinswales On Friday I was able to fix all but two regression tests (the error was my fault). One of the failures is the RRTMGP test (b4b differences, as expected, thus good), the other one might have been a fluke (will rerun).

@dustinswales Turns out the other test that failed (timed out), regional_noquilt, went through when I reran it. I didn't rerun the whole thing to get a "clean" log, since I know that we'll need updates for the RRTMGP test, that there will be several more PRs before this PR is at the top of the queue, etc.

[climbfuji]

The regional_noquilt failure was a fluke (test hung at initialization until it timed out), a rerun was successful. See also comment #2880 (comment)

[climbfuji]

@climbfuji Found it, dustinswales/fv3atm@004fb86 Should you include it here? Or should I add it to the framework hash update branch?

Excellent! I think a separate PR would be great. I can pull it into my cleanup PRs for the interstitial afterwards, but at least it's recorded separately. Thanks!

@dustinswales Just checking if you created a PR for this?

[dustinswales]

@climbfuji Found it, dustinswales/fv3atm@004fb86 Should you include it here? Or should I add it to the framework hash update branch?

Excellent! I think a separate PR would be great. I can pull it into my cleanup PRs for the interstitial afterwards, but at least it's recorded separately. Thanks!

@dustinswales Just checking if you created a PR for this?

@climbfuji Yes. Included in #2881

[climbfuji]

@climbfuji Found it, dustinswales/fv3atm@004fb86 Should you include it here? Or should I add it to the framework hash update branch?

Excellent! I think a separate PR would be great. I can pull it into my cleanup PRs for the interstitial afterwards, but at least it's recorded separately. Thanks!

@dustinswales Just checking if you created a PR for this?

@climbfuji Yes. Included in #2881

Great, thanks. I will mention this in the PR description

[climbfuji]

@jkbk2004 @dustinswales Just to avoid any confusion, this PR and its submodule components should go before #2881 (correct, @dustinswales?)

[dustinswales]

@jkbk2004 @dustinswales Just to avoid any confusion, this PR and its submodule components should go before #2881 (correct, @dustinswales?)

That is my understanding.

[gspetro-NOAA]

@climbfuji Is this PR ready to process/plan for commit queue?

[climbfuji]

@climbfuji Is this PR ready to process/plan for commit queue?

Yes, definitely. @dustinswales is waiting for this. Let me update the branches.

Thanks!

[gspetro-NOAA]

@climbfuji Could you sync your branch? We're ready to test this PR. :)

[gspetro-NOAA]

@grantfirl It looks like Dom's out till Monday. Are #2881 and #2882 dependent on this PR?

[grantfirl]

@grantfirl It looks like Dom's out till Monday. Are #2881 and #2882 dependent on this PR?

OK, I updated the ccpp-physics PR for this one, but it looks like I don't have write permissions on Dom's fork for fv3atm or ufs-weather-model, so I can't update them. It looks like the comments in #2881 suggest that #2880 goes in first, but #2882 is completely independent and it can go in just fine. I just updated it to the top of develop.