Bring CDEPS Inline capability to FV3

[uturuncoglu]

Commit Queue Requirements:

• Fill out all sections of this template.
• All sub component pull requests have been reviewed by their code managers.
• Run the full Intel+GNU RT suite (compared to current baselines) on either Hera/Derecho/Hercules
• Commit 'test_changes.list' from previous step

---

Description:

This PR aims to bring CDEPS Inline capability to FV3 to allow updating desired fields in CCPP.
Some additional information:

• Cookbook - https://gist.github.com/uturuncoglu/587b0eee3b95be9700e156249d939566
• Sildes from the past that shows the implementation - https://docs.google.com/presentation/d/1w9-n0NKSC3go3i7Y9WiWOz6cJ5D1VXxnjMDYnyjStn0/edit?slide=id.g2b3156c1e0e_0_249#slide=id.g2b3156c1e0e_0_249

Commit Message:

* UFSWM - 
  * AQM - 
  * CDEPS - 
  * CICE - 
  * CMEPS - 
  * CMakeModules - 
  * FV3 - 
    * ccpp-physics - 
    * atmos_cubed_sphere - 
  * GOCART - 
  * HYCOM - 
  * MOM6 - 
  * NOAHMP - 
  * WW3 - 
  * fire_behavior
  * stochastic_physics - 

Priority:

• Normal

Git Tracking

UFSWM:

• None

Sub component Pull Requests:

• AQM:
• CDEPS:
• CICE:
• CMEPS:
• CMakeModules:
• FV3: Bring CDEPS Inline capability NOAA-EMC/ufsatm#988
  • ccpp-physics: Create new scheme to get fields provided by CDEPS Inline ccpp-physics#298
  • atmos_cubed_sphere:
• GOCART:
• HYCOM:
• MOM6:
• NOAHMP:
• WW3:
• fire_behavior:
• stochastic_physics:
• None

UFSWM Blocking Dependencies:

• None

Documentation:

• This PR requires a documentation update, and the WM User's Guide has been updated based on the changes in this PR.
• This PR requires a documentation update, and a WM issue has been opened to track the need for a documentation update; a person responsible for submitting the update has been assigned to the issue (link issue).
• No documentation update is required for this PR (please explain).

---

Changes

Regression Test Changes (Please commit test_changes.list):

• PR Adds New Tests/Baselines. The new test is named as conus13km_gl_inline_intel. It is based on conus13km grid but slightly different than the exiting configurations in terms of input.nml. It basically update sea surface related variables (set, sea ice variables) over the Great Lakes using FVCOM forecast output.

Input data Changes:

• New input data - /glade/work/turuncu/GL/RT/NEMSfv3gfs/input-data-20250507/FV3_input_data_conus13km_inline_GL/INPUT

Library Changes/Upgrades:

• No Updates

---

Testing Log:

• RDHPCS
  • Hera
  • Orion
  • Hercules
  • GaeaC6
  • Derecho
  • Ursa
• WCOSS2
  • Dogwood/Cactus
  • Acorn
• CI
• opnReqTest (complete task if unnecessary)

[NickSzapiro-NOAA]

Thank you @uturuncoglu!

I am still catching up but I do see some reshape calls that may be important for GFS_coupling --> CCPP variables:
https://github.com/search?q=repo%3ANOAA-EMC%2Ffv3atm%20reshape&type=code

If I understand, maybe it is worth checking what lat,lon from FV3 mesh look like if passed over to CCPP in same way

[uturuncoglu]

@NickSzapiro-NOAA I think the issue is solved when I make change in the standard names after @DusanJovic-NOAA suggestion. At this point I am not sure what is the difference between horizontal_dimension vs. horizontal_loop_extent in the definition and how was it working in the first place before CCPP RP in the past but it looks it fixed the issue. See following commit - NOAA-EMC/ufsatm#988 (comment). I am running model with/without inline at this point to be sure it is working. The fields are in CCPP side looks good at this point. I'll add couple of plots to the PR.

[uturuncoglu]

@NickSzapiro-NOAA @junwang-noaa @DeniseWorthen It seems issue is fixed. @DusanJovic-NOAA Thanks again for your great help. Please see the latest comment in NOAA-EMC/ufsatm#988. Let me know if you want me to change those PRs ready to view.

[uturuncoglu]

Just let me know if you also want me to include a new RT that demonstrates the capability.

[uturuncoglu]

@junwang-noaa @DeniseWorthen @NickSzapiro-NOAA JFYI, I'll add a new RT (regional - RRFS_CONUScompact_13km, updates GL SST and ice conditions) to demonstrate the capability and this PR will be ready to review and merge. The test is created with SRW app.

[uturuncoglu]

I think I could modify existing conus13km_control and enable inline capability in it. So, new test would be conus13km_control_inline.

[uturuncoglu]

@gspetro-NOAA Okay. It was challenging since FV3 renamed as UFSATM now but synced now. I am starting to test and push the log files when it is done.

[uturuncoglu]

@gspetro-NOAA I have just push the log file. RTs are fine on Hercules.

[uturuncoglu]

@gspetro-NOAA BTW, I need to run the new test on Derecho (input files are staged in there) to be sure it is still working fine.

[uturuncoglu]

@gspetro-NOAA I am getting following error when I try to compile new RT (conus13km_gl_inline) on Derecho,

Lmod has detected the following error: These module(s) or extension(s) exist
but cannot be loaded as requested: "scotch/7.0.4"
   Try: "module spider scotch/7.0.4" to see how to load the module(s).

This was working fine in the past. Let me know what you think? I could also copy input files to Hercules and try over there.

[gspetro-NOAA]

@uturuncoglu At the moment, we are not running RTs on Derecho until PR #2863 goes through to fix a number of issues. If you can test on Hercules, that will speed things up. Otherwise, we may have to view #2863 as a dependency. @jkbk2004 Anything to add?

[uturuncoglu]

@gspetro-NOAA Okay. I'll test on Hercules and update you. Copying required files now.

[uturuncoglu]

@gspetro-NOAA Okay. The case is working fine on Hercules. The only issue that I noticed. The RT system assigning wrong number of processor to component (assigning 85 core to FV3 but layout was 12x12 in input.nml). So, I had to manually edit processor counts in config files. This was working fine on Derecho. I wonder if something changed in UFS WM model side. Any idea?

[uturuncoglu]

@gspetro-NOAA Okay. I found the issue and will be fix soon. I was getting input.nml from INPUT directory to make it simple but I might need to define a template for it.

[uturuncoglu]

@gspetro-NOAA I also fixed the test. So, it is ready in my end.

[gspetro-NOAA]

Great! Could you push your logs from your test run?

[uturuncoglu]

@gspetro-NOAA I have already pushed the test change list. Let me know if you want mw to push the logs too.

[gspetro-NOAA]

@uturuncoglu Yes, ideally developers should push the logs, too, unless the changes are simple documentation/CI changes that don't affect source code. If you can do that and get the UFSATM subcomponent PR approved, then we can schedule this PR for the Commit Queue.

[uturuncoglu]

@gspetro-NOAA Okay. That makes sense. Let me know which log file do I need to push. I could push RegressionTests_derecho.log but I think that would be generated by automated testing. Maybe I am wrong.

[gspetro-NOAA]

@uturuncoglu Since you did your most recent testing on Hercules, you can push tests/logs/RegressionTests_hercules.log.

[gspetro-NOAA]

@uturuncoglu Just looking for a status update on this because I haven't seen any updates to the logs/test_changes.list.

[uturuncoglu]

@gspetro-NOAA Let me push test change list and also tests/logs/RegressionTests_hercules.log.

[uturuncoglu]

@gspetro-NOAA Okay. I rerun all tests on Hercules and here is the failed tests,

fail_test_conus13km_gl_inline_intel
fail_test_hafs_regional_atm_wav_intel
fail_test_regional_atmaq_debug_intel

I could not see anything particular in the log files and it seems hafs_regional_atm_wav_intel is run without any issue and regional_atmaq_debug is failed because of the wall time limit. Any idea? I think I could not push the log file since it will also show these two failed runs but if you would like, I could push them too. BTW, my run directory is in /work2/noaa/stmp/tufuk/stmp/tufuk/FV3_RT/rt_643896

[gspetro-NOAA]

Ok, thanks! If you can sync w/WM develop branch, I can finalize the test_changes.list stuff for you, and we can get this into the commit queue. :)

[uturuncoglu]

@gspetro-NOAA Thanks. I have just sync the model. Let me know if you have any issues.