Model hangs after COMPLETED MOM INITIALIZATION?

[benjamin-cash]

Has anyone had the case where the model just hangs after COMPLETED MOM INITIALIZATION? I've seen this happen a few times now, on my local cluster and now on Frontera. I'm not seeing any error messages, it just hangs. The closest I have to an error message is the last entry in PET000, which is:

20241017 181452.975 DEBUG            PET000 Generic SetVM() is executing for: fv3_fcst
20241017 181452.976 DEBUG            PET000 Generic SetVM() is calling ESMF_GridCompSetVMMaxPEs() for: fv3_fcst with:
20241017 181452.976 DEBUG            PET000  -             MaxCount =          2
20241017 181452.976 DEBUG            PET000  -       OpenMpHandling =
20241017 181452.976 DEBUG            PET000  -     OpenMpNumThreads =         -1
20241017 181452.976 DEBUG            PET000  -   ForceChildPthreads =          F
20241017 181452.976 DEBUG            PET000  -  PthreadMinStackSize =         -1

Does this situation and log message ring a bell for anyone?

[jiandewang]

we had 2 hanging case in HR4 (C1152) but they finished several days of forecast before hanging and we saw lots of unreasonable temperature warning in the log file. So your case is different from HR4. I assume you are running C192.

I recall we had hanging issue like this when we tried HR2 on HERA (the system worked fine on wcoss2 but hanging on HERA). I will dig out our HR2 doc and get bak to you.

[jiandewang]

Yes it is related to memory. When we tried HR2 tag on wcoss2, everything is fine but it hanged on HERA with the same setting. What we found out is the setting of write_task_per_group is the cause. On wcoss2 it worked fine with 20 but this # caused hanging on HERA. Changed it to 10 solved the problem.

[benjamin-cash]

Thanks again!Reducing cores per node didn't affect it, so I will try this next. I am also running a memory profiler to see if that gives me any useful information. Did you increase the number of write groups when you reduced the tasks per group?

[jiandewang]

no, all I did is changing it from 20 to 10

[benjamin-cash]

I cut it from 30 to 10 but got the same behavior.

[benjamin-cash]

Well, I now know the what but not the why. It's something to do with the sfc IC files, because I rolled back to an earlier version and it now runs past this point. Figuring out what the difference is will have to wait until tomorrow, but at least one mystery is solved!

[benjamin-cash]

Fiddling with settings some more I finally got something a little more concrete. Any ideas as to where this error would be coming from? It seems to be sensitive to the choice if sfc_tile* files, but this field is not in those files.

20241018 115753.750 INFO             PET000 ... returned from wrtFB(08,02) FieldBundleRegridStore().
20241018 115753.767 ERROR            PET000 src/addon/NUOPC/src/NUOPC_Base.F90:2108 Invalid argument  - hflx_fire is not a StandardName in the NUOPC_FieldDictionary!
20241018 115753.767 ERROR            PET000 src/addon/NUOPC/src/NUOPC_Base.F90:486 Invalid argument  - Passing error in return code
20241018 115753.767 ERROR            PET000 module_fcst_grid_comp.F90:460 Invalid argument  - Passing error in return code
20241018 115753.767 ERROR            PET000 module_fcst_grid_comp.F90:1233 Invalid argument  - Passing error in return code
20241018 115753.767 ERROR            PET000 fv3_cap.F90:974 Invalid argument  - Passing error in return code
20241018 115753.767 ERROR            PET000 ATM:src/addon/NUOPC/src/NUOPC_ModelBase.F90:692 Invalid argument  - Passing error in return code
20241018 115753.767 ERROR            PET000 UFS Driver Grid Comp:src/addon/NUOPC/src/NUOPC_Driver.F90:2901 Invalid argument  - Phase 'IPDvXp01' Initialize for modelComp 2: ATM did not return ESMF_SUCCESS
20241018 115753.767 ERROR            PET000 UFS Driver Grid Comp:src/addon/NUOPC/src/NUOPC_Driver.F90:1349 Invalid argument  - Passing error in return code
20241018 115753.767 ERROR            PET000 UFS Driver Grid Comp:src/addon/NUOPC/src/NUOPC_Driver.F90:486 Invalid argument  - Passing error in return code
20241018 115753.767 ERROR            PET000 UFS.F90:397 Invalid argument  - Aborting UFS
20241018 115753.767 INFO             PET000  Finalizing ESMF with endflag==ESMF_END_ABORT

[DeniseWorthen]

You need to update your fd_ufs.yaml

src/addon/NUOPC/src/NUOPC_Base.F90:2108 Invalid argument  - hflx_fire is not a StandardName in the NUOPC_FieldDictionary!

[benjamin-cash]

Thanks @DeniseWorthen - there is an fd_ufs.yaml that includes this in my repo; I'm not sure why it didn't get staged to the run directory.

[benjamin-cash]

@DeniseWorthen - Something odd about this error that I just realized. The C96mx100 case that I ran using this same executable, and also created via the global-workflow, did not produce this error despite hflx_fire not appearing in fd_ufs.yaml. Do you know where the requirement for this field would be entering? It might help me understand why C96mx100 works and C192mx025 doesn't.

[DeniseWorthen]

See fire commit

Once the fire behaviour model was committed, the field names must be in the dictionary regardless of whether they are used.

[benjamin-cash]

The slow march through various instabilities continues! Updating fd_ufs.yaml got me through the previous one, only to crash with:

PASS: fcstRUN phase 1, n_atmsteps =                0 time is         6.575882
 warning: snow/ice temperature is too low:   166.046793781995
 fix snow/ice temperature: reset it to:   173.000000000000
 warning: snow/ice temperature is too low:   135.907350633521
 fix snow/ice temperature: reset it to:   173.000000000000
 warning: snow/ice temperature is too low:   72.8250271804138
 fix snow/ice temperature: reset it to:   173.000000000000
 warning: snow/ice temperature is too low:   105.151025745759
 fix snow/ice temperature: reset it to:   173.000000000000
WARNING, negative qv   -0.2710505E-19 at i,j,k=      31       1       1
 warning: snow/ice temperature is too low:   119.288363802619
 fix snow/ice temperature: reset it to:   173.000000000000
 warning: snow/ice temperature is too low:   118.348881619905
 fix snow/ice temperature: reset it to:   173.000000000000
 ---isec,seconds         300         300
  gfs diags time since last bucket empty:   8.333333333333333E-002 hrs
 in atmos_model update, fhzero=   24.0000000000000      fhour=  8.3333336E-02
  0.0000000E+00
 zeroing coupling accumulated fields at kdt=            1
 zeroing coupling accumulated fields at kdt=            1
PASS: fcstRUN phase 2, n_atmsteps =                0 time is         1.539477
  aft fcst run output time=         300 FBcount=           8 na=           1
forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image              PC                Routine            Line        Source
libc.so.6          00007F26D6619520  Unknown               Unknown  Unknown
ufs_model.x        0000000004B8D86C  ice_transport_dri         280  ice_transport_driver.F90
ufs_model.x        0000000004C49B5E  ice_step_mod_mp_s        1027  ice_step_mod.F90
ufs_model.x        0000000004A48F31  cice_runmod_mp_ci         270  CICE_RunMod.F90
ufs_model.x        000000000473A87B  ice_comp_nuopc_mp        1198  ice_comp_nuopc.F90

@DeniseWorthen @jiandewang does this one look familiar?

[DeniseWorthen]

Try looking in ice_diag.d for a CICE error message.

[benjamin-cash]

Nothing there, unfortunately. I've poked around some more and am pretty stumped at this point. I feel like it has to be a problem with the ICs that I am using, but without a working run to compare it to it's hard to tell what the problem might be.

[benjamin-cash]

@DeniseWorthen @jiandewang @NeilBarton-NOAA I used the gfsnc, sfcnc, mom.res*nc and cice_model.res.nc files from the regression tests below in my run directory, and also my ICs in the test run directories. Behavior was unchanged - everything still crashed in my directory using the regression test ICs and the regression tests ran without issue using my ICs. It seems unlikely now that the problem is in the ICs and has to be some kind of setting that is off for this configuration.

COMPILE | datm_cdeps | intel | -DAPP=NG-GODAS | - wcoss2         | fv3 |
RUN | datm_cdeps_mx025_gefs                             | - wcoss2 noaacloud                   | baseline |

COMPILE | s2swa | intel | -DAPP=S2SWA -DCCPP_SUITES=FV3_GFS_v17_coupled_p8,FV3_GFS_v17_coupled_p8_ugwpv1 | | fv3 |
RUN | cpld_control_c192_p8                              | - wcoss2 jet acorn s4 noaacloud      | baseline |

[jiandewang]

at least we can exclude the IC issue. Now you need to compare all those settings. At least you have one working case to compare with

[benjamin-cash]

Agree. It's a little tricky because the rt is for p8, so parts of the code have moved on quite a bit since then. There really should be a C192mx025 rt for the head of develop, since that is the operational target. That would make debugging this a lot easier...

[jiandewang]

@benjamin-cash one suggestion:
you have access to ORION, right ? run this case of UWM
https://github.com/ufs-community/ufs-weather-model/blob/develop/tests/rt.conf#L51
this will give you a good case for comparison
you can further use the run dir as a template, replace IC with yours (there will be time mismatch in the settings but system won't complain as there is no time stamp inside the IC nctcdf file) to see if it will move forward a bit.
Let me know if you need help on how to run UWM on ORION

[benjamin-cash]

Hi @jiandewang - thanks for the suggestion! I am running the following on Hercules, since that's where I was already set up to test:
./rt.sh -k -a nems cpld_control_c192_p8

Unfortunately, even with -k all of the test run directories appear to be empty, did I make a mistake in setting this up somehow?

[jiandewang]

I just checked
https://github.com/ufs-community/ufs-weather-model/blob/develop/tests/rt.conf#L51
is using half degree ocean and ice, so it may not fit for your case

[benjamin-cash]

If nothing else I could swap in my C192 gfs*nc and sfc*nc files and see what happens.

[jiandewang]

If nothing else I could swap in my C192 gfs*nc and sfc*nc files and see what happens.

yes at least you can use C192 case to try swap fv3 IC, or use a C384 case to swap ice IC

[jiandewang]

this is what usually I do to run a sample case
https://github.com/ufs-community/ufs-weather-model/blob/develop/tests/rt.conf
cp rt.conf rt.conf-test
delete everything except line 41 https://github.com/ufs-community/ufs-weather-model/blob/develop/tests/rt.conf#L41
and line 52 https://github.com/ufs-community/ufs-weather-model/blob/develop/tests/rt.conf#L52

./rt.sh -a nems -k -l rt.conf-test
this should work as I did in this way hundreds of times

[gspetro-NOAA]

@benjamin-cash Are you all set here or do you need further assistance?

[benjamin-cash]

Hmm. When I set ice_ic='none' I got the same segfault at the same line. Could you point me to your run directory - maybe there is a difference in the executable?

[DeniseWorthen]

/work2/noaa/stmp/dworthen/bcash.rundir

The executable is sym-linked up to where I compiled using

./compile.sh hercules "-DAPP=S2S -DCCPP_SUITES=FV3_GFS_v17_coupled_p8_ugwpv1" s2s intel NO NO 2>&1 | tee s2s.log

So far it has gotten to a full 24 hours (judging by restarts written).

[benjamin-cash]

Weird. My run crashed at that same point in ice_transport_driver with both ice_ic='none' and the file you pointed me too. Unfortunately I don't have permission to see /work2/noaa/stmp/dworthen - would you be able to open that up for me?

[benjamin-cash]

I made a guess that you had used /work/noaa/nems/dworthen/ufs_dststat/tests/fv3_s2s.exe as the executable but got the same crash. I think I'm stuck until I can see the run directory. :)

[DeniseWorthen]

My sandbox is a direct copy of your run directory, but with a job_card from an RT added. You can compile the exact same way I did in your own checkout.

Sorry, I'm other things right now and I'm going to have to let you attempt some things on your own before I can take any more time to resolve your issue.

[benjamin-cash]

The answer appears to be that these settings are missing from the job script:

ulimit -s unlimited
export OMP_STACKSIZE=512M

They are present in HERCULES.env and FRONTERA.env, but for some reason were never set in the job script. ulimit -s unlimited eliminated the seg fault in the ice transport code, and export OMP_STACKSIZE=512M eliminated the crash in the radiation, or at least that is what I am seeing in my test run. Somewhat concerningly, in my test run I am also seeing warnings like

(zap_snow_temperature)zqsn:  -110219996.370171
  (zap_snow_temperature)zap_snow_temperature: temperature out of bounds!

without the model crashing, but that is a concern for another day.