Failure at restart when DOUT_S_SAVE_INTERIM_RESTART_FILES=TRUE

[slevis-lmwg]

Brief summary of bug

I have confirmed @adrifoster's TRENDY2025 restart failure with my ctsm5.3.062 test (original 20-yr simulation described in NCAR/LMWG_dev#109). I restarted my case with
STOP_N = 2 days
REST_N = 1 day
DOUT_S_SAVE_INTERIM_RESTART_FILES = FALSE
RESUBMIT = 1
and it completed all 4 days correctly. Then I restarted again with
STOP_N = 2 days
REST_N = 1 day
DOUT_S_SAVE_INTERIM_RESTART_FILES = TRUE
RESUBMIT = 1
This time it completed the first 2 days but failed when it resubmitted.

General bug information

CTSM version you are using: ctsm5.3.062 (first version splitting hX files into hXa/hXi)

Does this bug cause significantly incorrect results in the model's science? No

Configurations affected: DOUT_S_SAVE_INTERIM_RESTART_FILES = TRUE

Details of bug

Important output or errors that show the problem

My case ran here
/glade/derecho/scratch/slevis/ctsm53062_f09_1850/run
and archived here
/glade/derecho/scratch/slevis/archive/ctsm53062_f09_1850

So far I have noticed the following behavior:

1. In .../archive/.../rest I type ls 0021-01-0[5-7]* and get (recall the model restarted successfully from 01-05, and then failed to restart from 01-07):

0021-01-05-00000:
ctsm53062_f09_1850.clm2.h0a.0020-12.nc            ctsm53062_f09_1850.mosart.h0i.0020-12.nc            rpointer.atm.0021-01-02-00000  rpointer.lnd.0006-01-01-00000  rpointer.lnd.0021-01-06-00000  rpointer.rof.0021-01-04-00000
ctsm53062_f09_1850.clm2.h0i.0020-12.nc            ctsm53062_f09_1850.mosart.r.0021-01-05-00000.nc     rpointer.atm.0021-01-03-00000  rpointer.lnd.0011-01-01-00000  rpointer.lnd.0021-01-07-00000  rpointer.rof.0021-01-05-00000
ctsm53062_f09_1850.clm2.r.0021-01-05-00000.nc     ctsm53062_f09_1850.mosart.rh0a.0021-01-05-00000.nc  rpointer.atm.0021-01-04-00000  rpointer.lnd.0016-01-01-00000  rpointer.rof.0006-01-01-00000  rpointer.rof.0021-01-06-00000
ctsm53062_f09_1850.clm2.rh0a.0021-01-05-00000.nc  ctsm53062_f09_1850.mosart.rh0i.0021-01-05-00000.nc  rpointer.atm.0021-01-05-00000  rpointer.lnd.0021-01-01-00000  rpointer.rof.0011-01-01-00000  rpointer.rof.0021-01-07-00000
ctsm53062_f09_1850.clm2.rh0i.0021-01-05-00000.nc  rpointer.atm.0006-01-01-00000                       rpointer.atm.0021-01-06-00000  rpointer.lnd.0021-01-02-00000  rpointer.rof.0016-01-01-00000
ctsm53062_f09_1850.cpl.r.0021-01-05-00000.nc      rpointer.atm.0011-01-01-00000                       rpointer.atm.0021-01-07-00000  rpointer.lnd.0021-01-03-00000  rpointer.rof.0021-01-01-00000
ctsm53062_f09_1850.datm.r.0021-01-05-00000.nc     rpointer.atm.0016-01-01-00000                       rpointer.cpl.0021-01-05-00000  rpointer.lnd.0021-01-04-00000  rpointer.rof.0021-01-02-00000
ctsm53062_f09_1850.mosart.h0a.0020-12.nc          rpointer.atm.0021-01-01-00000                       rpointer.lnd.0001-01-01-01800  rpointer.lnd.0021-01-05-00000  rpointer.rof.0021-01-03-00000

0021-01-06-00000:
 ctsm53062_f09_1850.clm2.h0a.0020-12.nc             ctsm53062_f09_1850.clm2.rh0i.0021-01-06-00000.nc   ctsm53062_f09_1850.mosart.h0i.0020-12.nc             rpointer.atm                   'rpointer.lnd$NINST_STRING'
 ctsm53062_f09_1850.clm2.h0i.0020-12.nc             ctsm53062_f09_1850.cpl.r.0021-01-06-00000.nc       ctsm53062_f09_1850.mosart.r.0021-01-06-00000.nc     'rpointer.atm$NINST_STRING'      rpointer.rof
 ctsm53062_f09_1850.clm2.r.0021-01-06-00000.nc      ctsm53062_f09_1850.datm.r.0021-01-06-00000.nc      ctsm53062_f09_1850.mosart.rh0a.0021-01-06-00000.nc   rpointer.cpl.0021-01-06-00000  'rpointer.rof$NINST_STRING'
 ctsm53062_f09_1850.clm2.rh0a.0021-01-06-00000.nc   ctsm53062_f09_1850.mosart.h0a.0020-12.nc           ctsm53062_f09_1850.mosart.rh0i.0021-01-06-00000.nc   rpointer.lnd

0021-01-07-00000:
ctsm53062_f09_1850.clm2.h0a.0020-12.nc         ctsm53062_f09_1850.clm2.rh0a.0021-01-07-00000.nc  ctsm53062_f09_1850.datm.r.0021-01-07-00000.nc  ctsm53062_f09_1850.mosart.r.0021-01-07-00000.nc     rpointer.cpl.0021-01-07-00000
ctsm53062_f09_1850.clm2.h0i.0020-12.nc         ctsm53062_f09_1850.clm2.rh0i.0021-01-07-00000.nc  ctsm53062_f09_1850.mosart.h0a.0020-12.nc       ctsm53062_f09_1850.mosart.rh0a.0021-01-07-00000.nc
ctsm53062_f09_1850.clm2.r.0021-01-07-00000.nc  ctsm53062_f09_1850.cpl.r.0021-01-07-00000.nc      ctsm53062_f09_1850.mosart.h0i.0020-12.nc       ctsm53062_f09_1850.mosart.rh0i.0021-01-07-00000.nc

01-07 seems to be missing numerous rpointer files.
01-06 has rpointer files with strange suffixes.

2. In .../run I see an empty rpointer.lnd and the following error in cesm.log:
   end-of-file during read, unit 99, file /glade/derecho/scratch/slevis/ctsm53062_f09_1850/run/./rpointer.lnd

---

@adrifoster and others please add information that you consider helpful.

[slevis-lmwg]

I tried git grep -i DOUT_S_SAVE_INTERIM_RESTART_FILES in various places:

• clm, ccs_config, share, cdeps, mosart, rtm, mizuroute, cism have nothing
• cmeps returns

cime_config/config_component_cesm.xml:  <entry id="DOUT_S_SAVE_INTERIM_RESTART_FILES">

• cime returns

CIME/case/case_st_archive.py:        datename_is_last or case.get_value("DOUT_S_SAVE_INTERIM_RESTART_FILES")
CIME/case/case_st_archive.py:        case.get_value("DOUT_S_SAVE_INTERIM_RESTART_FILES"),
CIME/case/case_st_archive.py:                if case.get_value("DOUT_S_SAVE_INTERIM_RESTART_FILES"):
CIME/case/case_st_archive.py:    dout_s_save_interim = self.get_value("DOUT_S_SAVE_INTERIM_RESTART_FILES")
CIME/data/config/config_tests.xml:    <DOUT_S_SAVE_INTERIM_RESTART_FILES>TRUE</DOUT_S_SAVE_INTERIM_RESTART_FILES>
CIME/data/config/config_tests.xml:    <DOUT_S_SAVE_INTERIM_RESTART_FILES>TRUE</DOUT_S_SAVE_INTERIM_RESTART_FILES>

[slevis-lmwg]

• Before diving deeper, confirming whether ctsm5.3.061 exhibits the same behavior (same test as above).

[slevis-lmwg]

The good news (for me) is that ctsm5.3.061 exhibits the same behavior.
The bad news (not that bad, considering we hadn't noticed this until now) is that this has been in the model for a little while.

I could try bisecting until I find the first tag that exhibits the behavior.

[slevis-lmwg]

@jedwards4b this may be a CIME bug, are you the right person to notify? If so, shall I hand this over to you?

[jedwards4b]

This is because changes in ctsm and mosart have either not been merged or were reverted after merging. I am still waiting for an explanation of why the merge was reverted.

I have submitted PR's ESCOMP/MOSART#115
and (again) #3354

I have also submitted
ESCOMP/CMEPS#576
and ESMCI/cime#4827
to test this interim restart with the ERR test. For example this issue was tested using ERR.f09_t232.I1850Clm60BgcCrop.derecho_intel.drv-interim_restart