Question regarding restarting coupled run

[lisa-bengtsson]

Hi! I have a S2S simulation using the HR3 tag of the global workflow (C1152 atm, 0.25 Ocean/Ice). My forecast ran 150 hours, and I wrote restart files after 120 hours. Now I would like to restart the run from 120 hours. I changed warm_start from .false. to .true., and removed the .db files to restart. The log file suggest that the workflow finds the correct atmosphere/ocean and ice restart files. But I get a bunch of messages like: "Abort(1) on node 0 (rank 0 in comm 496): application called MPI_Abort(comm=0x84000003, 1) - process 0".

Anyone have any ideas what I need to change? My log file is here on Hera:
/scratch2/BMC/gsienkf/Lisa.Bengtsson/global_workflow_HR3/comrot/control_c1152/logs/2011112303/gfsfcst.log

Thank you in advnance,
Lisa

[WalterKolczynski-NOAA]

Lisa,

I think we need more information about how you are using workflow to generate things after the restart. We have open issues to extend forecast resumption for the new coupled components (see #2273, #2274). It sounds like you are aware of that and trying to set things up manually, but I'm not sure how that plays back into the workflow.

[lisa-bengtsson]

Hi Walter, yes I could set things up manually to initialize the model from replay IC's by manually placing them as restart files

The main steps for that was:
2) Manually create a coupler.res file to let the model know the base cycle is 00, but I want to start at fhr 3.
3) set the warm_start flag to true
4) update forecast_postdet.sh to use "cyct" instead of "cyc" in the MOM6, mediator, and CICE6 postdets to recognize the start time from forecast_det (from the coupler.res file in the atmosphere dir).

The challenge I face now is that I don't have any ocean restart files written to the ocean/ice restart dirs to resume the forecast after integration (maybe I'm just missing a flag?) , only atmospheric restart files were written using the restart_interval_gfs=120. But it looks like even if I would have them, I would also need to add some functionality so that we are looking for the same date as identified in the atmosphere restart directory? Is that your understanding as well? Thanks!

[lisa-bengtsson]

I got it to work with some manual moving of files and some non-elegant patch work in forecast_postdet.sh. Not pretty but it's chugging along now!

Here's what I found:
MOM6_RESTART/ does not seem to be linked correctly to the COMROT/ ocean restart dir.
forecast_postedt.sh does not have the option of linking the ocean/ice/mediator restart files to the INPUT directory in the RUNDIR.

I propose to change the MOM6 postdet from:
# Copy MOM6 ICs local nn for nn in $(seq 1 4); do if [[ -f "${COM_OCEAN_RESTART_PREV}/${sPDY}.${scyc}0000.MOM.res_${nn}.nc" ]]; then ${NLN} "${COM_OCEAN_RESTART_PREV}/${sPDY}.${scyc}0000.MOM.res_${nn}.nc" "${DATA}/INPUT/MOM.res_${nn}.nc" fi done
to

`
if [[ ${RERUN} = "YES" ]]; then
PDYT="${CDATE_RST:0:8}"
cyct="${CDATE_RST:8:2}"
local nn
for nn in $(seq 1 4); do
if [[ -f "${COM_OCEAN_RESTART_PREV}/${PDYT}.${cyct}0000.MOM.res_${nn}.nc" ]]; then
${NLN} "${COM_OCEAN_RESTART_PREV}/${PDYT}.${cyct}0000.MOM.res_${nn}.nc" "${DATA}/INPUT/MOM.res_${nn}.nc"
fi
Done

else #not RERUN

local nn
for nn in $(seq 1 4); do
if [[ -f "${COM_OCEAN_RESTART_PREV}/${sPDY}.${scyc}0000.MOM.res_${nn}.nc" ]]; then
${NLN} "${COM_OCEAN_RESTART_PREV}/${sPDY}.${scyc}0000.MOM.res_${nn}.nc" "${DATA}/INPUT/MOM.res_${nn}.nc"
fi
done

fi
`
And add a similar if statement for the CMEPS_postdet and the CICE_postdet

For example:
if [[ ${RERUN} = "YES" ]]; then PDYT="${CDATE_RST:0:8}" cyct="${CDATE_RST:8:2}" cice_restart_file="${COM_ICE_RESTART_PREV}/${PDYT}.${cyct}0000.cice_model.res.nc" else cice_restart_file="${COM_ICE_RESTART_PREV}/${sPDY}.${scyc}0000.cice_model.res.nc" fi

This way the component will look for the same valid time restart files.

I also propose we figure out why the restart files written to MOM6_RESTART in the RUNDIR are not linked in the COMROT dir. For now I manually copied them over. Also noting that the restart dir for ocean and ice has changed to "gdas" even when running without DA, is that intentional?

Happy to share my experiences if it can help improve the workflow.

[aerorahul]

@lisa-bengtsson Is this discussion still relevant?

[aerorahul]

A restart capability was introduced in PR #2510

[aerorahul]

@lisa-bengtsson
Does this thread need further discussion?

[lisa-bengtsson]

Thanks for checking Rahul, you can close this as it's been taken care of now!