FATES doesn't work in branch runs

[samsrabin]

When trying to use FATES as part of a branch run, I get the following error:

forrtl: severe (151): allocatable array is already allocated
Image              PC                Routine            Line        Source
cesm.exe           0000000000983A08  edinitmod_mp_init         144  EDInitMod.F90
cesm.exe           00000000006403F4  clmfatesinterface        2089  clmfates_interfaceMod.F90
cesm.exe           00000000006063A6  clm_initializemod         757  clm_initializeMod.F90
cesm.exe           00000000005A8EA3  lnd_comp_nuopc_mp         658  lnd_comp_nuopc.F90

This is with CTSM tag ctsm5.3.009, FATES tag sci.1.78.3_api.36.1.0. It can be reproduced using the test ERI_Ld60.f45_f45_mg37.I2000Clm50FatesCruRsGs.derecho_intel.clm-Fates, which is the ERI version of an ERS test we already run as part of the fates suite.

Note parallel issue at ESCOMP/CTSM#2903.

[jennykowalcz]

Have also encountered this. The less strict hybrid mode does work though.

[XiulinGao]

I did encounter a similar issue when run with FATES API 36 (CTSM 5.2.013). If I want to do a hybrid run using restart file model build will fail warning that a restart file should not be defined when cold start is turned on. But I did turn it off.

[samsrabin]

@XiulinGao That's actually a separate issue that is now fixed in CTSM as of ctsm5.3.011.

[rosiealice]

Hi all. Do we have a plan for addressing this issue? It has suddenly cropped up as an problem in the NorESM workflow

NorESMhub/CTSM#115 (comment)

[samsrabin]

Hi all. Do we have a plan for addressing this issue? It has suddenly cropped up as an problem in the NorESM workflow

NorESMhub/CTSM#115 (comment)

Not as far as I know.

[ckoven]

Do we know why this is the case? could it just be that is_restart() returns .false. for hybrid cases here?

[jennykowalcz]

Hybrid cases do work. I'll try to dig up the errors I got trying a branch case.

[ckoven]

right, sorry, meant to say branch runs above, not hybrid.

[samsrabin]

I'm not sure of the ultimate issue(s), but I included an error message in the CTSM issue; reproducing here for convenience:

forrtl: severe (151): allocatable array is already allocated
Image              PC                Routine            Line        Source
cesm.exe           0000000000983A08  edinitmod_mp_init         144  EDInitMod.F90
cesm.exe           00000000006403F4  clmfatesinterface        2089  clmfates_interfaceMod.F90
cesm.exe           00000000006063A6  clm_initializemod         757  clm_initializeMod.F90
cesm.exe           00000000005A8EA3  lnd_comp_nuopc_mp         658  lnd_comp_nuopc.F90

If you have Derecho access, that test is at /glade/derecho/scratch/samrabin/tests_1028-110314de/ERI_Ld60.f45_f45_mg37.I2000Clm50FatesCruRsGs.derecho_intel.clm-Fates.1028-110314de/. That test was with CTSM tag ctsm5.3.009, FATES tag sci.1.78.3_api.36.1.0.

[jennykowalcz]

Wait.. I feel like I'm losing it here. 🤪 I couldn't find the logs from long ago when I tried to run a branch case. So I took one of my more recent hybrid cases, set RUN_TYPE to branch, refreshed the rpointer files in the run directory, and it seems to be working! As in history files are being produced.
I am using E3SM branch: https://github.com/rgknox/E3SM/tree/rknox/lnd/fates-api34-schwartz-cbal
and this FATES branch: https://github.com/jennykowalcz/fates/tree/jkowalcz-merge-test