EAMxx: output timestamps shifted on first restart #7961
Description
A recent ne256 case on Perlmutter (GPU) experienced this issue that had previously been exhibited by runs on Frontier in which the output timestamps become "shifted" by one output increment. In my case this only happened on the first restart, and then files remained consistent afterwards.
The problem can be easily seen by the change in the time stamp of the file name:
output.scream.3D.6hr.INSTANT.nhours_x6.0001-03-14-00000.nc
output.scream.3D.6hr.INSTANT.nhours_x6.0001-03-15-00000.nc <<< first restart happened here:
output.scream.3D.6hr.INSTANT.nhours_x6.0001-03-15-21600.nc
output.scream.3D.6hr.INSTANT.nhours_x6.0001-03-16-21600.nc
From the history restart files we can see that this was the first restart, while subsequent restarts did not affect the file names
> ls -1 output.scream.3D.6hr.rhist*
output.scream.3D.6hr.rhist.INSTANT.nhours_x6.0001-03-15-00000.nc
output.scream.3D.6hr.rhist.INSTANT.nhours_x6.0001-05-27-00000.nc
output.scream.3D.6hr.rhist.INSTANT.nhours_x6.0001-08-08-00000.nc
output.scream.3D.6hr.rhist.INSTANT.nhours_x6.0001-10-20-00000.nc
output.scream.3D.6hr.rhist.INSTANT.nhours_x6.0002-01-01-00000.nc
By examining the file coordinates we can see that the file written at the time of the restart was not properly reopened upon restarting:
...0001-03-13-00000.nc => time dimension length => 4
...0001-03-14-00000.nc => time dimension length => 4
...0001-03-15-00000.nc => time dimension length => 1
...0001-03-15-21600.nc => time dimension length => 4
...0001-03-16-21600.nc => time dimension length => 4
...0001-03-17-21600.nc => time dimension length => 4
I haven't checked if this is something we can reproduce at lower resolutions.