Skip to content

RRFS/REFS restart reproducibility issues and crash with DEBUG option #2926

@JiliDong-NOAA

Description

@JiliDong-NOAA

Description

  1. RRFS/REFS can't reproduce bitwise identical forecasts in restart runs to the continuous forecasts
  2. REFS ensemble forecasts crashed when compiling with DEBUG

To Reproduce:

  1. Run RRFS control continuous forecast from the warm start
  2. Run RRFS restart forecast initialized from the control restart
  3. compare the history netcdf from step 1 and 2 and they are not bitwise identical

Also:

  1. compile RRFS with DEBUG on
  2. run REFS ensemble forecast
  3. it will crash right away

Additional context

The restart reproducibility issues are caused by:

  1. RRFS smoke/dust components
  2. HAILCAST variables updraft duration and mask not written out and read in the restart runs
  3. snow equivalent water accumulation not written out to the restart file
  4. saSAS convection initialization logic (i.e. qadv) needs to be corrected
  5. Grell-Freitas convection initialization logic needs to be corrected (i.e. cold starting T/q tendency only applied in the first timestep)
  6. REFS ensemble restart reproducibility issues when running with 32 bit physics (SPP related variable names mismatch and data type precision inconsistency)

The crash with DEBUG is caused by:

  1. 0 index error under DEBUG mode for smcmin/smcmax(stype) when stype=0 over water
  2. floating overflow error under DEBUG mode when applying LSM-SPP to zorll where zorll would have missing values over water/ice (9x10e30)

The fix is #2925

Output

Metadata

Metadata

Labels

bugSomething isn't working

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions