ELM-FATES CNP mode `mass balance error` due to `sci.1.80.0_api.37.0.0`

[glemieux]

Describe the issue

During testing for E3SM-Project/E3SM#7027 it was discovered that the fates_eca and fates_rd tests are failing RUN. Spot testing using the latest e3sm master branch commit and working forwards through the fates tags resulted in discovering that the changes associated with sci.1.80.0_api.37.0.0 trigger the failure mode.

Relevant log output

61:  mass balance error detected
 61:  element type (see PRTGenericMod.F90):            1
 61:  error fraction relative to biomass stock:   1.118038134966836E-005
 61:  absolut error (flux in - change):   2.876421946683116E-002
 61:  call index:            1
 61:  Element index (PARTEH global):           1
 61:  net:    11.1017546883988
 61:  dstock:    11.0729904689319
 61:  seed_in:   0.000000000000000E+000
 61:  net_root_uptake:   0.000000000000000E+000
 61:  gpp_acc:    16.7568276755855
 61:  flux_generic_in:   0.000000000000000E+000
 61:  wood_product_harvest:   0.000000000000000E+000  0.000000000000000E+000
 61:   0.000000000000000E+000  0.000000000000000E+000  0.000000000000000E+000
 61:   0.000000000000000E+000  0.000000000000000E+000  0.000000000000000E+000
 61:   0.000000000000000E+000  0.000000000000000E+000  0.000000000000000E+000
 61:   0.000000000000000E+000  0.000000000000000E+000  0.000000000000000E+000
 61:   0.000000000000000E+000  0.000000000000000E+000
 61:  wood_product_landusechange:   0.000000000000000E+000  0.000000000000000E+000
 61:   0.000000000000000E+000  0.000000000000000E+000  0.000000000000000E+000
 61:   0.000000000000000E+000  0.000000000000000E+000  0.000000000000000E+000
 61:   0.000000000000000E+000  0.000000000000000E+000  0.000000000000000E+000
 61:   0.000000000000000E+000  0.000000000000000E+000  0.000000000000000E+000
 61:   0.000000000000000E+000  0.000000000000000E+000
 61:  error from patch resizing:   0.000000000000000E+000
 61:  burn_flux_to_atm:   0.000000000000000E+000
 61:  seed_out:   0.000000000000000E+000
 61:  flux_generic_out:   0.000000000000000E+000
 61:  frag_out:   3.251976394669682E-003
 61:  aresp_acc:    5.65182101079208
 61:  error=net_flux-dstock:  2.876421946683116E-002
 61:  biomass   2569.79932769763
 61:  litter   2.94113322451802
 61:  seeds  0.000000000000000E+000
 61:  total stock   2572.74046092215
 61:  previous total   2561.66747045322
 61:  lat lon  -34.0000000000000        145.000000000000

FATES tag

sci.1.80.0_api.37.0.0

Host land model tag

v3.0.1-2206-g2b545d3f6a

Machine

perlmutter

Other supported machine name

No response

Additional context

No response

[ckoven]

@glemieux I have no idea if this is relevant to the tests that are failing, but I found in the runs I am doing that I had to add some harvest carbon fluxes to the carbon balance check accounting in order to pass balance checks when harvest is active (via logging and/or land use change): ckoven/E3SM@6a011c6

[rgknox]

I think either @glemieux or @ckoven identified that this bug came in with the code changes to respiration at the daily timestep. In this change I had to update code that handled "excess respiration" that is triggered when the plants have nutrient limitations. Due to the timing of the bug and the nature of the failure (cnp test), I think the problem is associated with those changes.

I'm flagging this as a priority and will investigate this shortly.

[rgknox]

I was able to get the test passing by removing the term that accounts for "excess respiration". This flux is generated when nutrients are limiting growth and the plant can no longer hold carbon, and subsequently burns it off. I was able to turn this off by setting a flag that will alternatively hold and accumulate storage carbon indefinitely instead of respiring it off. (this is not a reasonable assumption, but proved this is the issue).

The next step is to figure out why our book-keeping of this term is not working.