Unexpected diffs in RH among patches at a single site

[samsrabin]

Describe the issue

As part of my forest edge flammability work, I discovered diffs when using each patch's climate forcings instead of always taking the values from the oldest vegetated patch (index iofp) here in UpdateFireWeather():

fates/fire/SFMainMod.F90

Lines 119 to 121 in 27a5b40

	precip = bc_in%precip24_pa(iofp)*sec_per_day
	rh = bc_in%relhumid24_pa(iofp)
	wind = bc_in%wind24_pa(iofp)

All three showed non-negligible diffs, but bc_in%precip24_pa and bc_in%wind24_pa only showed diffs in a few diagnostic variables related to fire, whereas bc_in%relhumid24_pa showed differences in variables like FATES_VEGC and many more.

This led me to check whether relative humidity from bc_in%relhumid24_pa varies among patches at a single site in canonical FATES (modified to add log output). It seems like the answer is yes.

I can't figure out why—it seems like neither that variable nor any of the patch-level variables it's derived from are modified within FATES or CTSM.

Relevant log output

Max RH diff between patches at a site:    99.9999999999999

FATES tag

sci.1.87.0_api.41.0.0-1-g4878104a

Host land model tag

ctsm5.3.075-1-g670aa3337

Machine

derecho

Additional context

Tested by examining output from this check in my rh-varies-by-patch branch using the test SMS_Lm6.f45_f45_mg37.I2000Clm50FatesCruRsGs.derecho_intel.clm-Fates.

[samsrabin]

Running the entire fates suite now to see if some tests don't have this issue—if so, that might be a clue.

[samsrabin]

Here are all the fates suite tests with the issue:

ERI_D_Ld20.f10_f10_mg37.I2000Clm50Fates.derecho_intel.clm-FatesCold
ERI_D_Ld20.f45_f45_mg37.I2000Clm50FatesRs.derecho_intel.clm-FatesColdTwoStream
ERI_Ld60.f45_f45_mg37.I2000Clm50FatesCruRsGs.derecho_intel.clm-Fates
ERP_D_Ld3.f19_g17.I2000Clm50FatesCruRsGs.derecho_intel.clm-FatesCold
ERP_D_P128x2_Ld3.f19_g17.I2000Clm50FatesCru.derecho_intel.clm-FatesCold
ERP_Ld3.f09_g17.I2000Clm50FatesRs.derecho_intel.clm-FatesCold
ERP_Ld9.f45_f45_mg37.I2000Clm50FatesCruRsGs.derecho_intel.clm-FatesColdAllVars
ERS_D_Ld20.5x5_amazon.I2000Clm50FatesRs.derecho_gnu.clm-FatesColdSeedDisp
ERS_D_Ld20.f45_f45_mg37.I2000Clm50FatesRs.derecho_intel.clm-FatesColdTreeDamage
ERS_D_Ld20.f45_f45_mg37.I2000Clm60FatesRs.derecho_gnu.clm-FatesColdElecTransJB
ERS_D_Ld30.f45_f45_mg37.HIST_DATM%CRUv7_CLM50%FATES_SICE_SOCN_SROF_SGLC_SWAV_SESP.derecho_intel.clm-FatesColdLandUse
ERS_D_Ld30.f45_f45_mg37.I2000Clm50FatesCruRsGs.derecho_intel.clm-FatesColdPRT2
ERS_D_Ld3.f19_g17.I2000Clm50FatesCruRsGs.derecho_gnu.clm-FatesCold
ERS_D_Ld3.f19_g17.I2000Clm50FatesCruRsGs.derecho_intel.clm-FatesCold
ERS_D_Ld5.f10_f10_mg37.I2000Clm50Fates.derecho_intel.clm-FatesCold
ERS_D_Ld5.f45_f45_mg37.I2000Clm50FatesRs.izumi_nag.clm-FatesCold
ERS_Ld30.f45_f45_mg37.I2000Clm50FatesCruRsGs.derecho_intel.clm-FatesColdFixedBiogeo
ERS_Ld30.f45_f45_mg37.I2000Clm50FatesCruRsGs.derecho_intel.clm-FatesColdNoCompFixedBioGeo
ERS_Ld30.f45_f45_mg37.I2000Clm50FatesCruRsGs.derecho_intel.clm-FatesColdSizeAgeMort
ERS_Ld397.f10_f10_mg37.I2000Clm50Fates.derecho_gnu.clm-FatesCold
ERS_Ld397.f45_f45_mg37.I2000Clm50Fates.derecho_intel.clm-FatesColdNoComp
ERS_Ld5.f19_g17.I2000Clm45Fates.derecho_intel.clm-FatesCold
ERS_Ld60.f45_f45_mg37.I2000Clm50FatesCruRsGs.derecho_intel.clm-FatesColdLogging
ERS_Ld60.f45_f45_mg37.I2000Clm50FatesCruRsGs.derecho_intel.clm-FatesColdPPhys
ERS_Ld60.f45_f45_mg37.I2000Clm60FatesRs.derecho_intel.clm-FatesColdManagedFire
ERS_Ld9.f10_f10_mg37.I2000Clm50FatesCruRsGs.derecho_intel.clm-FatesColdCH4Off
ERS_P128x1_Ld765.f10_f10_mg37.I2000Clm60Fates.derecho_intel.clm-FatesColdNoComp
SMS_Lm6.f45_f45_mg37.I2000Clm50FatesCruRsGs.derecho_intel.clm-Fates

And seemingly without the issue:

ERP_P128x2_Ld30.f45_f45_mg37.I2000Clm60FatesSpCruRsGs.derecho_intel.clm-FatesColdSatPhen
ERS_D_Ld5.1x1_brazil.I2000Clm50FatesCruRsGs.derecho_intel.clm-FatesColdHydro
ERS_D_Ld5.1x1_brazil.I2000Clm50FatesCruRsGs.izumi_nag.clm-FatesColdHydro
ERS_D_Ld5.f19_g17.I2000Clm50BgcCru.derecho_intel.clm-default
ERS_D_Mmpi-serial_Ld5.1x1_brazil.I2000Clm50FatesCruRsGs.derecho_gnu.clm-FatesCold
ERS_D_Mmpi-serial_Ld5.1x1_brazil.I2000Clm50FatesCruRsGs.izumi_nag.clm-FatesCold
ERS_Ld30.f45_f45_mg37.I2000Clm50FatesCruRsGs.derecho_intel.clm-FatesColdNoComp
ERS_Ld30.f45_f45_mg37.I2000Clm60FatesSpCruRsGs.derecho_intel.clm-FatesColdSatPhen
ERS_Ld366.1x1_brazil.I2000Clm50FatesCruRsGs.derecho_intel.clm-FatesFireLightningPopDens
ERS_Ld60.f45_f45_mg37.I2000Clm50FatesCruRsGs.derecho_intel.clm-FatesColdNoFire
ERS_Ld60.f45_f45_mg37.I2000Clm50FatesCruRsGs.derecho_intel.clm-FatesColdPPhys
ERS_Ld60.f45_f45_mg37.I2000Clm50FatesCruRsGs.derecho_intel.clm-FatesColdST3
SMS_D.1x1_brazil.I2000Clm60FatesSpCruRsGs.derecho_gnu.clm-FatesColdSatPhen
SMS.f45_f45_mg37.I2000Clm60FatesSpRsGs.derecho_nvhpc.clm-FatesColdSatPhen
SMS_Ld10_D_Mmpi-serial.CLM_USRDAT.I1PtClm60Fates.derecho_gnu.clm-FatesPRISM--clm-NEON-FATES-YELL
SMS_Ld10_D_Mmpi-serial.CLM_USRDAT.I1PtClm60Fates.izumi_nag.clm-FatesPRISM--clm-NEON-FATES-YELL
SMS_Ld10_D_Mmpi-serial.CLM_USRDAT.I1PtClm60Fates.derecho_intel.clm-FatesFireLightningPopDens--clm-NEON-FATES-NIWO
SMS_Ld10_D_Mmpi-serial.CLM_USRDAT.I1PtClm60Fates.izumi_nag.clm-FatesFireLightningPopDens--clm-NEON-FATES-NIWO
SMS_Lm13.1x1_brazil.I2000Clm50FatesCruRsGs.derecho_gnu.clm-FatesCold
SMS_Lm13.1x1_brazil.I2000Clm50FatesCruRsGs.derecho_intel.clm-FatesCold
SMS_Lm1.f45_f45_mg37.I2000Clm60FatesSpCruRsGs.derecho_intel.clm-FatesColdBasic
SMS_Lm3_D_Mmpi-serial.1x1_brazil.I2000Clm50FatesCruRsGs.derecho_intel.clm-FatesColdHydro
SMS_Lm3_D_Mmpi-serial.1x1_brazil.I2000Clm50FatesCruRsGs.izumi_nag.clm-FatesColdHydro

Here are compsets of runs with the issue:

HIST_DATM%CRUv7_CLM50%FATES_SICE_SOCN_SROF_SGLC_SWAV_SESP
I2000Clm45Fates
I2000Clm50Fates
I2000Clm50FatesCru
I2000Clm50FatesCruRsGs
I2000Clm50FatesRs
I2000Clm60Fates
I2000Clm60FatesRs

And compsets of runs seemingly without the issue:

I1PtClm60Fates
I2000Clm50BgcCru
I2000Clm50FatesCruRsGs
I2000Clm60FatesSpCruRsGs
I2000Clm60FatesSpRsGs

I2000Clm50FatesCruRsGs shows up on both. I'm not sure there's anything to be gleaned from this.

[rgknox]

I wonder if this has something to do with the bare-ground patch.

Also, since you are using 24-hour averages... If fates performs a re-shuffle of its patches mid-way through the 24 hour averaging cycle, this would explain it. FATES is supposed to do dynamics on the first time-step of the new day, and if I recall correctly, the 24-hour averaging updater is called after it. So it would seemingly not be a problem, but maybe something subtle is going on there.

[mvdebolskiy]

Have issue:

ERS_Ld30.f45_f45_mg37.I2000Clm50FatesCruRsGs.derecho_intel.clm-FatesColdFixedBiogeo
ERS_Ld30.f45_f45_mg37.I2000Clm50FatesCruRsGs.derecho_intel.clm-FatesColdNoCompFixedBioGeo
ERS_Ld397.f45_f45_mg37.I2000Clm50Fates.derecho_intel.clm-FatesColdNoComp

Don't have issue:

ERS_Ld30.f45_f45_mg37.I2000Clm50FatesCruRsGs.derecho_intel.clm-FatesColdNoComp
+ all SatPhen tests.

The NoComp test with Ld30 passing and Ld397 makes little sense to me.

[samsrabin]

@rgknox It's not a true running mean, which would require saving the value for the 48 timesteps in the past 24 hours, popping the oldest off and appending the newest at every timestep. Instead, it's

$$ RH24_t = RH24_{t-1} \times \frac{47}{48} + RH_t \times \frac{1}{48} $$

[samsrabin]

@mvdebolskiy:

The NoComp test with Ld30 passing and Ld397 makes little sense to me.

Could there be something happening at the end/beginning of the year that causes the problem?

[samsrabin]

Based on our conversation at the FATES SE meeting today, I thought the differences might be due to the bare ground patch never getting set, since this ifp loop in CLMFatesInterfaceMod dynamics_driv() starts at 1 instead of 0:

do ifp = 1, this%fates(nc)%sites(s)%youngest_patch%patchno !for vegetated patches
   ! Mapping between  IFP space (1,2,3) and HLM P space (looping by IFP)
   p = ifp+col%patchi(c)

   this%fates(nc)%bc_in(s)%precip24_pa(ifp) = &
         wateratm2lndbulk_inst%prec24_patch(p)

   this%fates(nc)%bc_in(s)%relhumid24_pa(ifp) = &
         wateratm2lndbulk_inst%rh24_patch(p)

   this%fates(nc)%bc_in(s)%wind24_pa(ifp) = &
         atm2lnd_inst%wind24_patch(p)

end do

Unfortunately, that doesn't appear to be the issue. I see equally large RH differences when excluding bare-ground patches.

The issue seems to be CLM-side, though, with this check showing the same RH differences across patches in wateratm2lndbulk_inst%rh24_patch that should all be on the same column.

I'm now suspicious that the indexing is wrong at the updating of RH24.

[samsrabin]

That was it! This change resolves the differences, at least in the ERS_D_Ld5.f10_f10_mg37.I2000Clm50Fates.derecho_intel.clm-FatesCold test.

I will file a CTSM issue and link it to this one.