Skip to content

Conversation

@slevis-lmwg
Copy link
Contributor

@slevis-lmwg slevis-lmwg commented Aug 11, 2025

Description of changes

For now see the issue #3378

Specific notes

Contributors other than yourself, if any:
@rgknox @adrifoster @wwieder

CTSM Issues Fixed (include github issue #):
#3378

Are answers expected to change (and if so in what way)?

Any User Interface Changes (namelist or namelist defaults changes)?

Does this create a need to change or add documentation? Did you do so?

Testing performed
...with the first two commits in this PR:

PASS ERS_D_Ld30.1x1_brazil.I2000Clm60FatesCrujraRs.derecho_intel.clm-FatesColdPRT2
PASS ERS_D_Ld30.1x1_brazil.I2000Clm60FatesCrujraRs.derecho_intel.clm-FatesCold

Later comments point out that these two tests were inadequate at catching problems, and that I switched to two other tests.

PASS ERS_D_Ld30.1x1_brazil.I2000Clm60FatesCrujraRs.derecho_intel.clm-FatesColdPRT2
FAIL ERS_D_Ld30.1x1_brazil.I2000Clm60FatesCrujraRs.derecho_intel.clm-FatesColdPRT2--clm-mimicsFatesCold--clm-nofireemis
The latter needs "nofireemis" to work with Fates, but it then dumps core in line 1180 SoilBiogeochemDecompCascadeMIMICSMod.F90
...calculating these variables:
nf_soil%decomp_npools_sourcesink_col
nf_soil%fates_litter_flux
@slevis-lmwg slevis-lmwg self-assigned this Aug 11, 2025
@slevis-lmwg slevis-lmwg added enhancement new capability or improved behavior of existing capability investigation Needs to be verified and more investigation into what's going on. science Enhancement to or bug impacting science test: aux_clm Pass aux_clm suite before merging test: fates Pass fates test suite before merging labels Aug 11, 2025
@slevis-lmwg slevis-lmwg linked an issue Aug 11, 2025 that may be closed by this pull request
1 task
@slevis-lmwg
Copy link
Contributor Author

slevis-lmwg commented Aug 13, 2025

With the latest commit, I repeated the two earlier tests and added another to check whether answers have changed from the baseline:

PASS ERS_D_Ld30.1x1_brazil.I2000Clm60FatesCrujraRs.derecho_intel.clm-FatesColdPRT2
PASS ERS_D_Ld30.1x1_brazil.I2000Clm60FatesCrujraRs.derecho_intel.clm-FatesCold
FAIL ERP_Ld9.f45_f45_mg37.I2000Clm50FatesRs.derecho_intel.clm-FatesColdAllVars -c /glade/campaign/cgd/tss/ctsm_baselines/ctsm5.3.065

The latter fails in case2, after reading the restart file, with a N balance error.

UPDATE

  • See below for suggestions to resolve FAIL
  • Add new test ERP_Ld9.f45_f45_mg37.I2000Clm50FatesRs.derecho_intel.clm-FatesColdPRT2 (sanity check first that it meets requirements set forth in the fourth checkbox here, which I think means pointing to an alternate fates paramfile)
  • Later comment explains why I decided to revert to preexisting ERS tests and skip adding this ERP test.

! to scale up to column
plant_ndemand(c) = 0._r8

! We fill the vertically resolved array to simplify some jointly used code
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Put the if fates_parteh_mode == 2 here and make the else set plant_ndemand(_vr) = 0 to hopefully resolve the N balance error

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Likely still need to resolve N bal. error when parteh == 2

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The 3rd test still fails with a N-bal error, and I'm running the 4th test to see what happens:

  • I still see indications of badly initialized variables...
  • Try uncommenting the harvest lines mentioned in an earlier post
  • Consider inconsistent if-statements

Copy link
Contributor Author

@slevis-lmwg slevis-lmwg Aug 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rethinking my tests:

  1. The 1x1 tests have not helped (I have now removed them)
  2. The ERP...PRT2 test fails with
    FAILED to get /glade/derecho/scratch/slevis/ERP_Ld9.f45_f45_mg37.I2000Clm50FatesRs.derecho_intel.clm-FatesColdPRT2.20250814_175600_rzyi7o/case2/ERP_Ld9.f45_f45_mg37.I2000Clm50FatesRs.derecho_intel.clm-FatesColdPRT2.20250814_175600_rzyi7o/fates_params_prt2_prescribed_np.nc
    so I will go back and try the preexisting PRT2 test and the closest non-PRT2:
./create_test ERS_D_Ld30.f45_f45_mg37.I2000Clm50FatesCruRsGs.derecho_intel.clm-FatesColdPRT2 -c /glade/campaign/cgd/tss/ctsm_baselines/fates-sci.1.84.0_api.40.0.0-ctsm5.3.066
./create_test ERS_D_Ld30.f45_f45_mg37.I2000Clm50FatesCruRsGs.derecho_intel.clm-FatesColdLUH2 -c /glade/campaign/cgd/tss/ctsm_baselines/fates-sci.1.84.0_api.40.0.0-ctsm5.3.066
  • before this PR's changes: caseids 20250815_113417_sx4t8g and 20250815_113743_mr25bk
  • with 01671b9: caseids 20250815_114817_ilbscj and 20250815_114840_il903r
  • FAIL with the latest commit:
  • testid 20250815_120848_v9vgmh gives this Traceback <-- TROUBLESHOOTING THIS CASE
error (65): floating invalid
Image              PC                Routine            Line        Source
libpthread-2.31.s  0000149243D098C0  Unknown               Unknown  Unknown
cesm.exe           0000000002F2D454  soilbiogeochemcom         608  SoilBiogeochemCompetitionMod.F90
cesm.exe           0000000003FCD516  cndrivermod_mp_cn         474  CNDriverMod.F90
cesm.exe           0000000001A99BF1  cnvegetationfacad        1030  CNVegetationFacade.F90
cesm.exe           0000000000A8B2C7  clm_driver_mp_clm        1058  clm_driver.F90

UPDATE I had placed the new code in the if (.not. use_nitrif_denitrif) opposite to what I should have done. I have now updated the code (while still keeping the new code in the ".not." section). Testing PASSed 20250825_174700_7w8gx9

  • testid 20250815_120902_1msot7 gives the same Traceback, but also before the Traceback this error
Nan values for pftareafrac. dumping site info.
----------------------------------------
Site Coordinates
----------------------------------------
latitude                    =   -74.0000000000000
longitude                   =    255.000000000000

Same test PASSed with the above update 20250826_101645_52b4zd

  • Tests now work, though should they DIFF from the baseline?

@slevis-lmwg
Copy link
Contributor Author

slevis-lmwg commented Aug 26, 2025

Enabled fixation and ran the same tests:

PASS ERS_D_Ld30.f45_f45_mg37.I2000Clm50FatesCruRsGs.derecho_intel.clm-FatesColdPRT2 -c /glade/campaign/cgd/tss/ctsm_baselines/fates-sci.1.84.0_api.40.0.0-ctsm5.3.066
PASS ERS_D_Ld30.f45_f45_mg37.I2000Clm50FatesCruRsGs.derecho_intel.clm-FatesColdLUH2 -c /glade/campaign/cgd/tss/ctsm_baselines/fates-sci.1.84.0_api.40.0.0-ctsm5.3.066

PRT2 still b4b with the baseline.
LUH2 now DIFF from the baseline.

PASS The same tests with the code change for harvest (same results relative to the baseline).

@slevis-lmwg
Copy link
Contributor Author

slevis-lmwg commented Sep 20, 2025

Worked on the next checkbox in the issue (#3378), submitted the same tests, and after some troubleshooting:
PASS PRT2 and b4b with the baseline.
PASS LUH2 and DIFF from the baseline as before.

These variables originate in fates, so this renaming requires the same
renaming in fates; I will open the corresponding PR very soon
@slevis-lmwg
Copy link
Contributor Author

slevis-lmwg commented Sep 23, 2025

OK ./build-namelist_test.pl
OK ./run_sys_tests -s fates -c fates-sci.1.84.0_api.40.0.0-ctsm5.3.066/ --skip-generate
REDO ./run_sys_tests -s aux_clm -c ctsm5.3.066 --skip-generate

Notes:

  • Gnu and nvhpc tests needed a bug-fix that intel didn't catch (next commit).
  • Then the fates test-suite worked (I didn't repeat aux_clm for now)
  • Many tests DIFFer from the baseline.
  • The PRT2 test had the expected NLCOMP change, though no DIFFs from the baseline (even after rebuilding/rerunning):
  BASE: suplnitro = 'ALL'
  COMP: suplnitro = 'NONE'

@slevis-lmwg
Copy link
Contributor Author

slevis-lmwg commented Sep 23, 2025

Updated the fates paramfile (see next commit) and submitted these two again

./create_test ERS_D_Ld30.f45_f45_mg37.I2000Clm50FatesCruRsGs.derecho_intel.clm-FatesColdLUH2 -c /glade/campaign/cgd/tss/ctsm_baselines/fates-sci.1.84.0_api.40.0.0-ctsm5.3.066
./create_test ERS_D_Ld30.f45_f45_mg37.I2000Clm50FatesCruRsGs.derecho_intel.clm-FatesColdPRT2 -c /glade/campaign/cgd/tss/ctsm_baselines/fates-sci.1.84.0_api.40.0.0-ctsm5.3.066

The first (LUH2) same as before (DIFF from baseline) since nothing changed for it.
The second (ERS_D_Ld30.f45_f45_mg37.I2000Clm50FatesCruRsGs.derecho_intel.clm-FatesColdPRT2.C.20250923_141838_qjxqou) now:

  • DIFF from baseline and
  • FAIL COMPARE_base_rest suggesting variable(s) missing from restart, I suspect

@slevis-lmwg
Copy link
Contributor Author

Submitted the same 2 tests, now with the error check suggested in NGEET/fates#1454 (see corresponding commit NGEET/fates@97839f1 in NGEET/fates#1472)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement new capability or improved behavior of existing capability investigation Needs to be verified and more investigation into what's going on. science Enhancement to or bug impacting science test: aux_clm Pass aux_clm suite before merging test: fates Pass fates test suite before merging

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Completing the FATES-CLM nitrogen coupling

1 participant