unit and functional testing fail to run on `derecho`

[glemieux]

Describe the issue

Attempting to run either unit or functional tests on dececho with fates results in an xml error referencing the attribute gpu_offload. I confirmed that I'm using the ctsm_pylib environment built with the yml file associated with the ctsm tag I am testing this with.

Relevant log output

(ctsm_pylib) ✔  ~/ctsm/src/fates/testing [sci.1.84.0_api.40.0.0|⚑ 25] 
10:15 $ python run_unit_tests.py 
/glade/u/home/glemieux/ctsm/src/fates/_build/env_mach_specific.xml already exists, delete to replace
ERROR: Command: '/glade/work/glemieux/conda-envs/ctsm_pylib/bin/xmllint --xinclude --noout --schema /glade/u/home/glemieux/ctsm/cime/CIME/data/config/xml_schemas/env_mach_specific.xsd /glade/u/home/glemieux/ctsm/src/fates/_build/env_mach_specific.xml' failed with error '/glade/u/home/glemieux/ctsm/src/fates/_build/env_mach_specific.xml:63: Schemas validity error : Element 'modules', attribute 'gpu_offload': The attribute 'gpu_offload' is not allowed.
/glade/u/home/glemieux/ctsm/src/fates/_build/env_mach_specific.xml fails to validate' from dir '/glade/u/home/glemieux/ctsm/src/fates/_build'

FATES tag

sci.1.84.0_api.40.0.0

Host land model tag

ctsm5.3.063

Machine

derecho

Other supported machine name

No response

Additional context

This came up in the context of testing #1426.

Relevant line of ~/ctsm/src/fates/_build/env_mach_specific.xml

 62     </modules>
 63     <modules mpilib="mpich" compiler="nvhpc" gpu_offload="!none">
 64       <command name="load">cuda/12.2.1</command>
 65     </modules>

I'm not sure if this specific build xml file is procedurally generated or should be a copy of the specific machine xml file.

I confirmed that ccs_config is on the default tag ccs_config_cesm1.0.43.

[glemieux]

Attempting to run this using the default tags on a fresh clone results in a different errors to than the one noted above, although it appears that passed the step dealing with xml configurations. Based on this I was able to replicate this new error using the main branch on my original clone, by removing the old _build directory in the fates base directory.

The first few lines of the new error:

15:14 $ python run_unit_tests.py 
/glade/u/home/glemieux/ctsm-fresh/src/fates/_build/env_mach_specific.xml already exists, delete to replace
Running make for all tests.
ERROR: Command: 'make -j 8' failed with error '[  1%] Built target generate_shr_infnan_mod
[  2%] Built target generate_shr_assert_mod
[ 21%] Built target csm_share
[ 22%] Building Fortran object CMakeFiles/fates.dir/main/EDPftvarcon.F90.o
ifort: remark #10448: Intel(R) Fortran Compiler Classic (ifort) is now deprecated and will be discontinued late 2024. Intel recommends that customers transition now to using the LLVM-based Intel(R) Fortran Compiler (ifx) for continued Windows* and Linux* support, new language support, new language features, and optimizations. Use '-diag-disable=10448' to disable this message.
ifort: command line warning #10121: overriding '-march=core-avx2' with '-march=core-avx2'
ifort: remark #10397: optimization reports are generated in *.optrpt files in the output location
/glade/u/home/glemieux/ctsm-fresh/src/fates/main/EDPftvarcon.F90(16): error #7002: Error in opening the compiled module file.  Check INCLUDE paths.   [LEAFBIOPHYSICSMOD]
  use LeafBiophysicsMod, only : lb_params
------^
...

[glemieux]

Testing this on #1426 (comment) after clearing the _build and _run directories resulted in both test types running successfully.

Looking back closer on the original issue the line /glade/u/home/glemieux/ctsm/src/fates/_build/env_mach_specific.xml already exists, delete to replace should have been the key that this was the necessary fix. Closing this.