When trying to build with GNU 14, see runtime error in elm, have work-around

[ndkeen]

On internal NERSC machine (alvarez), there are newer module versions that I can test now before they might be
installed on perlmutter. I'm testing with GNU v14 (and v14.2) and have only encountered one issue so far.
There was a runtime error in controlMod.F90 that I debugged for a bit. I don't see anything obviously wrong on our side.
So it's possible that GNU fortran may be getting confused with symbols -- however, after several hours sessions with gemini, I was unable to determine the issue or create a reproducer -- anything tried builds/runs fine.

In components/elm/src/main/controlMod.F90, I see there are several use module, only:, except there was one that
seemed odd. At top of use module section, we see a blanket use elm_varctl. When I went thru and found all variables needed to replace this with a use elm_varctl, only:, I no longer see the build error. I'm referring to this as a work-around as I'm not sure if it's a problem with compiler or not, but I would think this change is an improvement in terms of SW engineering as it now checks that only those vars are used. With this change, verified e3sm_developer tests passed with GNU 14 (and some other module versions upgraded).

branch: ndk/elm/use-only-variable-cleanup

The original error:

0:  ERROR: Unknown error submitted to shr_abort_abort.
0: #0  0x15507ba6f862 in ???
0: #1  0xdb979a in __shr_abort_mod_MOD_shr_abort_backtrace
0:      at /global/cfs/cdirs/e3sm/ndk/repos/ndkmf-alv-update2026/share/util/shr_abort_mod.F90:104
0: #2  0xdb9969 in __shr_abort_mod_MOD_shr_abort_abort
0:      at /global/cfs/cdirs/e3sm/ndk/repos/ndkmf-alv-update2026/share/util/shr_abort_mod.F90:61
0: #3  0x53ec9d in __controlmod_MOD_control_init
0:      at /global/cfs/cdirs/e3sm/ndk/repos/ndkmf-alv-update2026/components/elm/src/main/controlMod.F90:401
0: #4  0x54ade2 in __elm_initializemod_MOD_initialize1
0:      at /global/cfs/cdirs/e3sm/ndk/repos/ndkmf-alv-update2026/components/elm/src/main/elm_initializeMod.F90:144
0: #5  0x52f525 in __lnd_comp_mct_MOD_lnd_init_mct
0:      at /global/cfs/cdirs/e3sm/ndk/repos/ndkmf-alv-update2026/components/elm/src/cpl/lnd_comp_mct.F90:300
0: #6  0x4857ed in __component_mod_MOD_component_init_cc
0:      at /global/cfs/cdirs/e3sm/ndk/repos/ndkmf-alv-update2026/driver-mct/main/component_mod.F90:258
0: #7  0x474702 in __cime_comp_mod_MOD_cime_init
0:      at /global/cfs/cdirs/e3sm/ndk/repos/ndkmf-alv-update2026/driver-mct/main/cime_comp_mod.F90:1528
0: #8  0x45f1df in cime_driver
0:      at /global/cfs/cdirs/e3sm/ndk/repos/ndkmf-alv-update2026/driver-mct/main/cime_driver.F90:122
0: #9  0x45f1df in main
0:      at /global/cfs/cdirs/e3sm/ndk/repos/ndkmf-alv-update2026/driver-mct/main/cime_driver.F90:23

    if (masterproc) then

       ! ----------------------------------------------------------------------                                                                                                                                                    
       ! Read namelist from standard input.                                                                                                                                                                                        
       ! ----------------------------------------------------------------------                                                                                                                                                    

       if ( len_trim(NLFilename) == 0  )then
          call endrun(msg=' error: nlfilename not set'//errMsg(__FILE__, __LINE__))
       end if
       unitn = getavu()
       write(iulog,*) 'Read in elm_inparm namelist from: ', trim(NLFilename)
       open( unitn, file=trim(NLFilename), status='old' )
       call shr_nl_find_group_name(unitn, 'elm_inparm', status=ierr)
       if (ierr == 0) then
          read(unitn, elm_inparm, iostat=ierr)
          if (ierr /= 0) then
             call endrun(msg='ERROR reading elm_inparm namelist'//errMsg(__FILE__, __LINE__))  !<-- line 401
          end if
       end if

It's during the processing of the &elm_inparm section of lnd_in. Example of var it has trouble with is nu_con.
When I made a local nu_con_local, read in, copied back to nu_con, that worked, but another var caused similar
issue.

[ndkeen]

If anything pasting this here to remind myself of some of these notes. Note that variable nu_con was just the first variable with issue, there were others after that.

1. The Original Problem

The issue originated as a runtime namelist read error (5010) specifically when processing the string variable nu_com. While the model compiled, it crashed upon execution when trying to ingest the nu_com value from the namelist file. This suggested a memory corruption or "symbol merging" issue where the compiler was not providing a unique memory boundary or null-terminator for that specific string.

2. What We Tried

We attempted several strategies to isolate nu_com from the global module elm_varctl:

Rename and Equivalence: We tried to import the module while renaming the global variable (use elm_varctl, nu_com_original => nu_com) and declaring a local character(len=16) :: nu_com to act as a proxy.

Equivalence Attribute Conflict: This led to a persistent compiler error: EQUIVALENCE attribute conflicts with USE ASSOCIATED attribute in 'nu_com'. Gfortran 14 refused to allow nu_com to be both a local variable and a "USE associated" name simultaneously.

Recursive Header Cleaning: We systematically identified and commented out redundant use elm_varctl statements that were re-importing the original namespace and re-triggering the conflict.

3. What We Verified

Binary Integrity: We verified that nu_com was indeed entering the module from elm_varctl and that its definition was sensitive to how the module was imported.

Scope Requirements: We confirmed that controlMod.F90 has massive dependencies on elm_varctl, requiring 117 specific symbols to satisfy namelist declarations, MPI broadcasts, and logical checks.

4. Why We Were Unable to Create a Reproducer

We could not replicate the specific runtime crash in a small-scale standalone script because the error appears to be a side-effect of scale. The 5010 error likely stems from how Gfortran 14 manages the string table when a module with hundreds of variables is imported via a "blanket" USE statement. In a small script, memory layout is clean; in the full ELM model, the blanket import likely caused nu_com to be "shadowed" or merged into the memory of an adjacent symbol.

5. What the Issue Might Be

The evidence suggests a Namespace/Metadata Leak in Gfortran 14. Even when a variable is renamed during a blanket USE, the compiler may still reserve the original name's metadata in the background. If any other part of the file (or a subroutine) re-opens that module, the "renamed" name and "original" name become entangled, causing the EQUIVALENCE error and the runtime namelist corruption.

6. Why Your "Exhaustive ONLY" Workaround Works

By manually listing all 117 variables and removing the blanket use elm_varctl statement, you have implemented Strict Namespace Isolation.

• Hard Exclusion: By using ONLY and explicitly omitting the name nu_com, you legally bar the compiler from associating that name with the parent module.

Unique Memory Allocation: This forces the compiler to treat nu_com as a purely local variable with its own distinct memory address and null-terminator, resolving the 5010 read error.

• Standard Compliance: This follows modern software practices by explicitly defining dependencies, which prevents the "leaks" that occur with blanket imports in modern, stricter compilers.

Might also check iulog outout...