Error running eclm coupled with parflow

[AGonzalezNicolas]

job_error.txt

I am running eclm coupled with parflow. I am using the EUR-11 test case.
The simulation runs since the eclm and parflow outputs are generated. However, in the log.error file, an error is registered:

corrupted double-linked list
[jrc0189:1249806] *** Process received signal ***
corrupted double-linked list
[jrc0189:1249806] Signal: Aborted (6)
[jrc0189:1249806] Signal code:  (-6)
[jrc0189:1249641] *** Process received signal ***
[jrc0189:1249641] Signal: Aborted (6)
[jrc0189:1249641] Signal code:  (-6)
corrupted double-linked list
[jrc0189:1249851] *** Process received signal ***
corrupted double-linked list
corrupted double-linked list
corrupted double-linked list

and in the log.out file there is this other error:

(   94)  jrc0096
(   95)  jrc0096
 NetCDF: Invalid dimension ID or name
 NetCDF: Invalid dimension ID or name
 NetCDF: Invalid dimension ID or name
 NetCDF: Invalid dimension ID or name
 NetCDF: Invalid dimension ID or name
 NetCDF: Variable not found
 NetCDF: Invalid dimension ID or name
 NetCDF: Invalid dimension ID or name
 NetCDF: Invalid dimension ID or name
 NetCDF: Invalid dimension ID or name
 NetCDF: Invalid dimension ID or name
 NetCDF: Invalid dimension ID or name
 NetCDF: Invalid dimension ID or name
 NetCDF: Invalid dimension ID or name
 NetCDF: Invalid dimension ID or name
 NetCDF: Invalid dimension ID or name
 NetCDF: Variable not found
 NetCDF: Invalid dimension ID or name
 NetCDF: Invalid dimension ID or name
 NetCDF: Invalid dimension ID or name
 NetCDF: Invalid dimension ID or name
 NetCDF: Invalid dimension ID or name

 proc=            1  beg gridcell=         1083  end gridcell=         2164  total gridcells per proc=         1082

[AGonzalezNicolas]

When building with the DEBUG option and running the same test case the following error appears:

+ srun --multi-prog ./eclm-parflow_mapping.conf

Program received signal SIGFPE: Floating-point exception - erroneous arithmetic operation.

Backtrace for this error:

Program received signal SIGFPE: Floating-point exception - erroneous arithmetic operation.

Backtrace for this error:
#0  0x153f6f7e372f in ???
#0  0x14e9660e172f in ???
#1  0x6fc6af in ch4_tran
	at /p/scratch/cslts/gonzalez5/TSMP2/JURECA_tests/TSMP2_S2025_master/models/eCLM/src/clm5/biogeochem/ch4Mod.F90:3555
#2  0x72622f in __ch4mod_MOD_ch4
	at /p/scratch/cslts/gonzalez5/TSMP2/JURECA_tests/TSMP2_S2025_master/models/eCLM/src/clm5/biogeochem/ch4Mod.F90:1998
#1  0x6fc6af in ch4_tran
	at /p/scratch/cslts/gonzalez5/TSMP2/JURECA_tests/TSMP2_S2025_master/models/eCLM/src/clm5/biogeochem/ch4Mod.F90:3555
#2  0x72622f in __ch4mod_MOD_ch4
	at /p/scratch/cslts/gonzalez5/TSMP2/JURECA_tests/TSMP2_S2025_master/models/eCLM/src/clm5/biogeochem/ch4Mod.F90:1998
#3  0x502b9c in __clm_driver_MOD_clm_drv._omp_fn.4
	at /p/scratch/cslts/gonzalez5/TSMP2/JURECA_tests/TSMP2_S2025_master/models/eCLM/src/clm5/main/clm_driver.F90:1001

Line 3555 of ch4Mod.F90 corresponds to the else if of this function:

                  ! Now add new h2osfc form
                  if (.not. lake .and. sat == 1 .and. frac_h2osfc(c) > 0._r8 .and. t_h2osfc(c) >= tfrz) then
                     t_soisno_c = t_h2osfc(c) - tfrz
                     ponddiff = (d_con_w(s,1) + d_con_w(s,2)*t_soisno_c + d_con_w(s,3)*t_soisno_c**2) * 1.e-9_r8 &
                          * scale_factor_liqdiff
                     pondz = h2osfc(c) / 1000._r8 / frac_h2osfc(c) ! Assume all h2osfc corresponds to sat area
                     ! mm      /  mm/m
                     pondres = pondres + pondz / ponddiff
                  else if (.not. lake .and. sat == 1 .and. frac_h2osfc(c) > 0._r8 .and. &
                       h2osfc(c)/frac_h2osfc(c) > capthick) then ! Assuming short-circuit logic will avoid FPE here.
                     ! assume surface ice is impermeable
                     pondres = 1/smallnumber
                  end if

which is part of the subroutine ch4_tran (called in Line 1995-1998 of ch4Mod.F90):

         ! Solve CH4 reaction/diffusion equation 
         ! Competition for oxygen will occur here.
         call ch4_tran (bounds, &
              num_soilc, filter_soilc, &
              jwt(begc:endc), dtime_ch4, sat, lake, &
              soilstate_inst, temperature_inst, waterstate_inst, energyflux_inst, ch4_inst)

[jjokella]

I noticed a similar error in one of the if-conditions of ch4Mod.F90, for a debug run of eCLM-PDAF, environment jsc.2024.intel.psmpi.

[jwc02n263:685739:0:685739] Caught signal 8 (Floating point exception: floating-point invalid operation)
==== backtrace (tid:1077185) ====
 0 0x000000000003e730 __GI___sigaction()  :0
 1 0x00000000009f8e19 ch4mod_mp_ch4_prod_()  TSMP2/models/eCLM/src/clm5/biogeochem/ch4Mod.F90:2476
 2 0x00000000009f2969 ch4mod_mp_ch4_()  TSMP2/models/eCLM/src/clm5/biogeochem/ch4Mod.F90:1966
 3 0x00000000007669ef clm_driver_mp_clm_drv_.DIR.OMP.PARALLEL.LOOP.187626.split7665()  TSMP2/models/eCLM/src/clm5/main/clm_driver.F90:989
 4 0x0000000000163493 __kmp_invoke_microtask()  ???:0
 5 0x00000000000d1ca4 __kmp_fork_call()  ???:0
 6 0x0000000000089d23 __kmpc_fork_call()  ???:0
 7 0x0000000000757c07 clm_driver_mp_clm_drv_()  TSMP2/models/eCLM/src/clm5/main/clm_driver.F90:417
 8 0x000000000074c6f2 lnd_comp_mct_mp_lnd_run_mct_()  TSMP2/models/eCLM/src/clm5/cpl/lnd_comp_mct.F90:487
 9 0x0000000000651215 component_mod_mp_component_run_()  TSMP2/models/eCLM/src/eclm/component_mod.F90:728
10 0x00000000006385e4 cime_comp_mod_mp_cime_run_()  TSMP2/models/eCLM/src/eclm/cime_comp_mod.F90:2800
11 0x000000000061210d clm_advance()  TSMP2/models/pdaf/interface/model/eclm/enkf_clm_5.F90:209
12 0x0000000000611aca integrate_tsmp()  ???:0
13 0x0000000000420364 MAIN__()  TSMP2/models/pdaf/interface/framework/pdaf_terrsysmp.F90:89
14 0x0000000000418d0d main()  ???:0
15 0x00000000000295d0 __libc_start_call_main()  ???:0
16 0x0000000000029680 __libc_start_main_alias_2()  :0
17 0x0000000000418c25 _start()  ???:0
=================================

[AGonzalezNicolas]

Modifying the else if condition (at line 3555 of ch4Mod.F90), the error is no longer produced.
I modified the else if by adding a new if after.not. lake .and. sat == 1 .and. frac_h2osfc(c) > 0._r8

                  ! Now add new h2osfc form
                  if (.not. lake .and. sat == 1 .and. frac_h2osfc(c) > 0._r8 .and. t_h2osfc(c) >= tfrz) then
                     t_soisno_c = t_h2osfc(c) - tfrz
                     ponddiff = (d_con_w(s,1) + d_con_w(s,2)*t_soisno_c + d_con_w(s,3)*t_soisno_c**2) * 1.e-9_r8 &
                          * scale_factor_liqdiff
                     pondz = h2osfc(c) / 1000._r8 / frac_h2osfc(c) ! Assume all h2osfc corresponds to sat area
                     ! mm      /  mm/m
                     pondres = pondres + pondz / ponddiff
                  else if (.not. lake .and. sat == 1 .and. frac_h2osfc(c) > 0._r8) then
                     if (h2osfc(c)/frac_h2osfc(c) > capthick) then ! Assuming short-circuit logic will avoid FPE here.
                     ! assume surface ice is impermeable
                     pondres = 1/smallnumber
                     end if
                  end if

So now, log.error can continue to the "original" error:

+ srun --label --multi-prog ./eclm-parflow_mapping.conf
194: corrupted double-linked list
194: [jrc0675:188309] *** Process received signal ***
194: [jrc0675:188309] Signal: Aborted (6)
194: [jrc0675:188309] Signal code:  (-6)
221: corrupted double-linked list
221: [jrc0675:188212] *** Process received signal ***
172: corrupted double-linked list
213: corrupted double-linked list
213: [jrc0675:188284] *** Process received signal ***
213: [jrc0675:188284] Signal: Aborted (6)
213: [jrc0675:188284] Signal code:  (-6)
172: [jrc0671:136923] *** Process received signal ***
205: corrupted double-linked list

These double-linked list correspond to parflow (eclm's tasks are from 0 to 95).

[AGonzalezNicolas]

@jjokella, your error message points to Line 2476 of the same file ch4Mod.F90 of subroutine ch4_prod. It also includes and if statement with several .and. concatenated:

            ! If switched on, use pH factor for production based on spatial pH data defined in surface data.
            if (.not. lake .and. usephfact .and. pH(c) >  pHmin .and.pH(c) <  pHmax) then
               pH_fact_ch4 = 10._r8**(-0.2235_r8*pH(c)*pH(c) + 2.7727_r8*pH(c) - 8.6_r8)
               ! fitted function using data from Dunfield et al. 1993  
               ! Strictly less than one, with optimum at 6.5
               ! From Lei Meng
               f_ch4_adj = f_ch4_adj * pH_fact_ch4
            else
               ! if no data, then no pH effects
            end if

[AGonzalezNicolas]

Error occurs with OpenMPI (in the core-files /p/scratch/cslts/gonzalez5/TSMP2/JURECA_tests/TSMP2_S2025_master/RUNS/EUR-11_eclm-parflow_debug_gnu):

#8  0x0000151fa868cee4 in mca_coll_han_module_destruct ()
   from /p/software/default/stages/2025/software/OpenMPI/5.0.5-GCC-13.3.0/lib/libmpi.so.40
#9  0x0000151fa865e979 in mca_coll_cuda_module_destruct ()
   from /p/software/default/stages/2025/software/OpenMPI/5.0.5-GCC-13.3.0/lib/libmpi.so.40

[DCaviedesV]

It is odd though, that the error that @jjokella sees does not have a clear FP exception in the if statement:

 if (.not. lake .and. usephfact .and. pH(c) >  pHmin .and.pH(c) <  pHmax)

It is not obvious what can go wrong here (i.e., no division by zero). My guess is either pH(c), pHmin or pHmax are somehow undefined when either lake is true, or usephfact is false. Probably also needs to nested ifs.

[DCaviedesV]

Error occurs with OpenMPI (in the core-files `/p/scratch/cslts/gonzalez5/TSMP2/JURECA_tests/TSMP2_S2025_master/RUNS/EUR-11_eclm-parflow_debug_gnu`):

Only for OpenMPI? Or also for ParaStationMPI?

[jjokella]

Thanks @DCaviedesV and @AGonzalezNicolas for the feedback.

I will try the nested if-condition (#69).

[AGonzalezNicolas]

@DCaviedesV, right now, with modules in Stages2025 and with TSMP2 environments:
The gnu compiler only works with OpenMPI
The Intel compiler has ParaStationMPI, and when running the test case EUR-11 for eclm-parflow, it produces another error.

+ srun --label --multi-prog ./eclm-parflow_mapping.conf
  0: PSI: handleAnswer: spawn to node 832 failed: ""
  0: PSI: handleAnswer: spawn to node 832 failed: ""
  0: PSI: handleAnswer: spawn to node 832 failed: ""
  0: PSI: handleAnswer: spawn to node 832 failed: ""
  0: PSI: handleAnswer: spawn to node 832 failed: ""
  0: PSI: handleAnswer: spawn to node 832 failed: ""
  0: PSI: handleAnswer: spawn to node 832 failed: ""
  0: PSI: handleAnswer: spawn to node 832 failed: ""
  0: PSI: handleAnswer: spawn to node 832 failed: ""
  0: PSI: handleAnswer: spawn to node 832 failed: ""
  0: PSI: handleAnswer: spawn to node 832 failed: ""
  0: PSI: handleAnswer: spawn to node 832 failed: ""
  0: PSI: handleAnswer: spawn to node 832 failed: ""
  0: PSI: handleAnswer: spawn to node 832 failed: ""
  0: PSI: handleAnswer: spawn to node 832 failed: ""
  0: PSI: handleAnswer: spawn to node 832 failed: ""
  0: PSI: handleAnswer: spawn to node 832 failed: ""
  0: PSI: handleAnswer: spawn to node 832 failed: ""
  0: PSI: handleAnswer: spawn to node 832 failed: ""
  0: PSI: handleAnswer: spawn to node 832 failed: ""
  0: PSI: handleAnswer: spawn to node 832 failed: ""
  0: Could not spawn './parflow' process 96: Invalid argument

[DCaviedesV]

And how about Stages2024 + OpenMPI/ParaStationMPI? I know it's going backwards, but it might help to pinpoint the issue.

[AGonzalezNicolas]

Ilya suggested to try building TSMP2 with gnu + PSmpi. Right now it is not an option.
In Jureca Hypre/2.31.0-cpu is only available for gnu +OpenMPI or Intel.

 You will need to load all module(s) on any one of the lines below before the "Hypre/2.31.0-cpu" module is available to load.

      Stages/2024  Intel/2023.2.1  IntelMPI/2021.10.0
      Stages/2024  Intel/2023.2.1  ParaStationMPI/5.9.2-1
      Stages/2024  Intel/2023.2.1  ParaStationMPI/5.9.2-1-mt
      Stages/2025  GCC/13.3.0  OpenMPI/5.0.5
      Stages/2025  Intel/2024.2.0  IntelMPI/2021.14.1
      Stages/2025  Intel/2024.2.0  ParaStationMPI/5.10.0-1
      Stages/2025  Intel/2024.2.0  ParaStationMPI/5.10.0-1-mt
      Stages/2025  Intel/2024.2.0  ParaStationMPI/5.11.0-1
      Stages/2025  Intel/2024.2.0  ParaStationMPI/5.11.0-1-mt
      Stages/2025  Intel/2024.2.0-CUDA-12  IntelMPI/2021.14.1
      Stages/2025  Intel/2024.2.0-CUDA-12  ParaStationMPI/5.10.0-1
      Stages/2025  Intel/2024.2.0-CUDA-12  ParaStationMPI/5.10.0-1-mt
      Stages/2025  Intel/2024.2.0-CUDA-12  ParaStationMPI/5.11.0-1
      Stages/2025  Intel/2024.2.0-CUDA-12  ParaStationMPI/5.11.0-1-mt

Available version of Hypre (in Jureca):

Hypre:
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    Description:
      Hypre is a library for solving large, sparse linear systems of equations on massively parallel computers. The problems of interest arise in the simulation codes being developed at LLNL and elsewhere to study physical phenomena in the defense,
      environmental, energy, and biological sciences.

     Versions:
        Hypre/2.20.0-bigint
        Hypre/2.20.0
        Hypre/2.23.0-mixedint
        Hypre/2.23.0
        Hypre/2.25.0-cpu
        Hypre/2.27.0-cpu
        Hypre/2.27.0
        Hypre/2.29.0
        Hypre/2.31.0-cpu
        Hypre/2.31.0

Details about Hypre/2.27.0-cpu:

    You will need to load all module(s) on any one of the lines below before the "Hypre/2.27.0-cpu" module is available to load.

      Stages/2023  Intel/2022.1.0  ParaStationMPI/5.7.0-1
      Stages/2023  Intel/2022.1.0  ParaStationMPI/5.7.1-1
      Stages/2023  Intel/2022.1.0  ParaStationMPI/5.8.0-1
      Stages/2023  Intel/2022.1.0  ParaStationMPI/5.8.0-1-mt
      Stages/2023  Intel/2022.1.0  ParaStationMPI/5.8.1-1
      Stages/2023  Intel/2022.1.0  ParaStationMPI/5.8.1-1-mt

[AGonzalezNicolas]

Stages2024 Intel + ParaStationMPI, the EUR-11 test case (with eclm-parflow) finishes the simulation correctly.