Inventory file reading error when using ELM-FATES initialization for a gridded simulation

[danielletijerina]

Describe the changes

I have been informed by Marcos Longo that some code in ED2 exists that “has calls MPI-library subroutines to prevent multiple nodes accessing the same FATES inventory text file at the same time: https://github.com/EDmodel/ED2/blob/396cd575c9e39acc222c640070953888e7eeff16/ED/src/init/ed_init.F90#L374-L484” however, as I understand, ED2 is a standalone model, which is different from FATES, which requires the host land model.

Is this code that prevents the file being opened simultaneously by different parts of the code (nodes) something that would be possible to employ in ELM-FATES?

Additionally, is there a protocol or workflow for executing a multi-site initialization where I can use one set of inventory files per grid cell (the second bullet point above)? We are interested in being able to run spatially explicit vegetation (e.g. different vegetation types) for each grid cell, but there doesn't seem to be much or any documentation on how this would be done (and the runs I tried with this methodology were unsuccessful… this could merit another github issue).

Is your request related to a problem?

I am testing an ELM-FATES simulation in a small catchment with gridcell size of 100m and with dimensions of 150x170. I have been using different, but spatially homogeneous initialization scenarios, and am running into an error that appears to occur when the initialization file is being open, read, and then closed by different parts of the code simultaneously (not sure if these are grid cells or nodes... I also ran on a single node and the error still occurred).

This results in the model error:
1: The inventory list file is open but should not be. 1: Aborting. 1: ENDRUN:ERROR in /pscratch/sd/d/dtk/ELM_FATES_tree_density_runs/PLN2_2020inventory_Daily_WY2012_modParamFile.IELMFATES.pm-cpu.gnu..2025-04-03/SourceMods/src.elm/FatesInventoryInitMod.F90 at line 177 1: ERROR: Unknown error submitted to shr_abort_abort.

With the lines in the FatesInventoryInitMod.F90 from 167-177:
inquire(file=trim(hlm_inventory_ctrl_file),exist=lexist,opened=lopen) if( .not.lexist ) then ! The inventory file list DNE write(fates_log(), *) 'An inventory Initialization was requested.' write(fates_log(), *) 'However the inventory file: ',trim(hlm_inventory_ctrl_file),' DNE' write(fates_log(), *) 'Aborting' call endrun(msg=errMsg(sourcefile, __LINE__)) end if if( lopen ) then ! The inventory file should not be open write(fates_log(), *) 'The inventory list file is open but should not be.' write(fates_log(), *) 'Aborting.' call endrun(msg=errMsg(sourcefile, __LINE__)) end if

Things I have tried to rectify this, which were unsuccessful:

1. Run on a single node
2. Create unique sets of inventory files (one .pss and one .css) for each grid cell and add individual lines to the inventory control file that specifies the lat/long of each grid cell

Alternative options

No response

Additional context

e3sm.log.37461220.250404-092906-inventoryOpen.txt

[rgknox]

@danielletijerina we discussed this issue a the developer meeting and there was general agreement that the current state of the code cannot handle inventory initialization over multiple sites in an MPI scenario. You are actually the first person to attempt this type of run. We believe this is due to a race-condition in that different nodes will be trying to open not just the pss/css files, but also the control file. One solution to this would be to use blocking send/receives (or other similar modern alternatives :/ ), applied at the host-model level, so that the nodes could have sole access to the text files. Another solution would be for us to move towards netcdf, netcdf is more conducive to more than one node accessing a file. I also wonder if it is possible to change our fortran code to open the text files in such a way that they don't become inaccessible to other nodes...

I'm surprised though that a single node was not able to work. I think we could figure this out...

If we can, then you could run a simulation with one node for a short period of time (such as a day or an hour, so that only one node is reading in the inventory), then have it stop and write a restart, and then restart the model using more nodes...?

[mpaiao]

Thanks for the follow up, Ryan! Indeed the restart solution may be the best one in the short term, but maybe this should become an item for future model development in the longer-term.

[rgknox]

I found this in the intel forums: https://community.intel.com/t5/Intel-Fortran-Compiler/Multithreaded-File-I-O/m-p/837743#M56146

Here is how we open the ctrl file: https://github.com/NGEET/fates/blob/main/main/FatesInventoryInitMod.F90#L188

pss...: https://github.com/NGEET/fates/blob/main/main/FatesInventoryInitMod.F90#L254

[mpaiao]

@rgknox I think the link to the intel forum in your previous comment is incorrect.

Another alternative that occurred to me is to add these barriers in ELM directly, and make the entire FATES initialisation step to be "node serial" when we need to read in pss/css files. It will make this step less efficient, but would allow for any potential difference on how host land models handle parallel runs.

For reference, below is a summary on how we handle this in ED2. By the way, I think you were the one who originally implemented this @rgknox.

use mpi
use ed_node_coms, only : mynum    ! ! intent(in) - A unique ID for this node, starting at 1 and going to nnodetot
use ed_node_coms, only : nnodetot ! ! intent(in) - Total number of nodes

# Local variables
integer :: sendnum ! The node to which we notify that this node finished reading the file
integer :: recvnum ! The node that will send information that this node can start reading this file

# Local parameters
integer, parameter :: unique_id = 101 # Arbitrary number, but unique to this MPI_send/recv code.

sendnum = mynum + 1
recvnum = mynum - 1

if (mynum /= 1) then
   call MPI_Recv(ping,1,MPI_INTEGER,recvnum,unique_id,MPI_COMM_WORLD,MPI_STATUS_IGNORE,ierr)
end if
      
call read_pss_css_files(...)

if (mynum < nnodetot) then
   call MPI_Send(ping,1,MPI_INTEGER,sendnum,unique_id,MPI_COMM_WORLD,ierr)
end if

[rgknox]

updated my link. Since these are reads, I feel like multiple threads should be able to access the files, good chance this is just unit number mangling and our own error protections getting in our way.

I have a prototype that generates new unit numbers and suppresses our error checking here, it is untested:

branch: https://github.com/rgknox/fates/tree/multi-site-files

diff: https://github.com/NGEET/fates/compare/main...rgknox:fates:multi-site-files?w=1

[mpaiao]

Ops, I just saw your last comment here and fix suggestion @rgknox! Without seeing this, I drafted a pull request to E3SM.

@danielletijerina if @rgknox's proposed changes address the issue, I'd say we should stick to his solution, because it would be robust across all host land models, as opposed to a solution for ELM only.