Hang on loading meshing on INL HPC with multiple nodes

[GiudGiud]

Describe the bug
The simulation of the PB-FHR reflector using Nek, through Cardinal, are hanging on INL HPC when reading the mesh. This is not the only failing input but it's the one I'm working on. David Reger (@regerdavid) might have more.

pbfhr/mark1/reflector.nek: Test spec: /scratch/romebuild7-service/build/copied_tests/pbfhr/mark1/reflector/hpc_tests:2
pbfhr/mark1/reflector.nek: Working directory: /scratch/romebuild7-service/build/copied_tests/pbfhr/mark1/reflector/cht
pbfhr/mark1/reflector.nek: Running command: ssh bitterroot2.hpc.inl.gov 'cd /scratch/romebuild7-service/build/copied_tests/pbfhr/mark1/reflector/cht; FILE="/scratch/romebuild7-service/build/copied_tests/pbfhr/mark1/reflector/cht/nek.hpc_submit"; for i in {1..40}; do if [ -e "$FILE" ]; then sbatch $FILE; exit $?; else sleep 0.25; fi; done; exit 1'
pbfhr/mark1/reflector.nek: 
pbfhr/mark1/reflector.nek: ################################################################################
pbfhr/mark1/reflector.nek: Begin runner_run output
pbfhr/mark1/reflector.nek: 
pbfhr/mark1/reflector.nek: Displaying last 100 lines due to -x/--sep-files
pbfhr/mark1/reflector.nek: 
pbfhr/mark1/reflector.nek: <<<   polynomialOrder = 3
pbfhr/mark1/reflector.nek: <<< 
pbfhr/mark1/reflector.nek: <<< [VELOCITY]
pbfhr/mark1/reflector.nek: <<<   viscosity = -100.0
pbfhr/mark1/reflector.nek: <<<   density = 1.0
pbfhr/mark1/reflector.nek: <<<   residualTol = 1.0e-8
pbfhr/mark1/reflector.nek: <<<   boundaryTypeMap = W, W, symy, W, v, o, W, W
pbfhr/mark1/reflector.nek: <<< 
pbfhr/mark1/reflector.nek: <<< [PRESSURE]
pbfhr/mark1/reflector.nek: <<<   residualTol = 1.0e-8
pbfhr/mark1/reflector.nek: <<< 
pbfhr/mark1/reflector.nek: <<< [TEMPERATURE]
pbfhr/mark1/reflector.nek: <<<   conductivity = -1500.5
pbfhr/mark1/reflector.nek: <<<   rhoCp = 1.0
pbfhr/mark1/reflector.nek: <<<   residualTol = 1.0e-8
pbfhr/mark1/reflector.nek: <<<   boundaryTypeMap = f, f, I, I, t, o, f, I
pbfhr/mark1/reflector.nek: <<< 
pbfhr/mark1/reflector.nek: 
pbfhr/mark1/reflector.nek: 
pbfhr/mark1/reflector.nek: <<< #include "udf.hpp"
pbfhr/mark1/reflector.nek: <<< 
pbfhr/mark1/reflector.nek: <<< void UDF_LoadKernels(occa::properties & kernelInfo)
pbfhr/mark1/reflector.nek: <<< {
pbfhr/mark1/reflector.nek: <<<   // Compute the inlet velocity boundary condition and propagate it to a kernel variable
pbfhr/mark1/reflector.nek: <<<   kernelInfo["defines/Vz"] = 1.0;
pbfhr/mark1/reflector.nek: <<< 
pbfhr/mark1/reflector.nek: <<<   // Compute the inlet temperature and propagate it to a kernel variable
pbfhr/mark1/reflector.nek: <<<   kernelInfo["defines/inlet_T"] = 0.0;
pbfhr/mark1/reflector.nek: <<< }
pbfhr/mark1/reflector.nek: <<< 
pbfhr/mark1/reflector.nek: <<< void UDF_Setup(nrs_t *nrs)
pbfhr/mark1/reflector.nek: <<< {
pbfhr/mark1/reflector.nek: <<<   mesh_t * mesh = nrs->cds->mesh[0];
pbfhr/mark1/reflector.nek: <<< 
pbfhr/mark1/reflector.nek: <<<   // loop over all the GLL points and assign directly to the solution arrays by
pbfhr/mark1/reflector.nek: <<<   // indexing according to the field offset necessary to hold the data for each
pbfhr/mark1/reflector.nek: <<<   // solution component
pbfhr/mark1/reflector.nek: <<<   int n_gll_points = mesh->Np * mesh->Nelements;
pbfhr/mark1/reflector.nek: <<<   for (int n = 0; n < n_gll_points; ++n)
pbfhr/mark1/reflector.nek: <<<   {
pbfhr/mark1/reflector.nek: <<<     nrs->U[n + 0 * nrs->fieldOffset] = 0.0; // x-velocity
pbfhr/mark1/reflector.nek: <<<     nrs->U[n + 1 * nrs->fieldOffset] = 0.0; // y-velocity
pbfhr/mark1/reflector.nek: <<<     nrs->U[n + 2 * nrs->fieldOffset] = 1.0; // z-velocity
pbfhr/mark1/reflector.nek: <<< 
pbfhr/mark1/reflector.nek: <<<     nrs->P[n] = 0.0; // pressure
pbfhr/mark1/reflector.nek: <<<   }
pbfhr/mark1/reflector.nek: <<< }
pbfhr/mark1/reflector.nek: <<< 
pbfhr/mark1/reflector.nek: 
pbfhr/mark1/reflector.nek: <<< void velocityDirichletConditions(bcData * bc)
pbfhr/mark1/reflector.nek: <<< {
pbfhr/mark1/reflector.nek: <<<   if (bc->id == 5)
pbfhr/mark1/reflector.nek: <<<   {
pbfhr/mark1/reflector.nek: <<<     bc->u = 0.0; // x-velocity
pbfhr/mark1/reflector.nek: <<<     bc->v = 0.0; // y-velocity
pbfhr/mark1/reflector.nek: <<<     bc->w = Vz;  // z-velocity
pbfhr/mark1/reflector.nek: <<<   }
pbfhr/mark1/reflector.nek: <<< }
pbfhr/mark1/reflector.nek: <<< 
pbfhr/mark1/reflector.nek: <<< void scalarDirichletConditions(bcData * bc)
pbfhr/mark1/reflector.nek: <<< {
pbfhr/mark1/reflector.nek: <<<   if (bc->id == 5)
pbfhr/mark1/reflector.nek: <<<     bc->s = inlet_T;
pbfhr/mark1/reflector.nek: <<< }
pbfhr/mark1/reflector.nek: <<< 
pbfhr/mark1/reflector.nek: <<< void scalarNeumannConditions(bcData * bc)
pbfhr/mark1/reflector.nek: <<< {
pbfhr/mark1/reflector.nek: <<<   if (bc->id == 1 || bc->id == 2 | bc->id == 7)
pbfhr/mark1/reflector.nek: <<<     bc->flux = bc->usrwrk[bc->idM];
pbfhr/mark1/reflector.nek: <<< }
pbfhr/mark1/reflector.nek: <<< #ifdef __okl__
pbfhr/mark1/reflector.nek: <<< void meshVelocityDirichletConditions(bcData *bc){}
pbfhr/mark1/reflector.nek: <<< void velocityNeumannConditions(bcData *bc){}
pbfhr/mark1/reflector.nek: <<< void pressureDirichletConditions(bcData *bc){}
pbfhr/mark1/reflector.nek: <<< #endif
pbfhr/mark1/reflector.nek: <<< 
pbfhr/mark1/reflector.nek: 
pbfhr/mark1/reflector.nek: loading udf kernels ... 
pbfhr/mark1/reflector.nek: done (0.669249s)
pbfhr/mark1/reflector.nek: benchmarking hot kernels ...
pbfhr/mark1/reflector.nek: WARNING: kernel[0] timings differ by up to 2.21 across ranks!
pbfhr/mark1/reflector.nek: WARNING: kernel[0] timings differ by up to 3.04 across ranks!
pbfhr/mark1/reflector.nek: WARNING: kernel[0] timings differ by up to 1.34 across ranks!
pbfhr/mark1/reflector.nek: WARNING: kernel[0] timings differ by up to 1.20 across ranks!
pbfhr/mark1/reflector.nek: WARNING: kernel[0] timings differ by up to 1.34 across ranks!
pbfhr/mark1/reflector.nek: JIT compiling kernels (this may take awhile if they are not in cache) ...
pbfhr/mark1/reflector.nek: done (28.5335s)
pbfhr/mark1/reflector.nek: 
pbfhr/mark1/reflector.nek: wdotp nFields=01 min/avg/max: 7.609e-05s 8.363e-05s 1.445e-04s  (avg local: 6.484e-05s / 3.264e+01GB/s)
pbfhr/mark1/reflector.nek: wdotp nFields=03 min/avg/max: 6.155e-04s 6.717e-04s 1.063e-03s  (avg local: 5.748e-04s / 8.592e+00GB/s)
pbfhr/mark1/reflector.nek: Ax: N=3 wordSize=64 GDOF/s=0.021336 GB/s=4.45052 GFLOPS/s=3.48962 constCoeff=0 poisson=0 kernelVer=0
pbfhr/mark1/reflector.nek: Ax: N=3 wordSize=64 GDOF/s=0.030309 GB/s=4.59799 GFLOPS/s=4.59799 constCoeff=1 poisson=1 kernelVer=0
pbfhr/mark1/reflector.nek: Ax: N=3 wordSize=32 GDOF/s=0.138877 GB/s=10.5341 GFLOPS/s=21.0681 constCoeff=1 poisson=1 kernelVer=0
pbfhr/mark1/reflector.nek: fdm: N=5 wordSize=32 GDOF/s=0.0923908 GB/s=2.23512 GFLOPS/s=11.6546 kernelVer=0
pbfhr/mark1/reflector.nek: Ax: Ndim=3 N=3 wordSize=64 GDOF/s=0.0353469 GB/s=3.35141 GFLOPS/s=5.78118 constCoeff=0 poisson=0 kernelVer=0
pbfhr/mark1/reflector.nek: initializing linAlg ...
pbfhr/mark1/reflector.nek: 
pbfhr/mark1/reflector.nek:  reading fluid.re2                                                                                                                           
pbfhr/mark1/reflector.nek:  hdr:#v003   264597  3   264597 this is the hdr                                      
pbfhr/mark1/reflector.nek: slurmstepd: error: *** JOB 2693376 ON br277 CANCELLED AT 2025-07-06T00:21:59 DUE TO TIME LIMIT ***
pbfhr/mark1/reflector.nek: 
pbfhr/mark1/reflector.nek: ################################################################################
pbfhr/mark1/reflector.nek: End runner_run output
pbfhr/mark1/reflector.nek: 
[3617s ]  TIMEOUT pbfhr/mark1/reflector.nek FAILED (SLURM JOB TIMEOUT) [min_cpus=192,job=2693376]

This happens with this version of OpenMPI (other packages for completeness), which is the one used by MOOSE for all applications

Installed packages
Python (/opt/miniforge3/envs/moose/bin/python3): Python 3.13.5
git (/usr/bin/git): git version 2.43.5
Conda (/opt/miniforge3/condabin/conda): conda 25.5.1
Conda prefix: /opt/miniforge3/envs/moose
C compiler (/opt/openmpi/bin/mpicc): gcc (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7)
C++ compiler (/opt/openmpi/bin/mpicxx): g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7)
F77 compiler (/opt/openmpi/bin/mpif77): GNU Fortran (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7)
F90 compiler (/opt/openmpi/bin/mpif90): GNU Fortran (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7)
MPI (/opt/openmpi/bin/mpiexec): Open MPI (5.0.7)
PETSc (PETSC_DIR=/opt/petsc): v3.23.0-9-g95934b0d393
libMesh (LIBMESH_DIR=/opt/libmesh): f0a69d16b43414eba6d1c1bfe8482ccd6bda220b
MFEM (MFEM_DIR=/opt/mfem): 4.8.1

It can run successfully with these other modules, but we don't want to have a Cardinal-specific openmpi version, too costly maintenance-wise.

module load gcc/10.5.0-gcc-8.5.0-to6d
module load cmake
module load **openmpi/4.1.5_ucx1.14.1**

To Reproduce
Run this case from the NRIC virtual test bed on one of INL's cluster (bitterroot for example)
https://github.com/idaholab/virtual_test_bed/tree/devel/pbfhr/mark1/reflector
Ping me for access to HPC if needed

Expected behavior
Load the mesh, run the case in ~20 minutes with 240 CPUs

Desktop (please complete the following information):

• OS: Rocky Linux 8.10 (Green Obsidian)
• Version: The current NekRS version in Cardinal
  https://github.com/neams-th-coe/nekRS/tree/48e408eb9a1f4de674efd243a982c84300c792e4

Additional context
@loganharbour @aprilnovak for awareness

Needed for VTB HPC testing. You can use the VTB charge number with @eshemon 's permission at ANL

[yslan]

Have you tried other Modular Component Architecture?

On my laptop (and some other ubuntu 2204 servers and a mac), the same location triggers a MPI_Win_Create error but I can bypass it via either ucx or rdma.

 nrs->init ...
 call nek setup
 reading channel.re2
 hdr:#v002     1024  3      256 this is the hdr
[mcswl082:00000] *** An error occurred in MPI_Win_create
[mcswl082:00000] *** reported by process [4037804033,0]
[mcswl082:00000] *** on communicator MPI COMM 7 DUP FROM 5
[mcswl082:00000] *** MPI_ERR_WIN: invalid window
[mcswl082:00000] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
[mcswl082:00000] ***    and MPI will try to terminate your MPI job as well)

$ ompi_info |grep -i osc
                 MCA osc: pt2pt (MCA v2.1.0, API v3.0.0, Component v4.1.2)
                 MCA osc: monitoring (MCA v2.1.0, API v3.0.0, Component v4.1.2)
                 MCA osc: ucx (MCA v2.1.0, API v3.0.0, Component v4.1.2)
                 MCA osc: sm (MCA v2.1.0, API v3.0.0, Component v4.1.2)
                 MCA osc: rdma (MCA v2.1.0, API v3.0.0, Component v4.1.2)

mpirun --mca osc ucx 

[GiudGiud]

I have not. I ll take a look. Not sure this is an option for automated testing either but I ll ask
Is NekRS using MPI3 RMA?

[yslan]

Not sure this is an option for automated testing either

You simply add it into the submit script. Or, you can also set it via an environment variable e.g. OMPI_MCA_osc=ucx

Is NekRS using MPI3 RMA?

Our re2 reader, set_proc_map in Nek5000, uses MPI_WIN_Create. So, yes.

[loganharbour]

No luck with ucx either; littered with:

[wr12n39:585018] ../../../../../ompi/mca/osc/ucx/osc_ucx_component.c:369  Error: OSC UCX component priority set inside component query failed

This is UCX 1.18.1

[yslan]

No luck with ucx either; littered with:

[wr12n39:585018] ../../../../../ompi/mca/osc/ucx/osc_ucx_component.c:369  Error: OSC UCX component priority set inside component query failed

This is UCX 1.18.1

Does your desired OMPI module support ucx? check ompi_info.
I thought you are trying to avoid openmpi/4.1.5_ucx1.14.1?

I'd also suggest testing a smaller example regarding MPI_Win_Create.

$ mpic++ t.cpp
$ mpirun --mca osc ucx -n 2 ./a.out
$ mpirun --mca osc rdma -n 2 ./a.out

where

$ cat t.cpp

#include <mpi.h>
#include <iostream>

int main(int argc, char** argv) {
    MPI_Init(&argc, &argv);

    int rank, size;
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
    MPI_Comm_size(MPI_COMM_WORLD, &size);

    int myData = rank;     // Local buffer
    int winBuffer = -1;    // Exposed memory for remote access

    // Create a window exposing winBuffer to RMA
    MPI_Win win;
    MPI_Win_create(&winBuffer, sizeof(int), sizeof(int), MPI_INFO_NULL, MPI_COMM_WORLD, &win);

    MPI_Win_fence(0, win);  // Start RMA epoch

    if (rank == 0 && size > 1) {
        int valueToPut = 42;
        // Put value into rank 1's window
        MPI_Put(&valueToPut, 1, MPI_INT, 1, 0, 1, MPI_INT, win);
    }

    MPI_Win_fence(0, win);  // End RMA epoch

    std::cout << "Rank " << rank << " sees winBuffer = " << winBuffer << std::endl;

    MPI_Win_free(&win);
    MPI_Finalize();
    return 0;
}

[loganharbour]

No issues with the simple example.

This is ucx 1.18.1, openmpi 5.0.7. With the following components:

$ ompi_info | grep osc
                 MCA osc: sm (MCA v2.1.0, API v3.0.0, Component v5.0.7)
                 MCA osc: monitoring (MCA v2.1.0, API v3.0.0, Component v5.0.7)
                 MCA osc: rdma (MCA v2.1.0, API v3.0.0, Component v5.0.7)
                 MCA osc: ucx (MCA v2.1.0, API v3.0.0, Component v5.0.7)

[yslan]

Have you tried rdma?

[yslan]

If you can run the example, then it might not be a MPI_Win_Create issue
Might have to attach the process with gdb to get the exact location..

[loganharbour]

rdma is the case that hangs.

I'll work on getting a trace for both the rdma hang and a trace for the ucx error

[stgeke]

Can you please try to run the code with:

mpirun --mca osc p2pt ...

[loganharbour]

I'm confused - how is this not an issue exactly? The tool not running with a modern openmpi implementation is not an issue?

[loganharbour]

And, for the above, we don't have p2pt:

$ ompi_info | grep osc
                 MCA osc: sm (MCA v2.1.0, API v3.0.0, Component v5.0.7)
                 MCA osc: monitoring (MCA v2.1.0, API v3.0.0, Component v5.0.7)
                 MCA osc: rdma (MCA v2.1.0, API v3.0.0, Component v5.0.7)
                 MCA osc: ucx (MCA v2.1.0, API v3.0.0, Component v5.0.7)

[stgeke]

The issue you're facing is related to your system configuration and not the code itself. Something is off with the MPI installation.
Sorry I didn't realize that p2pt is not available. Can you try to reproduce with an older MPI e.g. 4.x