MSRE pipe cardinal fails on HPC #643
Description
Bug Description
msr/msre/pipe_cardinal.multiphysics: Test spec: /scratch/romebuild12-service/build/copied_tests/msr/msre/pipe_cardinal/hpc_tests:10
msr/msre/pipe_cardinal.multiphysics: Working directory: /scratch/romebuild12-service/build/copied_tests/msr/msre/pipe_cardinal
msr/msre/pipe_cardinal.multiphysics: Running command: mpiexec -n 1 apptainer exec /scratch/romebuild12-service/build/container/cardinal-openmpi-x86_64_03401f1.sif /opt/cardinal/share/moose/python/TestHarness/schedulers/hpc_run.py /opt/cardinal/share/moose/python/TestHarness/../../../../bin/cardinal-opt%20-i%20cardinal.i%20Outputs/exodus%3Dfalse%20--error-override%20--timing%20Outputs/perf_graph%3Dtrue
msr/msre/pipe_cardinal.multiphysics:
msr/msre/pipe_cardinal.multiphysics: ################################################################################
msr/msre/pipe_cardinal.multiphysics: Begin runner_run output
msr/msre/pipe_cardinal.multiphysics:
msr/msre/pipe_cardinal.multiphysics: TestHarness slurm job on br275 in job 2826349
msr/msre/pipe_cardinal.multiphysics: Time: Sun Aug 10 16:29:06 MDT 2025
msr/msre/pipe_cardinal.multiphysics: Test: /scratch/romebuild12-service/build/copied_tests/msr/msre/pipe_cardinal/hpc_tests:multiphysics
msr/msre/pipe_cardinal.multiphysics: Directory: /scratch/romebuild12-service/build/copied_tests/msr/msre/pipe_cardinal
msr/msre/pipe_cardinal.multiphysics: Submitted hostname: romebuild12.headbuild.hpc.inl.gov
msr/msre/pipe_cardinal.multiphysics: Submission script: /scratch/romebuild12-service/build/copied_tests/msr/msre/pipe_cardinal/multiphysics.hpc_submit
msr/msre/pipe_cardinal.multiphysics: Output: /scratch/romebuild12-service/build/copied_tests/msr/msre/pipe_cardinal/multiphysics.hpc_out.txt
msr/msre/pipe_cardinal.multiphysics: Result: /scratch/romebuild12-service/build/copied_tests/msr/msre/pipe_cardinal/multiphysics.hpc_result
msr/msre/pipe_cardinal.multiphysics:
msr/msre/pipe_cardinal.multiphysics: Currently Loaded Modules:
msr/msre/pipe_cardinal.multiphysics: 1) use.moose 4) openmpi/moose_5.0.5_gcc13.3.0
msr/msre/pipe_cardinal.multiphysics: 2) apptainer/1.3.6 5) moose-dev-container-openmpi/5.0.5_0
msr/msre/pipe_cardinal.multiphysics: 3) gcc/13.2.0
msr/msre/pipe_cardinal.multiphysics:
msr/msre/pipe_cardinal.multiphysics:
msr/msre/pipe_cardinal.multiphysics:
msr/msre/pipe_cardinal.multiphysics: ################################################################################
msr/msre/pipe_cardinal.multiphysics: Beginning TestHarness slurm test execution
msr/msre/pipe_cardinal.multiphysics: INFO: Detected Singularity user configuration directory
msr/msre/pipe_cardinal.multiphysics: Running decoded command: /opt/cardinal/share/moose/python/TestHarness/../../../../bin/cardinal-opt -i cardinal.i Outputs/exodus=false --error-override --timing Outputs/perf_graph=true
msr/msre/pipe_cardinal.multiphysics: Setting Up
msr/msre/pipe_cardinal.multiphysics: Finished Setting Mesh [ 0.32 s] [ 415 MB]
msr/msre/pipe_cardinal.multiphysics: __ ____ _____
msr/msre/pipe_cardinal.multiphysics: ____ ___ / /__ / __ \/ ___/
msr/msre/pipe_cardinal.multiphysics: / __ \ / _ \ / //_// /_/ /\__ \
msr/msre/pipe_cardinal.multiphysics: / / / // __// ,< / _, _/___/ /
msr/msre/pipe_cardinal.multiphysics: /_/ /_/ \___//_/|_|/_/ |_|/____/ v23.1.1 (48e408eb)
msr/msre/pipe_cardinal.multiphysics:
msr/msre/pipe_cardinal.multiphysics: COPYRIGHT (c) 2019-2023 UCHICAGO ARGONNE, LLC
msr/msre/pipe_cardinal.multiphysics:
msr/msre/pipe_cardinal.multiphysics: MPI tasks: 1
msr/msre/pipe_cardinal.multiphysics:
msr/msre/pipe_cardinal.multiphysics: Initializing device
msr/msre/pipe_cardinal.multiphysics: active occa mode: Serial
msr/msre/pipe_cardinal.multiphysics:
msr/msre/pipe_cardinal.multiphysics: using NEKRS_HOME: /opt/cardinal/contrib
msr/msre/pipe_cardinal.multiphysics: using NEKRS_CACHE_DIR: /ibmstor/scratch/romebuild12-service/build/copied_tests/msr/msre/pipe_cardinal/.cache
msr/msre/pipe_cardinal.multiphysics: using OCCA_CACHE_DIR: /ibmstor/scratch/romebuild12-service/build/copied_tests/msr/msre/pipe_cardinal/.cache/occa/
msr/msre/pipe_cardinal.multiphysics:
msr/msre/pipe_cardinal.multiphysics: building nekInterface for lx1=6, lelt=494592 and lelg=494592 ... done (28.0359s)
msr/msre/pipe_cardinal.multiphysics: loading nek ... Error in set_usr_handles: /ibmstor/scratch/romebuild12-service/build/copied_tests/msr/msre/pipe_cardinal/.cache/nek5000/libtjunc.so: cannot map zero-fill pages
msr/msre/pipe_cardinal.multiphysics: stack trace:
msr/msre/pipe_cardinal.multiphysics: /opt/cardinal/contrib/lib/libnekrs.so : set_usr_handles(char const*, int)+0x12a8
msr/msre/pipe_cardinal.multiphysics: /opt/cardinal/contrib/lib/libnekrs.so : nek::bootstrap()+0x428
msr/msre/pipe_cardinal.multiphysics: /opt/cardinal/contrib/lib/libnekrs.so : nekrs::setup(ompi_communicator_t*, ompi_communicator_t*, int, int, int, inipp::Ini*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, int, int, int)+0xceb
msr/msre/pipe_cardinal.multiphysics: /opt/cardinal/lib/cardinal/libcardinal-opt.so.0 : NekInitAction::act()+0x3d9
msr/msre/pipe_cardinal.multiphysics: /opt/cardinal/lib/cardinal/libmoose-opt.so.0 : Action::timedAct()+0x2e
msr/msre/pipe_cardinal.multiphysics: /opt/cardinal/lib/cardinal/libmoose-opt.so.0 : ActionWarehouse::executeActionsWithAction(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x4ab
msr/msre/pipe_cardinal.multiphysics: /opt/cardinal/lib/cardinal/libmoose-opt.so.0 : ActionWarehouse::executeAllActions()+0x17c
msr/msre/pipe_cardinal.multiphysics: /opt/cardinal/lib/cardinal/libmoose-opt.so.0 : MooseApp::runInputFile()+0x93
msr/msre/pipe_cardinal.multiphysics: /opt/cardinal/lib/cardinal/libmoose-opt.so.0 : MultiApp::createApp(unsigned int, double)+0x11de
msr/msre/pipe_cardinal.multiphysics: /opt/cardinal/lib/cardinal/libmoose-opt.so.0 : MultiApp::createApps()+0x192
msr/msre/pipe_cardinal.multiphysics: /opt/cardinal/lib/cardinal/libmoose-opt.so.0 : FEProblemBase::addMultiApp(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, InputParameters&)+0x1fc
msr/msre/pipe_cardinal.multiphysics: /opt/cardinal/lib/cardinal/libmoose-opt.so.0 : Action::timedAct()+0x2e
msr/msre/pipe_cardinal.multiphysics: /opt/cardinal/lib/cardinal/libmoose-opt.so.0 : ActionWarehouse::executeActionsWithAction(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x4ab
msr/msre/pipe_cardinal.multiphysics: /opt/cardinal/lib/cardinal/libmoose-opt.so.0 : ActionWarehouse::executeAllActions()+0x17c
msr/msre/pipe_cardinal.multiphysics: /opt/cardinal/lib/cardinal/libmoose-opt.so.0 : MooseApp::runInputFile()+0x93
msr/msre/pipe_cardinal.multiphysics: /opt/cardinal/lib/cardinal/libmoose-opt.so.0 : MooseApp::run()+0x578
msr/msre/pipe_cardinal.multiphysics: /opt/cardinal/share/moose/python/TestHarness/../../../../bin/cardinal-opt() [0x4053b9]
msr/msre/pipe_cardinal.multiphysics: /lib64/libc.so.6 : __libc_start_main()+0xe5
msr/msre/pipe_cardinal.multiphysics: /opt/cardinal/share/moose/python/TestHarness/../../../../bin/cardinal-opt() [0x40563e]
msr/msre/pipe_cardinal.multiphysics: --------------------------------------------------------------------------
msr/msre/pipe_cardinal.multiphysics: MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
msr/msre/pipe_cardinal.multiphysics: Proc: [[43169,1],0]
msr/msre/pipe_cardinal.multiphysics: Errorcode: 1
msr/msre/pipe_cardinal.multiphysics:
msr/msre/pipe_cardinal.multiphysics: NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
msr/msre/pipe_cardinal.multiphysics: You may or may not see output from other processes, depending on
msr/msre/pipe_cardinal.multiphysics: exactly when Open MPI kills them.
msr/msre/pipe_cardinal.multiphysics: --------------------------------------------------------------------------
msr/msre/pipe_cardinal.multiphysics: --------------------------------------------------------------------------
msr/msre/pipe_cardinal.multiphysics: prterun has exited due to process rank 0 with PID 0 on node br275 calling
msr/msre/pipe_cardinal.multiphysics: "abort". This may have caused other processes in the application to be
msr/msre/pipe_cardinal.multiphysics: terminated by signals sent by prterun (as reported here).
msr/msre/pipe_cardinal.multiphysics: --------------------------------------------------------------------------
msr/msre/pipe_cardinal.multiphysics: ################################################################################
msr/msre/pipe_cardinal.multiphysics: Completed TestHarness slurm test execution; exit code = 1, walltime = 32.25 sec
msr/msre/pipe_cardinal.multiphysics: Failed to find output /scratch/romebuild12-service/build/copied_tests/msr/msre/pipe_cardinal/cardinal_out.csv
msr/msre/pipe_cardinal.multiphysics:
msr/msre/pipe_cardinal.multiphysics: ################################################################################
msr/msre/pipe_cardinal.multiphysics: End runner_run output
msr/msre/pipe_cardinal.multiphysics:
msr/msre/pipe_cardinal.multiphysics: ################################################################################
msr/msre/pipe_cardinal.multiphysics: Begin tester output
msr/msre/pipe_cardinal.multiphysics:
msr/msre/pipe_cardinal.multiphysics: Exit Code: 1
msr/msre/pipe_cardinal.multiphysics:
msr/msre/pipe_cardinal.multiphysics:
msr/msre/pipe_cardinal.multiphysics: Tester failed, reason: CRASH
msr/msre/pipe_cardinal.multiphysics:
msr/msre/pipe_cardinal.multiphysics: ################################################################################
msr/msre/pipe_cardinal.multiphysics: End tester output
Steps to Reproduce
Run HPC testing with that case
Impact
Broken CI