Run failing on increasing resolution

[the21vk]

Hello. I am trying to do a turbulent simulation using the Turbulent driver with magnetic fields enabled. I am facing an error when I simply increase (double) the resolution. The error does not have to do with the number of blocks vs the number of processors etc. It occurs within around 15 seconds of starting the simulation and has a cryptic error message (pasted at the end). It does not even take up a huge amount of RAM (takes ~ 15 kB). If anyone has some ideas, please help me out.

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   PID 75432 RUNNING AT cn-0190.cm.cluster
=   EXIT CODE: 9
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
[proxy:0@cn-0204] HYD_pmcd_pmip_control_cmd_cb (proxy/pmip_cb.c:487): assert (!closed) failed
[proxy:0@cn-0204] HYDT_dmxu_poll_wait_for_event (lib/tools/demux/demux_poll.c:76): callback returned error status
[proxy:0@cn-0204] main (proxy/pmip.c:122): demux engine error waiting for event
[proxy:2@cn-0176] HYD_pmcd_pmip_control_cmd_cb (proxy/pmip_cb.c:487): assert (!closed) failed
[proxy:2@cn-0176] HYDT_dmxu_poll_wait_for_event (lib/tools/demux/demux_poll.c:76): callback returned error status
[proxy:2@cn-0176] main (proxy/pmip.c:122): demux engine error waiting for event
[mpiexec@cn-0204] HYDT_bscu_wait_for_completion (lib/tools/bootstrap/utils/bscu_wait.c:109): one of the processes terminated badly; aborting
[mpiexec@cn-0204] HYDT_bsci_wait_for_completion (lib/tools/bootstrap/src/bsci_wait.c:21): launcher returned error waiting for completion
[mpiexec@cn-0204] HYD_pmci_wait_for_completion (mpiexec/pmiserv_pmci.c:189): launcher returned error waiting for completion
[mpiexec@cn-0204] main (mpiexec/mpiexec.c:260): process manager error waiting for completion

[msbc]

Can you provide more details about how you compile Athena, what cluster you're running on, what type of CPUs it has, what version of MPI are you using? There's one known Intel-MPI bug (with the library) that I'm trying to rule out.

[the21vk]

Thank you for your attention. I configure athena using the following:

python configure.py --prob mturb_avec -b -fft -hdf5 -mpi --eos=isothermal --hdf5_path=/mnt/lustre/ictspallavi/vk/my_hdf5 --fftw_path=/mnt/lustre/ictspallavi/vk/my_fftw 

mturb_avec in the above command is my custom initial condition file. The simulation works upto a resolution of 1120^3 easily, but gives the error mentioned above starting from 1280^3 onwards. I tried parallelizing over different combinations of cores/nodes - the error still persists.

I am using mpich version 4.3.0rc4 with a configured with a ch4:ofi device.

The cpu architecture is x86_64 and it is a AMD EPYC 7742 processor. Please let me know if this information is what was asked for, or if I should mention something more. Thanks again.