Pre-compilation for CPUs

[turbulentphil]

Dear Neks,

I have been running NekRS v23 for a couple of months on GPUs. Due to circumstances beyond my control, I am forced to run NekRS on CPUs now.

I use to pre-compile my GPU cases on a single compute node before automatically submitting the production job to the system. This seems to work well for smaller CPU jobs, too. However, for larger jobs, my production job crashes (typically when generating the mesh) during the start-up of NekRS. There is no informative message about the reason in the standard output or error file.
Interestingly, if I run the exact same job without pre-compilation, the CPU job runs without issues.

Are there known issues with respect to the pre-compilation when running NekRS v23 on CPUs?

Thank you for your help,
Phil

[stgeke]

Can you please post stdout of your run (logfile).

[turbulentphil]

Thanks for your reply.
I attach several different files (corresponding to the different tests below) to my answer.

1) Pre-compilation on a single node for 128 production nodes

• Results in compilation.out and compilation.err
• Works fine.

2) Production run on 128 nodes based on pre-compilation

• Results in logfile.out.0 and logfile.err.0
• Crashes after reading the mesh.
• Unfortunately, I cannot find any useful information in the log files.

3) Production run on 128 nodes without pre-compilation

• For this run, I cleared the .cache before submission to get rid of the pre-compilation files.
• Results in logfile.out.1 and logfile.err.1
• Works fine.

4) Production run on 512 nodes without pre-compilation

• Here I repeat the above run on 512 nodes (instead of 128 nodes; .cache was cleared again).
• Results in logfile.out.2 and logfile.err.2
• Crashes again after reading the mesh. However, this time we find some more information in the error log (maybe you find also other messages important; but this is most striking to me):

      [jwc06n039.juwels:00000] *** An error occurred in MPI_Waitall
      [jwc06n039.juwels:00000] *** reported by process [1488744843,16630]
      [jwc06n039.juwels:00000] *** on communicator MPI COMMUNICATOR 9 DUP FROM 11
      [jwc06n039.juwels:00000] *** MPI_ERR_TRUNCATE: message truncated
      [jwc06n039.juwels:00000] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
      [jwc06n039.juwels:00000] ***    and potentially your MPI job)

Some information on the scale of the job:
The mesh offers 600x600x75 = 27,000,000 elements. This test job offers a polynomial order N=3 (to be increased).
The jobs are run on JUWELS Cluster where each node offers 48 CPUs.

What I can see:

• I find that the jobs from points (2, 4) crash at the same point. Only the job from point 4 prints some information in the error log.
• The simulation crashes once I use more than a certain number of nodes. Before that, no MPI errors show up.

Unfortunately, I do not know how to solve the reported MPI issues.

Do you have any ideas to solve the MPI issue?

compilation.out.txt
compilation.err.txt
logfile.out.0.txt
logfile.err.0.txt
logfile.out.1.txt
logfile.err.1.txt
logfile.out.2.txt
logfile.err.2.txt