how to improve the efficiency of HPC process?

[Sun-Liu-Yang]

I am trying to use MPI functionality for large-scale parallel computing on a supercomputer cluster. The number of solving units is in the tens of millions, and the number of degrees of freedom of nodes is in the billions.
When I was solving a problem with millions of degrees of freedom, I found that the solving efficiency was already very slow. The official suggested a single core degree of freedom of 20000, but currently my single core solving degree of freedom is only about 10000.
What aspects should I improve the calculation settings to significantly increase computational efficiency and meet the demand for solving billions of degrees of freedom?

[GiudGiud]

Hello

This is a complex problem and the simplest step to take is simply to profile a shorter run to see which step in your simulation is not scaling to tens of millions of cores. The limiting step is likely specific to your problem and to your solver setup.
https://mooseframework.inl.gov/application_development/profiling.html

[GiudGiud]

but when the model mesh from the abaqus(the .inp file ), only in one node can dealing fast.

do you mean with the split mesh? Reading an inp file directly would be slow since it's replicated.

There is an other way to run split meshes that might work better.
Remove --use-split and instead use a FileMeshGenerator with the file set to 2024-11-14-55.cpa.gz.
And add --distributed-mesh to the command line

[Sun-Liu-Yang]

I have try this method to solve the problem ,it seems not work inter-node.

[Sun-Liu-Yang]

this figure is the same model solved on 1nodes with 60cpus.

[GiudGiud]

I wonder if the partitioner is the problem here.

If it returned two different partitions on both runs, it would fail.

Can you try commenting it out? Parmetis is the default anyway, you are simply changing how we access parmetis by specifying this PetscPartitioner

[Sun-Liu-Yang]

you means when splitting the mesh and submit the job，commenting the [Partitioner] block out?
or
only commenting the [Partitioner] block out when submitting the job?

[Sun-Liu-Yang]

Just now, I comment the [Partitioner] block out both the two step, but it dont woke.

[GiudGiud]

I read the thread again. I think your executable may be hanging as soon as there is internode communication. The mesh preparation is the first step to have such communication.

You can try the same 2 node run without a split mesh to double check. You should see the same issue.
Do runs with just a Cartesian mesh generator solve properly? For them the first communication step will be during the solve

[Sun-Liu-Yang]

yes, the problem will be solve properly when the model mesh is constructed by the Moose's mesh geneartor (such as GeneratedMesh object )on more than one nodes.

you means that when dealing with the problem using the external mesh (such as .e, .inp file ), I should take more time for node communicatin until the preparing mesh state finshed ?

[GiudGiud]

yes, the problem will be solve properly when the model mesh is constructed by the Moose's mesh geneartor (such as GeneratedMesh object )on more than one nodes.

Ok that's good to know. So it is the FileMeshGenerator that is slow.
At this point it would be good to find out why.
You can either use a profiler (make sure to pick a mesh that is not too big so the mesh loading finishes):
https://mooseframework.inl.gov/application_development/profiling.html
or simply use a debugger and interrupt the code while it is "preparing" to see what it is actually is doing
https://mooseframework.inl.gov/application_development/debugging.html

[Sun-Liu-Yang]

Thank you very much , I will try to do this .
Whether you have met the similar case when dealing with problems using the external mesh (using the filemesh or filemeshgenerator)? I want to know whether is there some error when installing the moose the my HPC?

[GiudGiud]

Whether you have met the similar case when dealing with problems using the external mesh (using the filemesh or filemeshgenerator)?

Never had this issue. Splitting can be slow if done in parallel but loading splits has always been fine for me

I want to know whether is there some error when installing the moose the my HPC?

I dont know if it's that.

[yfun0815]

I have a similar issue when preparing mesh... for a long time. It is the issue of your installation of Openmpi of MOOSE. You can contact your hpc.

[yfun0815]

Hi @GiudGiud,
I have a 0.9 billion mesh, I want to load it with 1024 or 2048 cores, but both failed because of OOM.
I tried two methods. One is run with --use-split --split-file split.cpr. Another is run with --distributed-mesh and provide split.cpr in input file. The log is:
[pwx0406@login03] djdjdj_remesh_adjust_kernel_r3_core0]$ cat slurm-3516777.out Setting Up..-------------------------------------------------------------------------- Primary job terminated normally, but 1 process returned a non-zero exit code. Per user-direction, the job has been aborted. Current memory used: 4366955356 bytes Maximum memory used: 4366955356 bytes ***Memory allocation failed for gk_mcoremalloc: ptr. Requested size: 4294967296 bytes [qhcn355:1699617] *** Process received signal *** [qhcn355:1699617] Signal: Aborted (6) [qhcn355:1699617] Signal code: (-6) [qhcn355:1699617] [ 0] /usr/lib64/libpthread.so.0(+0x12cf0)[0x1551fb553cf0] [qhcn355:1699617] [ 1] /usr/lib64/libpthread.so.0(raise+0x10f)[0x1551fb553b8f]
Seems the mesh is replicated in every core.

[yfun0815]

So, do you mean I need to split the mesh with 1024 cores? Otherwise, I can not load the distributed mesh splited by 1 core.

[GiudGiud]

So, do you mean I need to split the mesh with 1024 cores? Otherwise, I can not load the distributed mesh splited by 1 core.

no your split it using 1 core in 1024 parts, then you load it using --use-split with 1024 processes

[GiudGiud]

note that the FileMeshGenerator might be easier to use than --use-split

[yfun0815]

I noticed memory is the limit for large mesh, so my logic is --use-split will load the distributed mesh. However the memory is still not enough. What do you mean by using FileMeshGenerator?

[GiudGiud]

In the [mesh] block you can create a FileMeshGenerator to load a split mesh