Running LAMMPS with DeePMD potential and several GPUs

[TypicalBarmalei]

Hello everyone,

I’ve read previous discussions, but they didn’t fully answer my questions.
I’m running DeePMD v2.2.11 with LAMMPS (29 Aug 2024) on an HPC cluster using SLURM, installed via Conda with GPU support.

When I try to use more than one GPU, performance doesn’t change, and nvidia-smi shows 0% usage on the second GPU.
This suggests that multi-GPU execution might not actually be happening.

Questions

1. Does the Conda-compiled LAMMPS + DeePMD build support multi-GPU execution out of the box?
2. Or should DeePMD and LAMMPS be recompiled from source with specific GPU flags (e.g. KOKKOS, CUDA, or GPU packages)?
3. Does the same limitation apply to model training (dp train)?
4. Is there anything wrong in my SLURM submission script that could prevent proper GPU utilization?

Thanks in advance for your help!

Attached files:
sbatch_example.txt
conda_list.txt
lammps_help.txt
mpirun_info.txt
nvidia_smi.txt
log_lammps.txt

[njzjz]

In log.lammps, there is a message 2 MPI tasks x 1 OpenMP threads, so the MPI seems to be good and can be recognized by LAMMPS. The software looks also good - it's the GPU version.

I think this may be related to CUDA_VISIBLE_DEVICES set by the Slurm. Could you run the following command?

mpirun -np 2 printenv CUDA_VISIBLE_DEVICES

Ideally, both ranks should have the environment variable CUDA_VISIBLE_DEVICES=0,1.

[TypicalBarmalei]

The output (with same SLURM tags as in file above) is:

0,1
0,1

[TypicalBarmalei]

So I was able to run some tests with different SLURM sbatch parameters and found that I can actually utilize several GPUs. nvidia-smi still shows underusage (GPU usage doesn't go above 40%), not sure whether this is fine or not. If it can be fine-tuned for better performance, that would be great. But the question is more or less closed now.
I believe that missing cpus-per-task and "wrong" OMP_NUM_THREADS was the case - requested CPU cores weren't enough to process several GPUs.

ntasks	cpus-per-task	gpus	OMP_NUM_THREADS	timesteps/s
2	1 (not set)	1	1		19
4	4		4	4		30
2	2		2	2		16
4	2		4	2		26
4	8		4	8		21
2	4		1	4		13
8	2		4	2		51
4	2		4	2		23
8	2		4	1		40
8	1		4	1		14
8	4		4	4		52
16	2		4	1		50
16	2		4	2		20
4	2		2	2		31

• The system is not very big and consists of ~ 7000 atoms in a solid phase; cutoff+skin is 6+1 ... 6+2.
• Changing INTRA and INTER_OP tags didn't have any effect on performance.
• Since the cluster has slightly different nodes, performance with the same setup can vary. For the "best" cases, though, I did several tests on different node types to ensure consistency.