QUIP/LAMMPS different energies for different number of cores

[robertstella111]

Hello,

I am attempting to create a GAP potential for silicon and carbon. I have installed the QUIP library with the architecture linux_x86_64_gfortran. For the training of the GAP I am using the following command

gap_fit energy_parameter_name=energy force_parameter_name=forces do_copy_at_file=F sparse_separate_file=T gp_file=GAP.xml at_file=train.xyz gap={distance_Nb order=2 cutoff=4.5 n_sparse=15 covariance_type=ard_se delta=5 theta_uniform=2.0 sparse_method=uniform compact_clusters=T : distance_Nb order=3 cutoff=3.5 n_sparse=200 covariance_type=ard_se delta=0.3 theta_uniform=2.0 sparse_method=uniform compact_clusters=T : soap atom_sigma=0.5 l_max=8 n_max=8 cutoff=4.5 cutoff_transition_width=1.0 delta=0.1 covariance_type=dot_product n_sparse=2000 zeta=4} default_sigma={0.005 0.08 0.0 0.0} config_type_sigma={dimer:0.0008:0.02:0.0:0.0:single:0.0001:0.005:0.0:0.0:perfect_crystall:0.00001:0.01:0.0:0.0:interstitials:0.0002:0.05:0.0:0.0} sparse_jitter=1e-8 gp_file=GAP.xml

I now want to run bigger simulations in lammps. For lammps I tried several versions (lammps_stable_sep_2021, stable_5Jun2019,stable_29Aug2024) and installed the QUIP pair_style by using cmake directly and also by referring to the libquip.a library.

When I now run lammps with mpi on mulitple cores and on a single core the results for the energies are quite different. For a cell of 385 atoms, the energy difference is around 3eV. The command for the pair_style looks as follows

pair_coeff * * GAP/29_08/without_stress/GAP.xml "Potential xml_label=GAP_2024_8_30_120_8_42_16_201" 14 6

If I now change all silicon atoms to carbon or all carbon to silicon the results for the mpi and single core runs are the same. I also encountered the problem for bigger cells and smaller cells, with defects and without defects. The problem with different results for different numbers of cores is also not present for all GAP potentials I have trained so far. The one above is just an example of where the error occurs.

Does someone have an idea where the error comes from and if this will influence the results I get from longer and more realistic runs (physical wise)?

log_serial.txt
log_parallel.txt
training.txt
h_110_1.txt

[bernstei]

Really sounds like a bug in the LAMMPS-QUIP interface. Have you compared to using quippy.potential.Potential to evaluate the system, to see which value, if any, is correct?

[albapa]

Can you try the same experiment with the SW parameter file at QUIP/share/Parameters/ip.parms.SW_SiC_CASTEP_elastic.xml? You'd need something like

pair_coeff * * ip.parms.SW_SiC_CASTEP_elastic.xm "IP SW" 14 6

[robertstella111]

If I calculate the energy for the system with the following command

quip E=T atoms_filename=test.xyz param_filename=GAP.xml | grep AT | sed 's/AT//' > gap.xyz

I get an energy of -2768.55 eV. This result is closer to the single core result in lammps of -2768.9 eV. The parallel run in lammps yields -2771.475 eV. The output files are attached in the initial question. I used the initial configuration. If I use the same type of atoms all three energies are the same.

gap_output.txt

When I use the same lammps input script with the SW parameter file ip.parms.SW_SiC_CASTEP_elastic.xml I get no core energy dependence. This is also something I recognized for my trained potentials. Some of them have a core dependence and some don't.

Do you think this has something to do with QUIP version and I could get rid of it by installing an older version because I don't think it has something to do with the lammps version.

Thank you in advance.
log_serial.txt
log_parallel.txt

[bernstei]

Those are way larger errors than can be explained by anything other than a bug. If we can get a complete setup to reproduce the issue that'd help. Can you can reproduce this behavior with a GAP xml (+sparsepoints) files we have access to (e.g. one of the published publicly available ones), or are you willing to share the one you created?

Also, can you figure out how small a system you can reproduce this issue with?

[robertstella111]

I uploaded all GAP files in the following git repository

https://github.com/robertstella111/SiC_GAP_bug.git

Do you also need the trainingsdata or is this enough? I have trained the potential with the commands written in the previous messages.

For system sizes of 5x5x10 (48 atoms) Angstrom the error is still there, but also scaled down as the system size.

[bernstei]

Well, I've definitely reproduced the error, FWIW. I'll see if I can figure out what's going on.

[bernstei]

@albapa @jameskermode do you remember how the lammps interface is supposed to behave with atom_local_mask? In particular, are atoms with that mask set to False supposed to contribute to the total energy?

[albapa]

I don't think so. I think we should only calculate descriptors of atoms with atom_local_mask==.true. and derivatives with respect to all, regardless of the mask. So only these atoms get a local energy, which is then accumulated by LAMMPS.

[bernstei]

Something weird is going on. I have two input files, one of which lammps serial and mpi agree, and a slightly different one (change one atom type) where they do not. Neither agrees with pure QUIP. And in both cases atoms with local mask false have non-zero energy, which appears to contribute to the LAMMPS reported PE. More debugging in a few hours, but I have to go out now.

[bernstei]

And if I only add the ones where the mask is True, I get energies that are very different.

        out.bad.quip    TF -1933.0435
         out.bad.mpi    TF -1933.2943  T only -1723.0396
      out.bad.serial    TF -1933.2522  T only -1776.5271

       out.good.quip    TF -1929.8993
        out.good.mpi    TF -1930.1078  T only -1720.6466
     out.good.serial    TF -1930.1078  T only -1773.8421

"good" is the input configuration where LAMMPS serial and mpi agree, "bad" is the one where they don't. "TF" is adding all the local energies, "T only" is only the ones where mask is True. These are manual sums of the local energies from QUIP, and the LAMMPS ones agree with the potential energy that LAMMPS prints out in its log file.

[bernstei]

Is there a chance that atom_mask is incorrectly implemented for some particular descriptor, which this GAP is using?

[bernstei]

A couple of weird things.

1. there's no e0 value for either 8 or 14
2. I tried to produce stripped down versions of the GAP.xml file with a subset of the descriptors. When I remove the two soaps (and reduce n_coordinate from 9 to 7), it runs. When I remove all the distance_NB, leaving only the 2 soaps, the xml fails to parse.

any guesses?

[added later] I can also read a 2b-only xml file, but not a 3b-only one.

[albapa]

I had the same idea - to doctor the XML to figure out which descriptor is causing the problem. Happy to look at it.

[bernstei]

I'm trying to debug the xml parsing. BTW, If I remove the SOAP descriptors the energy value mismatch is still there (i.e 2b+3b). If I do 2b only, there's no mismatch. I think that means there's definitely an issue with the 3b, but I can't prove there isn't one with soap yet.

[robertstella111]

As far as I remember I recently trained a potential without the 3b descriptor and there was still a missmatch. I will check check that in the next couple of hours when I am home again. I will let you know afterwars and if I remembered that correctly.