Low force score

[semoday]

Hi cschran,
Thanks for making the code available. It is very convenient and user-friendly.
I tried the same procedure as in the Zundel example using the AIMD (300K, 50ps, BLYP functional) data set of 64 water molecules. Trajectories and forces were saved every 10fs.
AIMD settings were not accurate enough, but they were for practice.
I used the setting files (run-QbC.py, inpu.nn, ... etc.) given in the example except for the exclude_triples command in run-QbC.py.
However, the force RMSE score was too low in 05-validation step compared to the RDF and VDOS scores.

RDF Score Summary
Label | Accuracy [%]
H-H | 96.95
H-O | 96.43
O-O | 92.3
Mean | 95.23

VDOS Score Summary
Label | Accuracy [%]
H | 88.27
O | 82.2
Mean | 85.23

Force Score Summary
Label | Accuracy [%]
H | 73.08
O | 43.28
Mean | 58.18

In order to improve the force errors, I changed the value of short_force_fraction to 0.1 or force_weight to 5 in input.nn file.
However, such attempts didn't improve the force errors.

Besides, the mean force committee disagreement during the QbC process converged to ~40meV/A unlike the convergence of 20meV/A in Figure 3 top panel in J. Chem. Phys. 153, 104105 (2020).

I greatly appropriate it if you could share your experiences in this case with me.

Many thanks.
Yonghwan

[cschran]

Hi Yonghwan,

thanks for your interest in our work. A student has very recently reported similar sounding problems with the force score and I'm currently investigating where this is coming from. In his case the energy and force RMSE of the individual NNPs and also the committee were really low, so it points towards a bug in the AML code. I will get back to you once I have a better understanding of this issue.

In order to make sure that this is also a problem with the scoring code in your case and not with the actual model, you can have a quick look at the individual NNP learning curves. For liquid water I would expect an energy RMSE of ~0.5 meV/atom and force RMSE of ~70 meV/A for the test set.

The reported difference in the committee disagreement steams most likely from the difference in the symmetry functions. For the JCP we used established functions for water, while the AML package uses a generic set which is meant to work for a broader class of systems. It's definitely not the optimal though, but good enough for many applications we've tried so far.

All the best,
Christoph

[semoday]

Hi Christoph,

Thank you for your helpful comments.
As you recommended, the AIMD reference data was generated with more accurate setups.
The Force Score was indeed improved, but the force error is still large compared to the bulk system in your paper[PNAS 118, e2110077118 (2021)].

Force Score Summary
Label | Accuracy [%]
O | 67.1
H | 76.04
Mean | 71.57

AIMD computational setups seem to be similar with those of Sulfate ion system in your paper[PNAS 118, e2110077118 (2021)]. N2P2 settings are also the same.

Adapted from Figure 2 in PNAS 118, e2110077118 (2021).

I kindly ask you how I can reduce the energy/force error further.
I found an energy RMSE of ~1.8 meV/atom and force RMSE of ~220 meV/A for the test set in the final training.

Here are my input files.
bulk_water.inp.txt
input.nn.txt
I tried the same procedure as in the Zundel example.

Many thanks.
Yonghwan

[cschran]

Hi Yonghwan,

your AIMD settings look good indeed. There are three things I can directly suggest without running an AIMD with your settings myself:

1. Try increasing the force_weight in the input.nn file to 5.0 in order to see if this is leading to an improvement of the forces (Sufficient for final training).
2. Check if your AIMD has an equilibration period which has drastically different energies (and forces), e.g. because launching from FF structures or so. If that's the case, remove this period from the active learning run.
3. Perform a quick convergence test for your electronic structure setup to check how much the forces change when increasing the cutoff.

Best,
Christoph

[cschran]

Just a quick follow up on this: I ran the mentioned convergence test and got an RMSE of

1. 151.28 meV/A by comparing 400 to 600 Ry
2. 123.487 meV/A by comparing 600 to 800 Ry.

This indicates that you're still impacted severely by noise.

[semoday]

Hi Christoph,

Thank you for your quick response.

1. I've already tried increasing force_weight in the input.nn file to 5.0 as I mentioned in my first question.
   However, such an attempt did not reduce the force error.

2. I removed the first 10ps for equilibration and started the QbC run using the last 30ps, but it also did not reduce the force error.

3. I haven't tried whether increasing the cutoff can reduce force errors.
   Even though only cutoff of 280 Ry was used in the sulfate ion system [PNAS 118, e2110077118 (2021)], the force error was very low.
   Increasing the cutoff up to 800 Ry is computationally too demanding.
   It seems that a smoothing XC GRID is more important in reducing the error than increasing the cutoff.
   https://www.cp2k.org/events:2018_summer_school:converging_cutoff

As you mentioned, these errors are surprisingly high for liquid water.
I wonder how they achieved such a high force score in the bulk system [PNAS 118, e2110077118 (2021)].

Many thanks.
Yonghwan

[cschran]

Hi Yonghwan,

bottom line is that you need high quality reference data to get good fits. For our PNAS models we made sure that the electronic structure is well converged and that's why we have the reported force scores. Even for the 280 Ry setup of the sulfate ion in water this was confirmed. Note that convergence can be 1) system dependent 2) functional dependent 3) basis set dependent 4) pseudo dependent 5) cutoff dependent and there are multiple other settings in cp2k to keep an eye on.

The quick convergence test I ran with your setup suggests that there is a residual convergence error on the order of at least ~150 meV/A. You will not be able to get better force fits than that with any MLP. Please perform some additional tests with cp2k to determine where this is coming from. As mentioned above, it could be the pseudos, the basis set, the cutoff, the XC grid...

If you find that converged settings for your setup and system are too expensive, you can always simply rerun the final training set and retrain.

Best,
Christoph