Enable distributed training test with NCCL on ALPS (GH200)

[RMeli]

This PR builds on #687 and runs the distributed training example in tests/distributed. The example is run on a single node, with all 4 GPUs.

The CI workflow is now the following:

Build base container -> Build MTT container -> Run distributed example
          |
          ------------> Run tox tests

The base container (used by the tox tests) is now only re-built if the Containerfile changes.

This PR also adds torch.distributed.destroy_process_group() to the different trainers to get rid of the corresponding warnings.

This is a starting point, but we should discuss about how to expand the distributed training tests (currently the example only tests SOAP-BPNN).

EDIT: For the distributed training tests I built on top of the base container. I did not yet explore the performance of this solution.

---

📚 Documentation preview 📚: https://metatrain--689.org.readthedocs.build/en/689/

[Luthaf]

This is a starting point, but we should discuss about how to expand the distributed training tests (currently the example only tests SOAP-BPNN).

Yes, the current tests are only intended to check the metatrain side of distributed training, and we use soap-bpnn as a "simple" model for this. We would have to add distributed tests to the relevant architectures.

[Luthaf]

This looks good, thanks!

[RMeli]

cscs-ci run

[Luthaf]

Thanks a lot!