implement hardware collectives for slingshot

[garlick]

Let's collect notes on what's needed to support hardware collectives on El Cap.

I'll summarize what I got from talking to @trws just now - I may be misremembering so please correct me.

HPE arranged a test recently and demonstrated a clear benefit of hardware collectives on job sizes above about 4000 nodes. Below that not so much. Since the resources required to enable collectives are somewhat limited, it may be best to only enable them on large jobs, and even then, perhaps only when requested. Performance can actually degrade when collectives are used on smaller jobs.

The process involves making one request to the fabric manager per job to set up (up to 24?) "trees". A handle is returned. The handle must be distributed to all the MPI tasks via environment variables. (The actual consumer is libfabric, which is used by Cray MPI). At the end of the job, the fabric manager is told to destroy the handle. The fabric manager operations could be performed by the the job shell, authenticated as the end user. Thus a job shell plugin could handle making these requests and distributing the handle to other parts of the job, and cleaning up afterwards. All self contained within the job.

The details of the authentication need to be understood. There were two options, a pre shared key, which presumably implies that Flux would need to do key management for users; and an Oauth like mechanism.

Fabric manager communications are simple REST.

[bronis]

I do not seem to have the HPE presentation with the graph of the results from the demonstration runs handy but my recollection is that the benefit starts being significant at much smaller node counts than 4,000. I am fairly certain it was significant at node counts > 1,000 (at most).

[bronis]

The requirement was that HPE would provide a mechanism to enable HW collectives for jobs running on 25% or more of the system nodes (meaning the lower bound depends on system size so many fewer nodes on Tuolumne) for MPI_COMM_WORLD and copies of MPI_COMM_WORLD. I believe the actual implementation is a bit more flexible than that but the results indicated that was a reasonable limitation to impose.

With respect to my previous comment, I know that the results indicated they would be beneficial at least for full system jobs on Tuolumne and I am fairly certain even the 25% (so about 280 nodes) level was significant.

Again on the performance front, I do not recall seeing any evidence of performance degradations. The issue is that smaller jobs are not guaranteed to have the switch resources -- the point of the minimum size requirement is to avoid having to solve an NP-hard problem in setting up the trees, not to ensure jobs are large enough to perform well.

[garlick]

Thanks Bronis, understood. I'm sure we can easily make these thresholds site-configurable.

Maybe it would be helpful to have HPE describe how their slurm plugin works in detail and then we can try to map that onto Flux's architecture? Is there a design document or something we can look at?

[garlick]

Update: two internal documents were shared by HPE. Although not HPE-confidential, it was requested that they not be published on github. They are:

• Slurm Slingshot Plugin Design (12 pages, no author, no date)
• Slurm Slingshot Collectives Design (12 pages, no author, no date)

The first describes how Slurm sets up the Cassini NIC using the Cray libcxi library, which talks directly to the kernel device driver. It covers VNI allocation and NIC resource partitioning, as discussed in

• MPI: Integrate with HPE's CXI library for allocating VNIs #24

The second describes how Slurm sets up multicast trees for jobs/steps and interacts with the Slingshot fabric manager and the libfabric cxi provider to enable the Slingshot hardware collective offload.

The described design is contained in the Slurm slingshot switch plugin and shasta mpi plugin.