Add support for topology-restricted RMA windows

[hjelmn]

This is something I have been thinking about for some time and have intended to introduce for MPI-5.

Right now (MPI-4) the topology of a communicator is effectively ignored on window creation. There is no mechanism with RMA (or point-to-point for that matter) for the user to indicate that they will only communicate on the topology of the communicator. If this information was available to the MPI implementation it could make better optimization choices for the resulting MPI object.

Proposal:

Add a new info key for MPI window or communicator creation: mpi_assert_restrict_to_topology

This could be restricted to MPI windows but it might be worth elevating to the entire communicator and applying to point-to-point. If this info key is set then the resulting communicator will only allow communication to and synchronization with neighboring processes. The target rank would be the rank in the communicator. We could change that to be the neighbor index in the topology (which is fairly well defined) with another info key.

Benefits:

The information stored in the local MPI_Win object could be limited to just what is necessary to communicate with the immediate neighbors. This could improve performance, reduce memory overhead, reduce resource utilization, or all of the above. In the prototype implementation (which I completed several years ago) it had all of these impacts.

I will add more as I remember all the details.

Thoughts?

[hjelmn]

I should point out that many of the applications I have worked with that are looking at RMA communication have fairly stable communication patterns with a fairly constant (and small) number of peers even at larger scales.

[jeffhammond]

Every one-sided code I’ve analyzed communicated with essentially all the other processes. NWChem does this with most windows. Realistic SHMEM codes behave similarly. One can use MPI profilers with ARMCI-MPI or OSHMPI to see this. This feature will certainly help with codes that do structured communication, and we might see more of those in the future. I also support this feature if it encourages implementers to register memory during window construction instead of doing it on-demand.

[mhoemmen]

Commenting to follow.

We could change that to be the neighbor index in the topology (which is fairly well defined) with another info key.

Would this improve performance a lot? I'm not a huge fan of an optimization unexpectedly changing the meaning of an argument, e.g., for libraries that might not have control over setting up the window.

[hjelmn]

@mhoemmen It would change the peer lookup to being an index into an array vs determining which neighbor index has that rank. For topologies with small numbers of edges the difference is small but it could get bigger with higher connectively.

That said, I am also not completely sure that I want that additional feature. It seems useful given, as I pointed out, that neighbor ordering is well defined but it does change the meaning of target from MPI rank to neighbor index. That said, the user would specifically have to request the feature. It is easy enough to implement as an MPI-X feature.

[jdinan]

@hjelmn I think you could achieve the same result with an info assertion that the application only communicates with its neighbors in the input communicator's topology. Minor difference in how you name the info key, but we have precedent in MPI 4.0 for info assertions. I've also seen an increasing number of applications that use RMA for neighborhood communication. Of course Jeff is also correct, and both neighborhood and global communication patterns are important usage models. Knowing which to expect, the implementation can make different choices about how it manages connections, which can improve performance.

[hjelmn]

@jdinan Sure, I added assert to the info key name. I am sure it will go through a couple of revisions before a formal MPI forum proposal. What do you think about adding such a key to both communicators and windows?

The prototype (osc/rdma in an Open MPI branch) disables all the internal bookkeeping/storage for peer lookup and caches all peer info locally when the info key is set. Global windows continue to work as before. The change was relatively small and isolated.

[jdinan]

@hjelmn Sounds like a good idea to me. One obnoxious issue we will need to resolve is how to specify that an info key can be applied to both communicators and windows.

[mhoemmen]

@hjelmn wrote:

It would change the peer lookup to being an index into an array vs determining which neighbor index has that rank. For topologies with small numbers of edges the difference is small but it could get bigger with higher connectively.

Thanks for explaining! : - ) I do wish functions that took neighbor indices instead of process ranks had different names, or some other way to distinguish them from the usual case.

[hjelmn]

@mhoemmen I agree. Though, I think those who created MPI topologies sort of messed up. IMHO the rank in a topology should have always been defined as the neighbor index. For example, MPI_Comm_rank should return MPI_UNDEFINED if a MPI process is not a neighbor of itself in the topology. Otherwise it should return the first index where it appears in the topology (since a peer may appear more than once).

Topology communicators should have always have been special communicators with restricted communication patterns. This is an opinion I have shared at past forum meetings. I feel like MPI topologies were a missed opportunity. If they wanted rank-reordering based off topology they should have named them as such. Something like MPI_Comm_reorder_cartesian. But that is way before my time on MPI.

We are allowed to mess somewhat with the "ranks" as the standard does require that the rank of a neighbor be retrieved using one of the topology function (MPI_Cart_rank for example). That is what makes the prototype work with existing code.

And, I agree, generics are not the answer here. I am still 100% for breaking backwards compatibility with MPI-5. Something I know is controversial but does have support.

I will press forward with the first of the info keys and we will see where the other one goes. If it is in Open MPI only as MPIX then that is the way it is :).

[mhoemmen]

@hjelmn Thanks for explaining! I don't like using the same type for "process rank" and "neighbor index," but it's out of scope to fix that, alas.

[devreal]

Worth considering in the same vein: #5

[devreal]

Questions from the discussion at the RMA WG meeting 03/01/2022: @hjelmn can you provide more details on the performance benefits of topology restricted RMA? Are you still planning to pursue this?

@Wee-Free-Scot appears to know of a use-case at ECMWF that uses RMA with a restricted set of peers where synchronization might benefit from this restriction.

[Wee-Free-Scot]

The ECMWF use-case relates to their main weather code, IFS (used daily to produce forecast outputs used by many countries). One of the functions involves computing where wind will carry particular chunks of atmosphere -- the MPI process currently responsible for a chunk of atmosphere can easily compute the geographic location after one time-step and compute where the data should be placed in memory, either in its own memory space or the memory space of another MPI process. Given the assumptions about maximum wind speed (and consequently, maximum distance travelled in a single time-step), achieving the necessary communication with a "halo-exchange" pattern requires huge halo depths (sometimes over 50% of the neighbour process memory, i.e. all memory is part of at least one halo region). Point-to-point or collective communication in MPI would require the entire halo region to be transmitted every time step in order to make sure that the data always arrived at the right place, even if the wind speed is lower than its maximum (which it always is, in practice). An RMA implementation exposes the entire memory space to a limited set of "nearby" MPI processes and invites those MPI processes to MPI_Put any data that needs to be transferred to the correct memory locations (although MPI_Accumulate might be needed if there could be clashes). An RMA synchronisation is performed in each time step to guarantee visibility of the data transfers.

A topology-restricted MPI_Win_fence would be conceptually similar to a neighbourhood collective. Would that perform better than a global barrier-like collective synchronisation?

Alternatively, the PSCW approach would use MPI_Win_wait on the exposure group or MPI processes, i.e. multiple separate-but-overlapping group-wise synchronisations. Does that done-everywhere-but-not-collective approach perform better than a global collective (MPI_Win_fence as currently specified) or a neighbourhood collective (MPI_Win_fence with added topology restriction) approach?

The passive target approach is not suitable in this case because only the origin knows about completion at the target.