Expose hardware fence capabilities

[devreal]

Currently the MPI standard does not provide any ordering guarantees for puts and gets unless remote completion is performed in between two operations. For atomic operations, only accesses to same/overlapping memory regions are guaranteed to be ordered (if origin and target are the same).

As far as I can see, many (most?) high-performance networks provide the notion of a fence to inject an ordering request into a stream of operations, i.e., to ensure that one operation is completed before the next is performed without having to wait for remote completion at the origin.

Example:

a = 0;
PUT(1->win, 1);
FENCE();
GET(1->win, &a);
FLUSH();
assert(a == 1);

This may be more efficient than having two flushes because we only have one round-trip instead of two, which is esp. useful for latency-bound use-cases (single-value atomics/reads/writes) and to ensure memory consistency in frameworks built on top of MPI RMA.

Libraries such as OpenShmem and UCX both offer this functionality.

Hence my question: has this been discussed (and dismissed?) in the past? Is there a reason not to add something like MPI_Win_order that triggers a hardware fence if available? (since MPI_Win_fence is already taken) On networks that don't support fences in hardware, implementations may always fall-back to a full flush, which is what the user would do otherwise at the moment.

[jeffhammond]

I would just write this: ACCUMULATE(op=REPLACE) GET_ACCUMULATE(op=NOOP) This is ordered and implementation uses the best method it can for ordering. If there’s a hardware fence operation (there almost certainly isn’t) then it can inject that after the ACCUMULATE or before the GET_ACCUMULATE.

[pavanbalaji]

No, the proposed approach is more flexible. For example,

for (1000 loops)
   MPI_Put
MPI_Win_order
for (1000 loops)
   MPI_Put

... can be more efficient than simply using accumulates, which has to order all operations.

[hjelmn]

I agree with this proposal. This brings me back to one of my pet peeves about MPI_Win_fence(). It is a fence + barrier operation. Wish it had been deprecated in MPI-3.0.

The question is what do we call this new operation given the obvious name is taken?

[jeffhammond]

No, the proposed approach is more flexible. For example, for (1000 loops) MPI_Put MPI_Win_order for (1000 loops) MPI_Put

Can the difference between order and flush be observed reliably relative to 2000 RMA operations? I’m not sure you’ll even be able to see a difference in shared memory with an appropriate weakly ordered CPU like POWER. How many cycles difference between msync and mfence (or whatever the instructions are called)? How many cycles for loads from memory (not cache)? To be clear, we should not add features to the MPI standard to optimize SHMEM microbenchmarks.

... can be more efficient than simply using accumulates, which has to order all operations.

It has to order the operations that are used on the window in question. Add info key to subset operations per window if it will help.

[pavanbalaji]

I think for 1000 operations, you'll surely see some difference, at least with MPICH. If not, you can replace that with 100 or 500 or whatever number makes sense for your network. Moreover, for networks that natively provide ordered communication, the difference is even higher. The point is very simple: accumulate gives ordering for all operations; this proposal gives more fine-grained ordering of such ordering.

One aspect that the proposal does not explicitly mention (but I think is implied) is that of atomicity. Presumably, the operations after the MPI_Win_order operation are targeting the same memory locations as after the MPI_Win_order (otherwise you wouldn't need ordering anyway). So this proposal allows the user to tell MPI that everything before MPI_Win_order is nonatomic.

[devreal]

There is another use-case for fences that may be more intuitive and cannot be realized with the ordering guarantees of RMA atomics per-se: any algorithmic requirement for the order in which different variables should be modified. An example I'm currently working on is a signalling scheme where a process writes to one or more variables on another process and eventually sets a flag to signal completion of the modifications:

Process 0:

PUT(1->win, val_offset, val);
FENCE(1->win);
PUT(1->win, flag_offset, 1);
FLUSH(1->win);

Process 1 (waiting for Process 0 to complete, flushes omitted):

while (GET(1->win, flag_offset) != 0) { }
GET(1->win, val_offset);

This can easily be extended to more complex scenarios (where you couldn't just replace the sync through flag_offset with a send/recv or barrier). If something like the above code is done in a critical part of the application repeatedly, the latency of a second flush (instead of the fence) on Process 1 really adds up.

AFAICS, a flush with remote completion should really only be required in two scenarios:

1. If there are data dependencies between two operations at the origin, i.e., operation Op2 has the result of a previous Op1 as input (e.g., fetching an offset in Op1 to write to in Op2); or
2. If completion at the target has to be ensured before some out-of-band (read: non-RMA) signalling is done, e.g., before entering a barrier or sending a message.

I will try to come up with a benchmark to see whether there is a significant difference (I'd be surprised/disappointed if there wasn't, what's a fence for then otherwise? ^^).

[jeffhammond]

@devreal You're reimplementing the SHMEM send-recv pattern, which is best implemented in MPI using MPI_Send and MPI_Recv. Put-with-notification paired with spinning on a read is overrated as a message-passing implementation.

I can't remember right now, but I also thought that it is undefined whether MPI_Get can read a memory location that may be updated concurrently. When I write such code, I always use MPI_Fetch_and_op(NO_OP).

[jeffhammond]

@pavanbalaji I meant whether you'd notice fence vs flush in the middle of 1000 RMA ops. There are two scenarios to optimize for: latency and throughput. Accumulate ops probably minimize latency whereas non-atomic RMA plus flush optimizes for throughput. Sure, they aren't trivially interchangeable inside of OSHMPI, but a proper MPI application can use both depending on need.

[devreal]

@jeffhammond As I said, that is a trivial example. Consider all processes doing similar data exchange with random targets. Blocking send/recv won't get you very far because you easily end up with a pair of processes stuck in send. Once you use non-blocking sends and recvs you start spending quite some time testing for completed requests. I'm working on a paper that explores the merits of using RMA for this type of communication. Latency is not one of them (at least in its current state), throughput actually is on some systems.

Apart from that, in the PGAS framework I'm working on (DASH; based on MPI RMA as a communication backend) we are forced to finish pretty much every put with a remote flush (except if the users know what they are doing an tell us so) because the next put or get might point to the same remote memory location and we have to ensure correct ordering. Tracking accesses to individual global memory locations is not an option for us. Having support for ordered RMA without forcing remote completion would be tremendously helpful. Sure, serializing RMA accesses that way might incur some performance penalty compared to non-ordered puts (additional function call, serialized processing, ordering overhead) but I expect it to be significantly faster than what we have to do right now. (of course we can discuss whether single-element global memory access is worth it after all but that's beyond the point here)

Plus, many systems offer this feature so why not provide access to it?

I can't remember right now, but I also thought that it is undefined whether MPI_Get can read a memory location that may be updated concurrently.

The result in undefined in that you can end up with reading partial updates. That shouldn't be a problem if you flip a value from from zero to one. And the actual implementation indeed uses atomics because there may be more than one writer. The ordering issue I tried to show remains though.

[jdinan]

@devreal Having spent a fair amount of time working on both MPI RMA and OpenSHMEM, I can tell you that (1) fence is a concept worth having in MPI RMA and (2) the performance benefit is network dependent. On ordered networks like InfiniBand, the fence is effectively a no-op (yay). On most unordered networks, the fence is effectively the same thing as a flush (oh...). If you happen to have triggered ops, you can do something more creative, but there are few such networks.

I'm all for adding this to MPI RMA, but bear in mind the performance caveats -- if you have a network that is good at PGAS (i.e. reliable, unordered datagrams) there's a good chance it will be bad at fence.

FWIW, you can use MPI accumulates if you want ordering. This works fine for scalar operations, but most networks have lower throughput for vector atomics versus put/get.

[jeffhammond]

We let RMA users turn off ordering in accumulate. Why not let them turn off atomicity as well? We'd have to change a nontrivial amount of text but the end result would be what appears to be desired here, and perhaps a more efficient implementation of it, since it would not require an additional function call to prescribe ordered put and get.

Feature	ordered	unordered
atomic	accumulate ops	accumulate ops + info
not atomic	put/get + flush	put/get

Performance	ordered	unordered
atomic	🙂	🤔
not atomic	😕	🙂

[pavanbalaji]

What does ordering mean without atomicity for a single memory location?

I think @devreal's use case is for multiple separate memory locations. So, in some sense, it's orthogonal to PUT/GET vs. ACC/GACC.

[pavanbalaji]

I think the point of this proposal is that FLUSH does two things: (1) ensures operations after the FLUSH are ordered with operations before FLUSH and (2) makes sure the data is visible to other processes. This proposal is to decouple them, so the user can get just (1) without (2), if needed. This is independent of ACC/GACC vs. PUT/GET.

[jdinan]

Thanks, I had missed that. This is much stronger than what shmem_fence guarantees. shmem_fence only guarantees that the targeted process will see the operations in order. That is, point-to-point, not global ordering.

[pavanbalaji]

I believe this proposal is point-to-point ordering too, i.e., a subset of FLUSH, not FLUSH_ALL.