Shared Memory Windows and MPI_MODE_NOSTORE

[jdinan]

Background

The "local window" of a shared memory window is the portion allocated by the local process (Sec. 12.2.3, definition of size argument). A process can perform stores to both the local and non-local portions of a shared memory window.

The MPI_MODE_NOSTORE assertion is defined as follows:

MPI_MODE_NOSTORE—the local window was not updated by stores (or local get or receive calls) since last synchronization. This may avoid the need for cache synchronization at the post call.

Problem Statement

It may be impossible for implementations to optimize window synchronization using the MPI_MODE_NOSTORE flag because processes can perform stores to the non-local portions of the window.

Possible Solutions

1. Update MPI_MODE_NOSTORE for shared memory windows to cover both local and non-local window portions.
   • This is not backward compatible.
2. Add an advice that MPI_MODE_NOSTORE may not be useful with shared memory windows.
   • This could cause latent bugs in MPI implementations that treat this assertion the same for all window kinds.
3. Add a restriction that MPI_MODE_NOSTORE is ignored for shared memory windows.

Proposed Solution

Add a restriction that MPI_MODE_NOSTORE is ignored for shared memory windows.

[jeffhammond]

we should do 1.

[jdinan]

@jeffhammond To accomplish (1) in a way that's backward compatible, we would need to do (3) and introduce a new flag. I guess for MPI 4.1 we will simply do (3) and leave the new flag for a future version of the spec.

[jdinan]

I guess that https://github.com/mpi-forum/mpi-standard/pull/708 adds a wrinkle in this plan. @devreal How should we deal with MPI_MODE_NOSTORE on other window flavors where the user calls MPI_Win_shared_query? Perhaps @jeffhammond is right and the best thing to do is clarify that NOSTORE means "no store"?

[jeffhammond]

NOSTORE means "no store"

Exactly 😄

I don't think there is a practical consequence to breaking backwards compatibility in the wording. I don't think anyone is optimizing out memory barriers to a degree that would be impacted by this. For example, neither MPICH nor Open-MPI does anything with NOSTORE right now.

[devreal]

For example, neither MPICH nor Open-MPI does anything with NOSTORE right now.

Same for Open MPI, AFAICS.

[devreal]

If https://github.com/mpi-forum/mpi-standard/pull/708 goes in, I'd say we extend NOSTORE to local and non-local window portions of any window type. Any backwards incompatibilities seem of academic nature ^^

[devreal]

Another possible solution:

4. Deprecate MPI_MODE_NOSTORE because its effect is rather limited and its semantic is not clear in the context of shared memory windows.
5. Say that these assertions only apply to the separate memory model.