Preemption support

[rhc54]

As we start to work on defining preemption support, we have to consider several aspects of the problem:

• how to specify the local preemption policies. I suggest that this is not relevant to this organization - it really is a problem for the local host environment, each of which may choose to do it differently.

• how to query what the local preemption policy is, what options are definable by the app/tool, and how it is implemented. This is largely a question of attribute definition for query support. Should include indication of whether preemption is a “ctrl-z” (i.e., pause but remaining in memory) or a complete shutdown and removal - or both (perhaps selectable by app, maybe as required by replacement job)

• how to communicate an app’s preemption support to the host environment. For example, if I can support preemption, what handshake do I understand, what constraints exist on my support

• the preemption handshake itself. How does the host alert the app to proposed preemption, can the app respond with a counterproposal (e.g., take part of my allocation but leave some part of me running, I need N seconds to prepare, …), desired/required restart mechanism (e.g., restore from checkpoint), etc.

We currently have the following relevant definitions in pmix_common.h:

Session control attributes:

PMIX_SESSION_PREEMPT   (bool) preempt indicated jobs (given in accompanying pmix_info_t
                       via the PMIX_NSPACE attribute) in the specified session and recover
                       all their resources. If no PMIX_NSPACE is specified, then preempt
                       all jobs in the session.

Attributes relating to allocation requests:

PMIX_ALLOC_PREEMPTIBLE   (bool) by default, all jobs in the resulting allocation are
                         to be considered preemptible (overridable at per-job level)

Attributes relating to spawn requests:

PMIX_JOB_CTRL_PREEMPTIBLE    (bool) job can be pre-empted

Events:

PMIX_JCTRL_PREEMPT_ALERT    monitored by client to detect RM intends to preempt

PMIX_JCTRL_CHECKPOINT_COMPLETE   sent by client and monitored by server to
                        notify that a checkpoint operation has completed