use cases: add debugging chapter

Author: SteVwonder

App_Use_Cases.tex

@@ -85,143 +85,236 @@ \subsection{Use Case Details}
 
 There are other keys that are helpful to have before a synchronization point, this is not meant to be a comprehensive list.
 
-\section{Hybrid Programming Models}
-\label{app:uc-hybrid-programming-models}
+\section{Debugging}
+\label{app:uc-debugging}
 
-\subsection{Use Case Summary}
+This use case distills out the features/extensions requested in the RFCs that are related to debugging. We have identified parts of PR23 (Co-located process launch for debuggers), RFC0010 (MPIR-like query), RFC0002 (event pub/sub), and RFC0022 (Environmental Parameter Directives for Applications and Launchers) under this category.
 
-Hybrid applications (i.e., applications that utilize more than one lprogramming model, such as an MPI application that also uses OpenMP or PGAS) are growing in popularity, especially as chips with increasingly large numbers of cores and processors proliferate. Unfortunately, the various models currently operate under the assumption that they alone control execution. This leads to conflicts in hybrid applications. Deadlock of parallel applications can occur when one model prevents the other from making progress due to lack of coordination between the multiple programming models~\cite{2016:Hamidouche}.  Sub-optimal performance can also occur due to uncoordinated division of hardware resources between the programming models~\cite{2018:Vallee,ompix-moc}. This use-case offers potential solutions to the problem by providing a pathway for programming models to coordinate their actions.
+\subsection{Terminology}
 
-\subsection{Use Case Details}
+\subsubsection{Tools vs Debuggers}
+
+A \texttt{tool} is a process designed to monitor, record, analyze, or control the execution of another process.  Typically used for the purposes of profiling and debugging. A \texttt{first-party tool} runs within the address space of the application process while a \texttt{third-party tool} run within its own process.  A \texttt{debugger} is a third-party tool that inspects and controls an application process's execution using system-level debug APIs (e.g., \code{ptrace}).
+
+\subsubsection{Parallel Launching Methods}
+A \texttt{starter} program is a program responsible for launching a parallel runtime, such as \ac{MPI}.  \ac{PMIx} supports two primary methods for launching parallel applications under tools and debuggers: indirect and direct. In the indirect launching method, the tool is attached to the starter.  In the direct launching method, the tool takes the place of the starter.  
+\ac{PMIx} also supports attaching to already running programs via the \texttt{Process Acquisition} interfaces.
+
+\subsubsection{Process Synchronization}
+Process Synchronization is the technique tools use to start the processes of a parallel application such that the tools can still attach to the process early in it's lifetime.  Said another away, the tool must be able to start the application processes without them ``running away'' from the tool.  In the case of \ac{MPI}, this means stopping the applications processes before they return from \code{MPI_Init}.
+
+\subsubsection{Process Acquisition}\label{subsubsec:process-acq}
 
-\subsubsection{Identifying Active Programming Models}
+Process Acquisition is technique tools use to locate all of the processes, local and remote, of a given parallel application.  This typically boils down to collecting for every process in the parallel application: the hostname or IP of the machine running the process, the executable name, and the process ID.
 
-The current state-of-the-practice for programming models to detect one another is via set environment variables.  For example, OpenMP looks for environment variables to indicate that MPI is active.  Unfortunately, this technique is not completely reliable as environment variables change over time and with new software versions.  Also, the fact that an environment variable is present doesn't guarantee that a particular programming is in active use since Resource Managers routinely set environment variables "just in case" the application needs them. PMIx provides a reliable mechanism by which each library can determine that another library is in operation.
+\subsection{Use Case Details}
+\subsubsection{Direct-Launch Debugger Tool}
 
-When initializing PMIx, programming models can register themselves, including their name, version, and threading model.  This information is then cached locally and can then be read asynchronously by other programming models using PMIx's Event Notification system (see next section for more details).
+PMIx can support the tool itself using the PMIx spawn options to control the app’s startup, including directing the RM/application as to when to block and wait for tool attachment, or stipulating that an interceptor library be preloaded. However, this means that the user is restricted to whatever command line options the tool vendor has provided for operations such as process placement and binding, which places a significant burden on the tool vendor. An example might look like the following: \code{dbgr -n 3 ./myapp}.
 
-This initialization mechanism also allows libraries to share knowledge of each other's resources and intended resource utilization. For example, if OpenMP knows which hardware threads that MPI is using it could potentially avoid processor and cache contention.
+Assuming it is supported, co-launch of debugger daemons in this use-case is supported by adding a \code{pmix_app_t} to the \refapi{PMIx_Spawn} command, indicating that the resulting processes are debugger daemons by setting the \refattr{PMIX_DEBUGGER_DAEMONS} attribute.
 
-\littleheader{Code Example}
+\begingroup
+\begin{figure*}
+  \begin{center}
+    \includegraphics[width=\textwidth,height=\textheight,keepaspectratio]{figs/direct-launch}
+  \end{center}
+  \caption{Direct Launch}
+  \label{fig:direct_launch}
+\end{figure*}
+\endgroup
 
-\pmixCodeImportC[]{sources/hybrid-prog-model/declare_model.c}
 
 \littleheader{Related Interfaces}
 
-{\large \refapi{PMIx_Init}}
-\pasteSignature{PMIx_Init}
+{\large \refapi{PMIx_tool_init}}
+\pasteSignature{PMIx_tool_init}
+
+{\large \refapi{PMIx_Register_event_handler}}
+\pasteSignature{PMIx_Register_event_handler}
+
+{\large \refapi{PMIx_Query_info}}
+\pasteSignature{PMIx_Query_info}
+
+{\large \refapi{PMIx_Spawn}}
+\pasteSignature{PMIx_Spawn}
+
+{\large \refapi{PMIx_Get}}
+\pasteSignature{PMIx_Get}
+
+{\large \refapi{PMIx_Notify_event}}
+\pasteSignature{PMIx_Notify_event}
 
 \littleheader{Related Attributes}
 
-\pasteAttributeItem{PMIX_PROGRAMMING_MODEL}
-\pasteAttributeItem{PMIX_MODEL_LIBRARY_NAME}
-\pasteAttributeItem{PMIX_MODEL_LIBRARY_VERSION}
-\pasteAttributeItem{PMIX_THREADING_MODEL}
-\pasteAttributeItem{PMIX_MODEL_NUM_THREADS}
-\pasteAttributeItem{PMIX_MODEL_NUM_CPUS}
-\pasteAttributeItem{PMIX_MODEL_CPU_TYPE}
-\pasteAttributeItem{PMIX_MODEL_PHASE_NAME}
-\pasteAttributeItem{PMIX_MODEL_PHASE_TYPE}
-\pasteAttributeItem{PMIX_MODEL_AFFINITY_POLICY}
+\pasteAttributeItem{PMIX_QUERY_SPAWN_SUPPORT}
+\pasteAttributeItem{PMIX_QUERY_DEBUG_SUPPORT}
+\pasteAttributeItem{PMIX_DEBUG_STOP_IN_INIT}
+\pasteAttributeItem{PMIX_FWD_STDOUT}
+\pasteAttributeItem{PMIX_FWD_STDERR}
+\pasteAttributeItem{PMIX_NOTIFY_COMPLETION}
+\pasteAttributeItem{PMIX_SETUP_APP_ENVARS}
+\pasteAttributeItem{PMIX_DEBUGGER_DAEMONS}
+\pasteAttributeItem{PMIX_DEBUG_JOB}
+\pasteAttributeItem{PMIX_QUERY_LOCAL_PROC_TABLE}
 
-\subsubsection{Coordinating at Runtime}
+\littleheader{Related Constants}
 
-The PMIx Event Notification system provides a mechanism by which the resource manager can communicate system events to applications, thus providing applications with an opportunity to generate an appropriate response. Hybrid applications can leverage these events for cross-library coordination.
+\refconst{PMIX_DEBUG_WAITING_FOR_NOTIFY} \\
+\refconst{PMIX_DEBUGGER_RELEASE}
 
-Programming models can access the information provided by other programming models during their initialization using the event notification system.  In this case, programming models should register a callback for the \refconst{PMIX_MODEL_DECLARED} event.
+\subsubsection{Indirect-Launch Debugger Tool}
 
-Programming models can also use the PMIx event notification system to communicate dynamic information, such as entering a new application phase (\refconst{PMIX_MODEL_PHASE_NAME}) or a change in resources used (\refconst{PMIX_MODEL_RESOURCES}).  This dynamic information can be broadcast to other programming models using the \refapi{PMIx_Notify_event} function.  Other programming models can register callback functions to run when these events occur (i.e., callback functions) using \refapi{PMIx_Register_event_handler}.
+Executing a program under a tool using an intermediate launcher such as mpiexec can also be made possible. This requires some degree of coordination between the tool and the launcher. Ultimately, it is the launcher that is going to launch the application, and the tool must somehow inform it (and the application) that this is being done in a debug session so that the application knows to ``block'' until the tool attaches to it.
 
-\littleheader{Code Example}
+In this operational mode, the user invokes a tool (typically on a non-compute, or ``head'', node) that in turn uses mpiexec to launch their application – a typical command line might look like the following: \code{dbgr -dbgoption mpiexec -n 32 ./myapp}.
 
-Registering a callback to run when another programming model initializes:
-\pmixCodeJoinStart{}%
-\pmixCodeImportC[]{sources/hybrid-prog-model/declare_model_cb.c}
-\pmixCodeJoin{}%
-\pmixCodeImportC[]{sources/hybrid-prog-model/register_declare_model_cb.c}
-\pmixCodeJoinEnd{}%
+\begingroup
+\begin{figure*}
+  \begin{center}
+    \includegraphics[width=\textwidth,height=\textheight,keepaspectratio]{figs/indirect-launch}
+  \end{center}
+  \caption{Indirect Launch}
+  \label{fig:indirect_launch}
+\end{figure*}
+\endgroup
 
 
-Notifying an event:
-\pmixCodeImportC[]{sources/hybrid-prog-model/notify_event.c}
+\littleheader{Related Interfaces}
 
-\littleheader{Related Interfaces and Constants}
+{\large \refapi{PMIx_tool_init}}
+\pasteSignature{PMIx_tool_init}
+
+{\large \refapi{PMIx_Register_event_handler}}
+\pasteSignature{PMIx_Register_event_handler}
+
+{\large \refapi{PMIx_Spawn}}
+\pasteSignature{PMIx_Spawn}
 
 {\large \refapi{PMIx_Notify_event}}
 \pasteSignature{PMIx_Notify_event}
 
-\refconst{PMIX_MODEL_DECLARED} \\
-\refconst{PMIX_MODEL_RESOURCES} \\
-\refconst{PMIX_OPENMP_PARALLEL_ENTERED} \\
-\refconst{PMIX_OPENMP_PARALLEL_EXITED} \\
+{\large \refapi{PMIx_tool_attach_to_server}}
+\pasteSignature{PMIx_tool_attach_to_server}
+
+{\large \refapi{PMIx_Query_info}}
+\pasteSignature{PMIx_Query_info}
+
+{\large \refapi{PMIx_Get}}
+\pasteSignature{PMIx_Get}
+
+\littleheader{Related Attributes}
+
+\pasteAttributeItem{PMIX_SPAWN_TOOL}
+\pasteAttributeItem{PMIX_FWD_STDOUT}
+\pasteAttributeItem{PMIX_FWD_STDERR}
+\pasteAttributeItem{PMIX_SETUP_APP_ENVARS}
+\pasteAttributeItem{PMIX_DEBUG_STOP_IN_INIT}
+\pasteAttributeItem{PMIX_QUERY_PROC_TABLE}
+\pasteAttributeItem{PMIX_DEBUGGER_DAEMONS}
+\pasteAttributeItem{PMIX_DEBUG_JOB}
+\pasteAttributeItem{PMIX_FWD_STDOUT}
+\pasteAttributeItem{PMIX_FWD_STDERR}
+\pasteAttributeItem{PMIX_NOTIFY_COMPLETION}
+\pasteAttributeItem{PMIX_SETUP_APP_ENVARS}
+\pasteAttributeItem{PMIX_DEBUG_JOB}
+\pasteAttributeItem{PMIX_QUERY_LOCAL_PROC_TABLE}
+
+\littleheader{Related Constants}
+
+\refconst{PMIX_LAUNCHER_READY} \\
+\refconst{PMIX_LAUNCH_DIRECTIVE} \\
+\refconst{PMIX_LAUNCH_COMPLETE} \\
+\refconst{PMIX_DEBUG_WAITING_FOR_NOTIFY} \\
+\refconst{PMIX_DEBUGGER_RELEASE}
+
+\subsubsection{Attaching to a Running Job}
+
+PMIx supports attaching to an already running parallel job in two ways.  In the first way, the main process of a tool calls \refapi{PMIx_Query_info} with the \refattr{PMIX_QUERY_PROC_TABLE} attribute.  This returns an array of structs containing the information required for \hyperref[subsubsec:process-acq]{process acquisition}.  This includes remote hostnames, executable names, and process IDs.  In the second way, every tool daemon calls \refapi{PMIx_Query_info} with the \refattr{PMIX_QUERY_LOCAL_PROC_TABLE} attribute.  This returns a similar array of structs but only for processes on the same node.
+
+An example of this use-case may look like the following: \code{mpiexec -n32~./myApp \&\& dbgr attach \$!}.
+
+\begingroup
+\begin{figure*}
+  \begin{center}
+    \includegraphics[width=\textwidth,height=\textheight,keepaspectratio]{figs/process-acquisition}
+  \end{center}
+  \caption{Attaching to a Running Job}
+  \label{fig:proc_acq}
+\end{figure*}
+\endgroup
+
+{\large \refapi{PMIx_tool_init}}
+\pasteSignature{PMIx_tool_init}
 
 {\large \refapi{PMIx_Register_event_handler}}
 \pasteSignature{PMIx_Register_event_handler}
 
-%\pastePRIAttributeItem{PMIX_RANGE}
+{\large \refapi{PMIx_Query_info}}
+\pasteSignature{PMIx_Query_info}
+
+{\large \refapi{PMIx_Spawn}}
+\pasteSignature{PMIx_Spawn}
+
+\pasteAttributeItem{PMIX_QUERY_PROC_TABLE}
+\pasteAttributeItem{PMIX_DEBUGGER_DAEMONS}
+\pasteAttributeItem{PMIX_DEBUG_JOB}
+\pasteAttributeItem{PMIX_FWD_STDOUT}
+\pasteAttributeItem{PMIX_FWD_STDERR}
+\pasteAttributeItem{PMIX_NOTIFY_COMPLETION}
+\pasteAttributeItem{PMIX_SETUP_APP_ENVARS}
 
-{\large \refapi{pmix_event_notification_cbfunc_fn_t}}
-\pasteSignature{pmix_event_notification_cbfunc_fn_t}
+\pasteAttributeItem{PMIX_QUERY_NAMESPACES}
 
-\refconst{PMIX_EVENT_ACTION_COMPLETE}
+\subsubsection{Tool Interaction with RM}
 
+Tools can benefit from a mechanism by which they may interact with a local PMIx server that has opted to accept such connections along with support for tool connections to system-level PMIx servers, and a logging feature. To add support for tool connections to a specified system-level, PMIx server environments could choose to launch a set of PMIx servers to support a given allocation - these servers will (if so instructed) provide a tool rendezvous point that is tagged with their pid and typically placed in an allocation-specific temporary directory to allow for possible multi-tenancy scenarios. Supporting such operations requires that a system-level PMIx connection be provided which is not associated with a specific user or allocation. A new key has been added to direct the PMIx server to expose a rendezvous point specifically for this purpose.
 
+{\large \refapi{PMIx_Query_info_nb}}
+\pasteSignature{PMIx_Query_info_nb}
 
-\subsubsection{Coordinating at Runtime with Multiple Event Handlers}
+{\large \refapi{PMIx_Register_event_handler}}
+\pasteSignature{PMIx_Register_event_handler}
 
-Coordinating with a threading library such as OpenMP creates the need for separate event handlers for threads of the same process.  For example in an MPI+OpenMP hybrid application, the MPI thread and the main OpenMP thread may both want to be notified anytime an OpenMP worker thread enters a parallel region.  This requiring support for multiple threads to potentially register different event handlers against the same status code.
+{\large \refapi{PMIx_Deregister_event_handler}}
+\pasteSignature{PMIx_Deregister_event_handler}
 
-Multiple event handlers registered against the same event are processed in a chain-like manner based on the order in which they were registered, as modified by directive. Registrations against specific event codes are processed first, followed by registrations against multiple event codes and then any default registrations. At each point in the chain, an event handler is called by the PMIx progress thread and given a function to call when that handler has completed its operation. The handler callback notifies PMIx that the handler is done, returning a status code to indicate the result of its work. The results are appended to the array of prior results, with the returned values combined into an array within a single pmix_info_t as follows:
-\begin{itemize}
-\item \texttt{array[0]}: the event handler name provided at registration (may be an empty field if a string name was not given) will be in the key, with the pmix_status_t value returned by the handler
-\item \texttt{array[*]}: the array of results returned by the handler, if any.
-\end{itemize}
+{\large \refapi{PMIx_Notify_event}}
+\pasteSignature{PMIx_Notify_event}
 
-The current PMIx standard does not actually specify a default ordering for event handlers as they are being registered. However, it does include an inherent ordering for invocation. Specifically, PMIx stipulates that handlers be called in the following categorical order:
+{\large \refapi{PMIx_server_init}}
+\pasteSignature{PMIx_server_init}
 
-\begin{itemize}
-\item single status event handlers - i.e., handlers that were registered against a single specific status.
-\item multi status event handlers - those registered against more than one specific status
-\item default event handlers - those registered against no specific status
-\end{itemize}
+\littleheader{Job-specific events}
+\code{PMIX_EVENT_JOB_LEVEL       /* debugger attached, process failure */}
 
-\littleheader{Code Example}
+\littleheader{Environment events}
+\code{PMIX_EVENT_ENVIRO_LEVEL  /*ECC errors, temperature excursions */}
 
-From the OpenMP master thread:
+\littleheader{Errors detected by clients/peers}
+\code{Network fabric manager detects data corruption}
 
-\pmixCodeJoinStart{}%
-\pmixCodeImportC[]{sources/hybrid-prog-model/parallel_omp_cb.c}%
-\pmixCodeJoin{}%
-\pmixCodeImportC[]{sources/hybrid-prog-model/omp_thread.c}
-\pmixCodeJoinEnd{}%
+\subsubsection{Environmental Parameter Directives for Applications and Launchers}
 
-From the MPI thread:
+It is sometimes desirable or required that standard environmental variables (e.g., \code{PATH}, \code{LD_LIBRARY_PATH}, \code{LD_PRELOAD}) be modified prior to executing an application binary or a starter such as mpiexec - this is particularly true when tools/debuggers are used to start the application. This RFC proposes the definition of a new PMIx structure (\refstruct{pmix_envar_t}) and associated attributes for specifying such operations.
 
-\pmixCodeJoinStart{}%
-\pmixCodeImportC[]{sources/hybrid-prog-model/parallel_mpi_cb.c}
-\pmixCodeJoin{}%
-\pmixCodeImportC[]{sources/hybrid-prog-model/mpi_thread.c}
-\pmixCodeJoinEnd{}%
+\littleheader{Related Interfaces}
 
-\littleheader{Related Interfaces, Attributes, and Constants}
+{\large \refapi{PMIx_Spawn}}
+\pasteSignature{PMIx_Spawn}
 
-{\large \refapi{PMIx_Register_event_handler}}
-\pasteSignature{PMIx_Register_event_handler}
+\littleheader{Related Structs}
+
+\refstruct{pmix_envar_t}
+
+\littleheader{Related Attributes}
+
+\pasteAttributeItem{PMIX_SET_ENVAR}
+\pasteAttributeItem{PMIX_ADD_ENVAR}
+\pasteAttributeItem{PMIX_UNSET_ENVAR}
+\pasteAttributeItem{PMIX_PREPEND_ENVAR}
+\pasteAttributeItem{PMIX_APPEND_ENVAR}
 
-\pasteAttributeItem{PMIX_EVENT_HDLR_NAME}
-\pasteAttributeItem{PMIX_EVENT_HDLR_FIRST}
-\pasteAttributeItem{PMIX_EVENT_HDLR_LAST}
-\pasteAttributeItem{PMIX_EVENT_HDLR_FIRST_IN_CATEGORY}
-\pasteAttributeItem{PMIX_EVENT_HDLR_LAST_IN_CATEGORY}
-\pasteAttributeItem{PMIX_EVENT_HDLR_BEFORE}
-\pasteAttributeItem{PMIX_EVENT_HDLR_AFTER}
-\pasteAttributeItem{PMIX_EVENT_HDLR_APPEND}
-
-{\large \refapi{pmix_event_notification_cbfunc_fn_t}}
-\pasteSignature{pmix_event_notification_cbfunc_fn_t}
-
-\refconst{PMIX_EVENT_NO_ACTION_TAKEN} \\
-\refconst{PMIX_EVENT_PARTIAL_ACTION_TAKEN} \\
-\refconst{PMIX_EVENT_ACTION_DEFERRED} \\
+Resource managers and launchers must scan for relevant directives, modifying environmental parameters as directed. Directives are to be processed in the order in which they were given, starting with job-level directives (applied to each app) followed by app-level directives.
 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%