Skip to content

Commit 162242a

Browse files
rhc54rhc54
committed
checkpoint
Signed-off-by: Ralph Castain <[email protected]>
1 parent 371718f commit 162242a

File tree

1 file changed

+87
-128
lines changed

1 file changed

+87
-128
lines changed

‎Chap_API_Job_Mgmt.tex‎

Lines changed: 87 additions & 128 deletions
Original file line numberDiff line numberDiff line change
@@ -616,7 +616,7 @@ \section{Process and Job Monitoring}
616616
job. Various watchdog methods have been developed for detecting this situation, including requiring a periodic ``heartbeat''
617617
from the application and monitoring a specified file for changes in size and/or modification time.
618618

619-
The following \acp{API} allow applications to request monitoring, directing what is to be monitored, the frequency of the associated check, whether or not the application is to be notified (via the event notification subsystem) of stall detection, and other characteristics of the operation.
619+
The following \acp{API} allow applications to request monitoring, directing what is to be monitored, the frequency of the associated check, whether or not the application is to be notified (via the event notification subsystem) of stall detection, and other characteristics of the operation. In addition, statistics on the resource usage at the individual process level and/or overall node level (including usage information on disks and/or network interfaces) can be measured and periodically reported back to the requestor.
620620

621621
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
622622
\subsection{\code{PMIx_Process_monitor}}
@@ -664,13 +664,13 @@ \subsection{\code{PMIx_Process_monitor}}
664664
\item \refattr{PMIX_MONITOR_HEARTBEAT_DROPS}
665665
\end{itemize}
666666
\item \refattr{PMIX_SEND_HEARTBEAT}
667-
\item \refattr{PMIX_MONITOR_FILE} The associated \refarg{directives} array may include any of the following:
667+
\item \refattr{PMIX_MONITOR_FILE_CHANGES} The associated \refarg{directives} array may include any of the following:
668668
\begin{itemize}
669-
\item \refattr{PMIX_MONITOR_FILE_SIZE}
670-
\item \refattr{PMIX_MONITOR_FILE_ACCESS}
671-
\item \refattr{PMIX_MONITOR_FILE_MODIFY}
672669
\item \refattr{PMIX_MONITOR_FILE_CHECK_TIME}
673670
\item \refattr{PMIX_MONITOR_FILE_DROPS}
671+
\item \refattr{PMIX_MONITOR_TARGET_FILES}
672+
\item \refattr{PMIX_MONITOR_TARGET_NODES}. Monitor the given files on the specified nodes, where present.
673+
\item \refattr{PMIX_MONITOR_TARGET_NODEIDS}. Monitor the given files on the specified nodes, where present.
674674
\end{itemize}
675675
\item \refattr{PMIX_MONITOR_PROC_RESOURCE_USAGE} The associated \refarg{directives} array may include any of the following:
676676
\begin{itemize}
@@ -685,7 +685,23 @@ \subsection{\code{PMIx_Process_monitor}}
685685
\item \refattr{PMIX_MONITOR_RESOURCE_RATE}
686686
\item \refattr{PMIX_MONITOR_TARGET_NODES}
687687
\item \refattr{PMIX_MONITOR_TARGET_NODEIDS}
688-
\item \refattr{PMIX_MONITOR_TARGET_PROCS} Monitor the nodes where the specified processes are located.
688+
\item \refattr{PMIX_MONITOR_TARGET_PROCS}. Monitor the nodes where the specified processes are located.
689+
\end{itemize}
690+
\item \refattr{PMIX_MONITOR_DISK_RESOURCE_USAGE} The associated \refarg{directives} array may include any of the following:
691+
\begin{itemize}
692+
\item \refattr{PMIX_MONITOR_RESOURCE_RATE}
693+
\item \refattr{PMIX_MONITOR_TARGET_DISKS}
694+
\item \refattr{PMIX_MONITOR_TARGET_NODES}
695+
\item \refattr{PMIX_MONITOR_TARGET_NODEIDS}
696+
\item \refattr{PMIX_MONITOR_TARGET_PROCS}. Monitor the nodes where the specified processes are located.
697+
\end{itemize}
698+
\item \refattr{PMIX_MONITOR_NETWORK_RESOURCE_USAGE} The associated \refarg{directives} array may include any of the following:
699+
\begin{itemize}
700+
\item \refattr{PMIX_MONITOR_RESOURCE_RATE}
701+
\item \refattr{PMIX_MONITOR_TARGET_NETS}
702+
\item \refattr{PMIX_MONITOR_TARGET_NODES}
703+
\item \refattr{PMIX_MONITOR_TARGET_NODEIDS}
704+
\item \refattr{PMIX_MONITOR_TARGET_PROCS}. Monitor the nodes where the specified processes are located.
689705
\end{itemize}
690706
\end{itemize}
691707

@@ -695,27 +711,33 @@ \subsection{\code{PMIx_Process_monitor}}
695711
\item \refattr{PMIX_MONITOR_ID}
696712
\item \refattr{PMIX_MONITOR_APP_CONTROL}
697713
\item \refattr{PMIX_RANGE} Non-default range to be used when generating the associated event for this monitoring action.
714+
\item \refattr{PMIX_MONITOR_LOCAL_ONLY}
698715
\end{itemize}
699716
\optattrend
700717

701718
%%%%
702719
\descr
703720

704-
Request that application processes and/or files be monitored via several possible methods.
705-
For example, that the server monitor a given process for periodic heartbeats as an indication that the process has not become ``wedged''.
706-
When a monitor detects the specified alarm condition, it will generate an event notification using the provided error code and passing along any available relevant information.
707-
It is up to the caller to register a corresponding event handler.
721+
This \ac{API} can be used for two purposes:
722+
723+
\begin{itemize}
724+
\item Request that application processes and/or files be monitored for activity via several possible methods. For example, that the server monitor a given process for periodic heartbeats as an indication that the process has not become ``wedged''. When a monitor detects the specified alarm condition, it will generate an event notification using the provided error code and passing along any available relevant information. It is up to the caller to register a corresponding event handler.
725+
\item Report resource usage statistics for processes and/or nodes, including disk and network interfaces attached to nodes. This can be done on a per-request basis, or periodically updated on a time interval specified by the \refattr{PMIX_MONITOR_RESOURCE_RATE} attribute.
726+
\end{itemize}
708727

709728
The \refarg{monitor} argument is an attribute indicating the type of monitor being requested.
710-
For example, \refattr{PMIX_MONITOR_FILE_CHANGES} to indicate that the requestor is asking that a file be monitored.
729+
For example, \refattr{PMIX_MONITOR_FILE_CHANGES} to indicate that the requestor is asking that a file be monitored, or \refattr{PMIX_MONITOR_PROC_RESOURCE_USAGE} to obtain a report on process resource usage.
711730

712-
The \refarg{error} argument is the status code to be used when generating an event notification alerting that the monitor has been triggered.
713-
The range of the notification defaults to \refconst{PMIX_RANGE_NAMESPACE}.
714-
This can be changed by providing a \refattr{PMIX_RANGE} directive.
731+
The \refarg{error} argument is the status code to be used when generating an event notification alerting that the monitor has been triggered or to receive a periodic resource usage update.
732+
The range of the notification defaults to \refconst{PMIX_RANGE_NAMESPACE} for alarm events, and to \refconst{PMIX_RANGE_CUSTOM} for resource usage updates to ensure delivery solely to the requesting process. This can be changed by providing a \refattr{PMIX_RANGE} directive.
715733

716-
The \refarg{directives} argument characterizes the monitoring request (e.g., monitor file size) and frequency of checking to be done
734+
The \refarg{directives} argument characterizes the monitoring request (e.g., monitor file size or specific resource usage metrics to be measured) and frequency of checking to be done.
717735

718-
The returned \refarg{status} indicates whether or not the request was granted, and information as to the reason for any denial of the request shall be returned in the \refarg{results} array.
736+
The returned \refarg{status} indicates whether or not the request was granted, and information as to the reason for any denial of the request shall be returned in the \refarg{results} array. If the request was successful, then any measured values will be returned in the \refarg{results}.
737+
738+
\adviceuserstart
739+
A \refconst{PMIX_SUCCESS} return status only indicates that no error was encountered when executing the request and does not guarantee a non-\code{NULL} value for \refarg{results} when requesting resource usage statistics. For example, a request for resource usage by processes on a specified node that is not currently executing any user-level processes will return success to indicate that the request was executed without error, but a \code{NULL} \refarg{results} array because no statistics were returned.
740+
\adviceuserend
719741

720742
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
721743
\subsection{\code{PMIx_Process_monitor_nb}}
@@ -753,77 +775,14 @@ \subsection{\code{PMIx_Process_monitor_nb}}
753775

754776
\returnstart
755777
\begin{itemize}
756-
\item \refconst{PMIX_OPERATION_SUCCEEDED}, indicating that the request was immediately processed and returned \textit{success} - the \refarg{cbfunc} will \textit{not} be called
778+
\item \refconst{PMIX_OPERATION_SUCCEEDED}, indicating that the request was immediately processed and returned \textit{success} - the \refarg{cbfunc} will \textit{not} be called, and no resource usage results will be returned by the \ac{API} itself.
757779
\end{itemize}
758780
\returnend
759781

760-
\optattrstart
761-
The following attributes may be implemented by a \ac{PMIx} library or by the host environment. If an attribute is supported by the \ac{PMIx} server library, then the library must not pass the supported attributes to the host environment unless the requested action involves other nodes. In addition, the library is \textit{required} to add the \refAttributeItem{PMIX_USERID} and the \refAttributeItem{PMIX_GRPID} attributes of the requesting process to the directives array when it passes actions to its host.
762-
763-
The \refarg{monitor} argument may contain any of the following actions:
764-
765-
\begin{itemize}
766-
\item \refattr{PMIX_MONITOR_CANCEL}
767-
\item \refattr{PMIX_MONITOR_HEARTBEAT} The associated \refarg{directives} array may include any of the following:
768-
\begin{itemize}
769-
\item \refattr{PMIX_MONITOR_HEARTBEAT_TIME}
770-
\item \refattr{PMIX_MONITOR_HEARTBEAT_DROPS}
771-
\end{itemize}
772-
\item \refattr{PMIX_SEND_HEARTBEAT}
773-
\item \refattr{PMIX_MONITOR_FILE_CHANGES} The associated \refarg{directives} array may include any of the following:
774-
\begin{itemize}
775-
\item \refattr{PMIX_MONITOR_FILE_CHECK_TIME}
776-
\item \refattr{PMIX_MONITOR_FILE_DROPS}
777-
\item \refattr{PMIX_MONITOR_TARGET_FILES}
778-
\item \refattr{PMIX_MONITOR_TARGET_NODES}. Monitor the given files on the specified nodes, where present.
779-
\item \refattr{PMIX_MONITOR_TARGET_NODEIDS}. Monitor the given files on the specified nodes, where present.
780-
\end{itemize}
781-
\item \refattr{PMIX_MONITOR_PROC_RESOURCE_USAGE} The associated \refarg{directives} array may include any of the following:
782-
\begin{itemize}
783-
\item \refattr{PMIX_MONITOR_RESOURCE_RATE}
784-
\item \refattr{PMIX_MONITOR_TARGET_PROCS}
785-
\item \refattr{PMIX_MONITOR_TARGET_PIDS}
786-
\item \refattr{PMIX_MONITOR_TARGET_NODES}. All processes on the specified nodes are to be monitored.
787-
\item \refattr{PMIX_MONITOR_TARGET_NODEIDS}. All processes on the specified nodes are to be monitored.
788-
\end{itemize}
789-
\item \refattr{PMIX_MONITOR_NODE_RESOURCE_USAGE} The associated \refarg{directives} array may include any of the following:
790-
\begin{itemize}
791-
\item \refattr{PMIX_MONITOR_RESOURCE_RATE}
792-
\item \refattr{PMIX_MONITOR_TARGET_NODES}
793-
\item \refattr{PMIX_MONITOR_TARGET_NODEIDS}
794-
\item \refattr{PMIX_MONITOR_TARGET_PROCS}. Monitor the nodes where the specified processes are located.
795-
\end{itemize}
796-
\item \refattr{PMIX_MONITOR_DISK_RESOURCE_USAGE} The associated \refarg{directives} array may include any of the following:
797-
\begin{itemize}
798-
\item \refattr{PMIX_MONITOR_RESOURCE_RATE}
799-
\item \refattr{PMIX_MONITOR_TARGET_DISKS}
800-
\item \refattr{PMIX_MONITOR_TARGET_NODES}
801-
\item \refattr{PMIX_MONITOR_TARGET_NODEIDS}
802-
\item \refattr{PMIX_MONITOR_TARGET_PROCS}. Monitor the nodes where the specified processes are located.
803-
\end{itemize}
804-
\item \refattr{PMIX_MONITOR_NETWORK_RESOURCE_USAGE} The associated \refarg{directives} array may include any of the following:
805-
\begin{itemize}
806-
\item \refattr{PMIX_MONITOR_RESOURCE_RATE}
807-
\item \refattr{PMIX_MONITOR_TARGET_NETS}
808-
\item \refattr{PMIX_MONITOR_TARGET_NODES}
809-
\item \refattr{PMIX_MONITOR_TARGET_NODEIDS}
810-
\item \refattr{PMIX_MONITOR_TARGET_PROCS}. Monitor the nodes where the specified processes are located.
811-
\end{itemize}
812-
\end{itemize}
813-
814-
In addition to action-specific directives, the \refarg{directives} array may include:
815-
816-
\begin{itemize}
817-
\item \refattr{PMIX_MONITOR_ID}
818-
\item \refattr{PMIX_MONITOR_APP_CONTROL}
819-
\item \refattr{PMIX_RANGE} Non-default range to be used when generating the associated event for this monitoring action.
820-
\end{itemize}
821-
\optattrend
822-
823782
%%%%
824783
\descr
825784

826-
Non-blocking form of the \refapi{PMIx_Process_monitor} \ac{API}. The \refarg{cbfunc} function provides a \refarg{status} to indicate whether or not the request was granted, and to provide some information as to the reason for any denial in the \refapi{pmix_info_cbfunc_t} array of \refstruct{pmix_info_t} structures.
785+
Non-blocking form of the \refapi{PMIx_Process_monitor} \ac{API}. The \refarg{cbfunc} function provides a \refarg{status} to indicate whether or not the request was granted, and to provide some information as to the reason for any denial in the \refapi{pmix_info_cbfunc_t} array of \refstruct{pmix_info_t} structures. Any resource usage data generated by the function will be returned there as well.
827786

828787
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
829788
\subsection{\code{PMIx_Heartbeat}}
@@ -998,9 +957,9 @@ \subsection{Monitoring attributes}
998957
attributes include:
999958

1000959
\begin{itemize}
1001-
\refattr{PMIX_MONITOR_FILE_SIZE}
1002-
\refattr{PMIX_MONITOR_FILE_ACCESS}
1003-
\refattr{PMIX_MONITOR_FILE_MODIFY}
960+
\item \refattr{PMIX_MONITOR_FILE_SIZE}
961+
\item \refattr{PMIX_MONITOR_FILE_ACCESS}
962+
\item \refattr{PMIX_MONITOR_FILE_MODIFY}
1004963
\end{itemize}
1005964
}
1006965
%
@@ -1064,7 +1023,8 @@ \subsection{Monitoring attributes}
10641023
}
10651024
%
10661025
\declareAttributeProvisional{PMIX_MONITOR_LOCAL_ONLY}{"pmix.monitor.local"}{bool}{
1067-
Restrict data collection to the local host, regardless of any provided targets}
1026+
Restrict data collection to the local host, regardless of any provided targets
1027+
}
10681028
%
10691029
\declareAttributeProvisional{PMIX_MONITOR_PROC_RESOURCE_USAGE}{"pmix.monitor.presuse"}{pmix_data_array_t*}{
10701030
Monitor the resources specified in the provided \refstruct{pmix_data_array_t} of \refstruct{pmix_info_t}. If the provided array
@@ -1340,53 +1300,52 @@ \subsubsection{Node resource usage}
13401300
implementations and \acp{OS}, depending upon availability and access restrictions, and the
13411301
provided list of requested values. Except for
13421302
the node ID as the first element, ordering of information in the array is arbitrary.
1343-
}
13441303

13451304
Optional information that may be included (see \href{https://www.kernel.org/doc/html/latest/filesystems/proc.html#kernel-data}{KERNEL} and \href{https://www.kernel.org/doc/html/latest/filesystems/proc.html#meminfo}{MEMINFO} for a detailed description of the following fields):
13461305
\begin{itemize}
1347-
\item \refattr{PMIX_HOSTNAME}
1348-
\item \refattr{PMIX_NODEID}
1349-
\item \declareAttributeProvisional{PMIX_NODE_LOAD_AVG}{"pmix.node.la"}{float}{
1350-
Load average of last minute
1351-
}
1352-
\item \declareAttributeProvisional{PMIX_NODE_LOAD_AVG5}{"pmix.node.la5"}{float}{
1353-
Load average of last five minutes
1354-
}
1355-
\item \declareAttributeProvisional{PMIX_NODE_LOAD_AVG15}{"pmix.node.la15"}{float}{
1356-
Load average of last fifteen minutes
1357-
}
1358-
\item \declareAttributeProvisional{PMIX_NODE_MEM_TOTAL}{"pmix.node.mtot"}{float}{
1359-
Total usable RAM (i.e., physical RAM minus reserved bits and kernel binary code). In MBytes
1360-
}
1361-
\item \declareAttributeProvisional{PMIX_NODE_MEM_FREE}{"pmix.node.mfree"}{float}{
1362-
Total free RAM. In MBytes
1363-
}
1364-
\item \declareAttributeProvisional{PMIX_NODE_MEM_BUFFERS}{"pmix.node.mbuf"}{float}{
1365-
Temporary storage for raw disk blocks. In MBytes
1366-
}
1367-
\item \declareAttributeProvisional{PMIX_NODE_MEM_CACHED}{"pmix.node.mcache"}{float}{
1368-
In-memory cache for files read from the disk (the pagecache) as well as tmpfs and shmem. In MBytes.
1369-
Doesn't include \refattr{PMIX_NODE_MEM_SWAP_CACHED}.
1370-
}
1371-
\item \declareAttributeProvisional{PMIX_NODE_MEM_SWAP_CACHED}{"pmix.node.mswpc"}{float}{
1372-
Memory that once was swapped out, is swapped back in but still also is in the swapfile. In MBytes
1373-
}
1374-
\item \declareAttributeProvisional{PMIX_NODE_MEM_SWAP_TOTAL}{"pmix.node.mswpt"}{float}{
1375-
Total amount of swap space available. In MBytes
1376-
}
1377-
\item \declareAttributeProvisional{PMIX_NODE_MEM_SWAP_FREE}{"pmix.node.mswpfree"}{float}{
1378-
Memory which has been evicted from RAM, and is temporarily on the disk. In MBytes
1379-
}
1380-
\item \declareAttributeProvisional{PMIX_NODE_MEM_MAPPED}{"pmix.node.mmap"}{float}{
1381-
files which have been mmapped, such as libraries. Note that some kernel configurations might consider all pages part of a larger allocation (e.g., THP) as “mapped”, as soon as a single page is mapped. In MBytes
1382-
}
1383-
\item \refattr{PMIX_DISK_RESOURCE_USAGE} One for each disk attached to the node, if requested.
1384-
\item \refattr{PMIX_NETWORK_RESOURCE_USAGE} One for each network interface on the node, if requested.
1385-
\item \declareAttributeProvisional{PMIX_NODE_SAMPLE_TIME}{"pmix.node.samptime"}{time_t}{
1386-
Time when sample was taken
1387-
}
1306+
\item \refattr{PMIX_HOSTNAME}
1307+
\item \refattr{PMIX_NODEID}
1308+
\item \declareAttributeProvisional{PMIX_NODE_LOAD_AVG}{"pmix.node.la"}{float}{
1309+
Load average of last minute
1310+
}
1311+
\item \declareAttributeProvisional{PMIX_NODE_LOAD_AVG5}{"pmix.node.la5"}{float}{
1312+
Load average of last five minutes
1313+
}
1314+
\item \declareAttributeProvisional{PMIX_NODE_LOAD_AVG15}{"pmix.node.la15"}{float}{
1315+
Load average of last fifteen minutes
1316+
}
1317+
\item \declareAttributeProvisional{PMIX_NODE_MEM_TOTAL}{"pmix.node.mtot"}{float}{
1318+
Total usable RAM (i.e., physical RAM minus reserved bits and kernel binary code). In MBytes
1319+
}
1320+
\item \declareAttributeProvisional{PMIX_NODE_MEM_FREE}{"pmix.node.mfree"}{float}{
1321+
Total free RAM. In MBytes
1322+
}
1323+
\item \declareAttributeProvisional{PMIX_NODE_MEM_BUFFERS}{"pmix.node.mbuf"}{float}{
1324+
Temporary storage for raw disk blocks. In MBytes
1325+
}
1326+
\item \declareAttributeProvisional{PMIX_NODE_MEM_CACHED}{"pmix.node.mcache"}{float}{
1327+
In-memory cache for files read from the disk (the pagecache) as well as tmpfs and shmem. In MBytes.
1328+
Doesn't include \refattr{PMIX_NODE_MEM_SWAP_CACHED}.
1329+
}
1330+
\item \declareAttributeProvisional{PMIX_NODE_MEM_SWAP_CACHED}{"pmix.node.mswpc"}{float}{
1331+
Memory that once was swapped out, is swapped back in but still also is in the swapfile. In MBytes
1332+
}
1333+
\item \declareAttributeProvisional{PMIX_NODE_MEM_SWAP_TOTAL}{"pmix.node.mswpt"}{float}{
1334+
Total amount of swap space available. In MBytes
1335+
}
1336+
\item \declareAttributeProvisional{PMIX_NODE_MEM_SWAP_FREE}{"pmix.node.mswpfree"}{float}{
1337+
Memory which has been evicted from RAM, and is temporarily on the disk. In MBytes
1338+
}
1339+
\item \declareAttributeProvisional{PMIX_NODE_MEM_MAPPED}{"pmix.node.mmap"}{float}{
1340+
files which have been mmapped, such as libraries. Note that some kernel configurations might consider all pages part of a larger allocation (e.g., THP) as “mapped”, as soon as a single page is mapped. In MBytes
1341+
}
1342+
\item \refattr{PMIX_DISK_RESOURCE_USAGE} One for each disk attached to the node, if requested.
1343+
\item \refattr{PMIX_NETWORK_RESOURCE_USAGE} One for each network interface on the node, if requested.
1344+
\item \declareAttributeProvisional{PMIX_NODE_SAMPLE_TIME}{"pmix.node.samptime"}{time_t}{
1345+
Time when sample was taken
1346+
}
13881347
\end{itemize}
1389-
1348+
}
13901349

13911350
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
13921351
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

0 commit comments

Comments
 (0)