Skip to content

ASC Q2 2020 Meeting

Jump to bottom
Kathryn Mohror edited this page Sep 3, 2020 · 19 revisions

PMIx Standard Administrative Steering Committee (ASC) Q2 2020 Meeting

Quick Links

Agenda (timeline in chart below)

  • Discuss 2020 quarterly meetings
    • Reminder of quarterly meeting dates
    • Discussion of Q4 face-to-face meeting logistics
  • PMIx 4.0 release update
  • Call for new ASC Members
  • Vote on new ASC members
    • LANL
  • Roll call for attendance/vote
    • Review voting eligibility
  • Governance PRs up for a vote:
  • PMIx Standard PRs up for a vote:
  • PMIx Standard PRs up for a Reading
  • Plenary discussion items
  • Working Group Presentations
    • Client Separation / Implementation Agnostic Document Working Group
    • Slicing/Grouping of functionality Working Group
    • Dynamic Workflows Working Group
    • Storage Working Group
  • Technical presentations
  • Use case presentations
    • (None at this time)
  • Open call for new Working Groups
  • Discussion items:
    • (None at this time)

Agenda Timeline

We will try to keep to this timeline as best as we can. However, discussion items may take longer/shorter than anticipated and as a result, the agenda may need to be adjusted during the meeting.

We will start roll call promptly at 11:25 am.

All times in US Central. (Last Update: March 19, 2020)

Start End Topic
11:00 am 11:05 Gathering
11:05 am 11:15 Discuss Quarterly Meetings
11:15 am 11:25 PMIx 4.0 Release Update
11:25 am 11:35 Roll Call, Vote on New ASC Members, Call for New ASC Members
- Roll call
- Vote on New ASC Member: LANL
- Call for New ASC Members
11:35 am 11:45 Governance PR Votes
- Reading and First vote: https://github.com/pmix/governance/pull/11
- Second vote: https://github.com/pmix/governance/pull/10
11:45 am 12:00 Standard PR Votes
- First vote: https://github.com/pmix/pmix-standard/pull/192
12:00 pm 12:45 Standard PR Readings
- https://github.com/pmix/pmix-standard/pull/235
12:45 pm 1:15 Break
1:15 pm 1:45 Plenary: Chapter 3 Error Codes
1:45 pm 2:30 Plenary: Use Cases
- https://drive.google.com/drive/folders/1eN7aBxyzPD0a_GJFq1KH2ZHpoONj76op
2:30 pm 3:30 Working Group Updates
- Client Separation / Implementation Agnostic Document Working Group
- Slicing/Grouping of functionality Working Group
- Dynamic Workflows Working Group
- Storage Working Group
3:30 pm 3:35 Open Call for New Working Groups
3:35 pm 3:45 Break
3:45 pm 4:30 Technical Presentations
- Artem Polyakov (Mellanox) "The Optimizations Of PMIx Inter-Node Exchanges"
- Josh Hursey (IBM) "Modex Exchange Modes: PMIx_Commit/Fence/Get Semantics" (Postponed to Q3 Meeting)
4:30 pm 5:00 Additional Discussion Items

Attendees

    Joshua Hursey (IBM)
	Brice Goglin (Inria)
	Aurelien Bouteiller (UTK)
	Kathryn Mohror (LLNL)
	David Solt (IBM)
	Jai Dayal (Intel)
	Michael Karo (Altair)
	Ralph Castain (Intel)
	Shane Snyder (ANL)
	Stephen Herbein (LLNL)
	Ken Raffenetti (ANL)
	Artem Polyakov (Mellanox)
	Swaroop Pophale (ORNL)
	Thomas Naughton (ORNL)
	Karthik Manian (OSU)
	Howard Pritchard (LANL) 
	Mahdieh (OSU)
    Martin Schulz (TUM)

Notes

  • Discuss 2020 quarterly meetings
    • Reminder of quarterly meeting dates
      • July 22 (Virtual)
      • Oct 1 (face-to-face)
    • Discussion of Q4 face-to-face meeting logistics
      • Still planning on meeting face-to-face currently
      • Travel restrictions may affect this
      • Will decide closer to July meeting if this becomes virtual
      • No verbal comments from attendees expressing problems with a later decision
  • PMIx 4.0 release update
    • Some delays on release
    • Contracts were written that guarantee functionality in V4
      • Don’t have software from vendor to verify this is complete
    • Kathryn: did the contract stipulate 4.0 specifically?
      • Ralph: There isn’t a version number explicitly. Main concern is that a new API is required and changes to it would be required without verifying it is correct
    • Josh: Already voting on a textual change to v5, should consider that v5 might be ready before v4
      • Ralph: changes to chapter 1 & 2 would not conflict with any changes from v4. Expect that there is still several months before conflicts would arise.
      • Josh: if we vote yes on chapter 1 changes today, then the V4 and V5 release managers should meet and discuss how we should handle things. Do we branch?
    • Josh: are you also planning on adding textual changes to tools?
      • Ralph: yes, adding a new chapter
    • Kathryn: how does the new tools chapter connect with the slices WG?
      • Ralph: not sure. Debating whether the chapter should actually be a whitepaper or a chapter. It doesn’t define any new attributes or APIs, just how to use them.
      • Kathryn: if we can get that chapter merged into v4, then maybe that helps the slices WG and sets a precedent
  • Call for new ASC Members
  • Vote on new ASC members
    • LANL
    • Passed: 10 yes/3 not present
  • Roll call for attendance/vote
    • Review voting eligibility
  • Governance PRs up for a vote:
    • Reading and First Vote
      • pmix/governance/pull/11
        • Adjust ASC quarterly meeting agenda finalization
      • Ken: one nit: 2 weeks + 2 weeks != 1 month
      • Verbal consent from those on the call to make a quick hotfix of Ken’s nit rather than delay another quarter
      • Note: Ralph really wants to vote no to spice things up :)
    • Reading and Second Vote
      • pmix/governance/pull/10
        • A fast track for administrative and minor typo changes
      • No comments from those on the call
  • PMIx Standard PRs up for a vote:
    • First Vote
      • pmix/pmix-standard/pull/192
    • Link to the slides and PDF that Dave presented: https://drive.google.com/open?id=1gp_1fH6Wl8TAzPGwXqGUhVoAr0KmWJpx
    • Aurelien: comment about how the PMIx server APIs aren’t required and an SMS may choose to completely implement/embed the PMIx server interfaces directly
    • Aurelien: PRI acronym is used but not first defined
      • Dave: that section has been removed so that PRI is no longer mentioned (it’s in the negative part of the diff on Github)
    • Aurelien: neither of these comments should delay the current PR (will still vote as yes as is). They can be addressed in subsequent changes.
  • PMIx Standard PRs up for a Reading
    • Reading of Chapter 2 changes from Implementation Agnostic Working Group
      • pmix/pmix-standard/pull/235
    • “Numerical location” is an odd phrase, recommended changing to “ordinal”
    • Aurelien: Is the term Session not italicized for a specific reason (in the workflow definition)?
      • Dave: The diff macro removes formatting, so we need to confirm that it is italicized in the regular text (without a diff)
    • Martin: in Section 2.3, changle “should” to “shall” in the last sentence: “Users should not use the …”
    • Josh: note that this first PR just changes existing verbiage/definitions without adding any. Second PR will introduce new terminology
  • Plenary discussion items
    • Implementation Agnostic Document Working Group
      • Link to the slides and PDF that Dave presented: https://drive.google.com/open?id=1gp_1fH6Wl8TAzPGwXqGUhVoAr0KmWJpx
      • Overview of issues with error codes in standard
      • Stems from work on client separation / API working group efforts (see also slides)
      • Breakdown of different error types by “categories”
      • Concerns: 28 of 108 referenced elsewhere in standard, but in some cases description sometimes very terse. Category separations are not always clear/clean, e.g., PMIX_JCTRL_CHECKPOINT, PMIX_ERR_JOB_CHKPT??
      • Concerns: Some error codes are more implementation detail, and never returned to user so may not be needed in standard itself. Found many occur in OpenPMIX but not actually returned at User level (i.e., not necessarily part of PMIX standard)
      • Question: What extend should PMIx standardize error codes? Some foundational to PMIx, others specific to APIs or very generic.
      • Concern: client developer likely to want to check and handle error codes for each API, or which error codes his app should consider fatal
      • Concern: PMIx developers may lack guidance on return codes that are necessary for an API, e.g., when making new implementation.
      • Question: Is this worth investigation/effort given scope of work to do?
      • Possibilities: use error classes, reduce documented error codes, or create list of general error codes that many APIs might return and then create API specific set of codes (could require a lot of effort for each API)
      • Dave asking for comments/feedback
      • Kathyrn: Important to help users/developers reason about things. Not about timing, addressing sooner rather than later can help users, and developers.
      • Dave - challenge of variety of systems used (backend) with PMIx make this very challenging
      • Kathryn: helpful to delineate concerns, is this my app’s problem, a tool, system, etc.
      • Ralph - always going to be difficult to reason about error codes b/c the errors are chained. If PMIx finds a problem, it can reason about how to find things like Bad Param. But wiill be difficult to have every error covered.
      • Kathryn: nice to have some error types/classes, and things may fall outside those general cases
      • Ralph - error codes are hard b/c you do not have complete control, interactions at different levels and no way to dictate to all what they should return
      • Dave - you could impose that if using the server PMIx backend, to enforce some consistency. But not help if not using PMIX server backend.
      • Ralph - can be challenging to server implementors, have a translation table to ensure some consistency
      • Dave - challenges with user-defined error codes to have conformity
      • Ralph - need users to be able to define their own status codes. Could say outside of a user-defined code, then if using standard call would need to select from specific system defined codes.
      • Dave - clarifying things here - “user defined” is case where a caller is passing a status code via PMIx event handlers so the user code can pass their own error code
      • Ralph - could be anybody User/Apps, RM, etc. that are returning these status codes via handlers. The intent of defining the range of “user-defined” was to avoid conflicts in error codes.
      • Warning to have explosion in doc size if have to list all error codes for all calls. May be good to say a general set could be returned and only clarify when they are specific.
      • Martin - MPI does error-code to error-class mapping, which puts burden on user to lookup things
      • Aurelien: Need to take care to avoid having User defined codes conflict. In MPI declare error-code and later user can convert to error-class as needed.
      • Ralph - seems challenging to avoid conflicts
      • Aurelien: yes, MPI all error-codes are local and not have to propogate elsewhere (global)
      • Ralph - seems like a challenging problem. The current error codes are used and helpful. If going to some global error constants, things get much more complicated and difficult and possibly not something anyone needs.
      • Martin - how big is this User-defined error space?
      • Ralph - all positive. Only reserve a small space, so should be plenty of space for the user-defined space.
      • Martin - could define a space/prefix to reserve space. To avoid having global operations.
      • Aurelien: could also use strings
      • Ralph - do you want to use strings?
      • Aurelien: could carry strings in msg info? (TJN: Not capture exactly this clarification)
      • Kathryn: Try to get some concensus for time/agenda
      • Dave - yes, sounds like we have feedback and will continue to look at in the working group
      • Dave - user-defined space is based on a base that is not in standard, so possibly something could refine
    • Slicing/Grouping Working Group Use case presentations
      • drive.google.com/drive/folders/1eN7aBxyzPD0a_GJFq1KH2ZHpoONj76op
      • Updates on use cases: Hybrid Programming Models and IO-Forwarding
      • Hybrid Programming Models
        • https://github.com/pmix/pmix-standard/issues/232
        • Identify what the active programming models are
        • PMIX provides reliable method for detecting the programming models at play
        • Allows for sharing of information among active programming models
        • Example showing how you can use this (see ticket)
        • Programming models can use event notification during their initialization to gather these details, also can be used in dynamic manner via event handlers (example: shows event handler for what programming model was declared)
        • Also includes specific use case for coordination with example of openmp for master/worker thread within a single programming model (multiple event handler registrations), with ordering attributes for when handler runs/notified
        • Jai: question have you thought of adding GPU attributes?
        • Steven: no, these attributes were taken directly from RFC. Not in standard yet. Set to land in v4 of spec.
        • Steven: Not sure this WG is best place for adding new attributed, more focused on documenting existing
        • Jai: Have some slides that will address GPUs later.
      • IO Forwarding for Tools/Debuggers use-case
      • Steven - have few other WG use-case tickets as drafts on github. Starting to look at having these use cases as chapters in the standard. Trying to find a way to avoid lots of copy/paste.
  • Working Group Presentations
    • Client Separation / Implementation Agnostic Document Working Group
      • Dave - lots of presentations today, so covered lots already. Have some additional work on Ch2 terms that need to be addressed. Working through Ch3 now, somewhat side tracked with error codes. Also considering the notion of “required attributes”, what that means, does it have to recognize attribute, etc. There was a past presentation on that. (slide URL TODO).
      • Will take feedback from today’s reading and continue working on Ch3, have some pending Ch4 bits still todo.
      • No questions
    • Slicing/Grouping of functionality Working Group
      • Steven - covered most in plenary, working on use-cases and hoping to have ways to integrate into standard. At Q3 Josh planning to present update on the sparse modex.
      • See Google drive for all use cases or add comments on tickets
        • drive.google.com/drive/folders/1eN7aBxyzPD0a_GJFq1KH2ZHpoONj76op
    • Dynamic Workflows Working Group
      • Jai: Initially looking at some implementation drivers. There was some effort to use Spark (Intel) + PMIx, but that’s be delayed. Will be picking up new customer with mOS (Intel) for using PMIx. Maybe mOS can leverage PMIx for launch strings/attributes (pinning, etc.) Also, Josh working on Kubernetes (IBM) + PMIx
      • Link to Slides
      • Looking at various science workflows and runtimes/workflow managers, e.g., Radical Pilot, Swift-T, Spark, Kubernetes
      • New functionality to add to standard:
        • GPUs - many users hardcode GPU selection, but would be good to have GPUs in the spec would help workflow managers to avoid having to resort to hardcodes. Common GPU affinity options: Auto, Specific-Devices, NUMA/Socket, etc.
        • Preemption - UrgentHPC type jobs could benefit from these capabilities
      • PMIx GPU Attributes
        • Programming Model Attributes, General Process Level attributes,Spawn Attributes, etc.
      • Job Allocation directives -- preemption attributes would be useful, working on this but lacking immediate driver
      • Upcoming topics - task workflows Robert Clay - Sandia 5/15
      • ADIOS workflows (ORNL)
      • Swift-T using spawn and Radical pilot also used prrte
      • Martin - other accelerators, or just GPUs
      • Jai: open to other examples for considerations
      • Martin - is there really a difference, between accelerator vs. gpu, fpga?
      • Possibly consider naming change to cover accelerator (general) that could cover GPU, FPGA, etc.
      • Jai: Would be open to more info
      • Ralph - might want to add another attribute to know what type of accelerator you have in this more general form, e.g., is that a GPU or FPGA, etc.
    • Storage Working Group
      • Shane: Brief overview - extend PMIX to support interaction of apps, tools, and RMs with storage hierarchies
      • Link to Slides
      • Decided to focus effort on integrating the following storage-related functionality
        • A. Storage system query (“What storage systems are available?”)
        • B. Storage system event notification
        • C. Storage system primitives (system specific primitives to move data/reservations/etc. staging/unstaging)
      • Starting with A. and B. for less invasive extensions, using query & event notification capabilities already in PMIx. Later look into C.
      • Focus on pmix_query_info (see slides for details on initial new keys of interest, etc.)
      • Current status - Ralph already prototyped most query extensions in OpenPMIx, so can begin evaluating our design using Lustre, DAOS, etc.
      • Next Steps - take current query pieces and move forward to event notifications to flesh out more comprehensive list of event/query points
      • Plan to bring text forward at future meeting for reading on query interface, etc.
      • Bi-weekly meetings on Tues @ 4-5pm CST
      • Kathryn: Like that you are using Lustre and DAOS to keep paths and object-storage cases to keep design generic
  • Open call for new Working Groups
    • No verbal comments
    • Josh - some mentioned the idea of container use cases or possibly debuggers. So if those are interesting to folks, possibly some could get together there
    • Also, the Hybrid programming models could be of interest
    • Jai - also some mentioned an energy-efficient WG might be of interest.
    • Josh - Possibly something from Power API?
    • Ralph - They invited me to talk about this. Were thinking of using PMIx to “set” power settings, and use Power API as way to perform that.
    • Jai - that was from GOPM group
    • Martin - groups interested to possibly leverage PMIx and other pieces for power-manage, GOPM. Possibly some European projects
    • Ralph - at time of previous discussion, not a lot of different efforts. Possibly a good time to have a working group on this to avoid having all the Resource Manger companies get lots of requests. Instead could use PMIx as way to get something that would work for the different RMs.
    • Martin - stephen, there are some (names?) are interested and could possibly present at use-case working group
    • Ralph - Jai, possibly have GOPM people get in touch with Steven. Do not want to throw all on Steven, but possibly have him get something started/meeting. Ryan Grant leads Sandia effort.
    • Martin - Syd Gana from GOPM you familiar?
    • Jai - Yes, familiar, he raised point to me. :-)
  • Technical presentations
    • Optimizations of PMIx Inter-Node Exchanges by Artem
    • Summary: optimization of out-of-band exchange of keys by Slurm
      • Initially optimized message sizes (12 kb per rank -> 1 kb)
      • Message sizes may vary by a few bytes (across ranks and jobs)
      • Investigated the Bruck algorithm for exchange (inspired by MPI Allgatherv)
      • No global knowledge of block sizes prevents direct application of the Bruck algorithm
      • After applying aggregated message unpack, bruck performs logarithmically compared to linear performance of ring and tree
    • Ken: is the buffer for just a single key? Is this a per-key optimization or for all of the data?
      • Artem: some of the optimizations are per-key, but the exchange is not necessarily per-key (summary may not be exactly correct)
    • Josh: Is this implementation available in SLURM
      • Artem: Not currently, have to get into production (currently in research stage)
      • We were able to provide optimal Bruck algorithm w/o receive-count knowledge
  • Discussion items:
    • (None at this time)
    • Next monthly meeting is May 8, 2020

Home

Working Groups

Events

Helpful How To...

ASC Quarterly Meetings

ASC Monthly Meetings

Clone this wiki locally