Added PjrtClient::UpdateGlobalProcessInfo method.
#28011
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![copybara-service[bot]](https://avatars.githubusercontent.com/in/44061?s=60&v=4){kind=small}
copybara-service
bot
force-pushed
the
test_773072434
branch
3 times, most recently
from
d278fa8 to
7746175
Compare
copybara-service
bot
force-pushed
the
test_773072434
branch
4 times, most recently
from
f5985a7 to
12d82b6
Compare
copybara-service
bot
force-pushed
the
test_773072434
branch
from
12d82b6 to
e6cd06a
Compare
## Overview
Recall that a [multi-controller JAX program][mcjax] involves multiple PjRt
clients running across multiple processes. These clients perform collective
operations, like AllReduce and AllGather, to execute a distributed program.
This commit adds an `UpdateGlobalProcessInfo` method that updates a client with
information about all processes. For example, if there are four processes, we
might call `UpdateGlobalProcessInfo` on process 0 with the information that
process 0, 1, and 2 are healthy but process 3 is dead.
## Motivation
I am currently working on making multi-controller JAX fault tolerant. Part of
this work involves cancelling collectives where one of the participants of the
collective has failed. The `UpdateGlobalProcessInfo` method will allow a PjRt
client to notice when a peer process has failed and abort any collectives it is
performing with this failed peer.
Previously, I was using the [coordination service][coordination_service] to
determine when processes failed, but PjRt clients executed via C plugins do not
have access to the coordination service.
## Future Work
This commit introduces the new `UpdateGlobalProcessInfo` method and pipes it
through the C++ sandwich, but it doesn't actually implement it yet. Nothing is
calling the new method either. These things will come in future changes.
## Alternatives
Rather than introducing a new `PjRtClient` method, I could have piped a
coordination service client through the C plugin API into `PjRtClient`s.
However, this would be very complicated. The [code to pipe a key-value store
client is complicated][kvs], and the API for the coordination service client is
significantly more complex.
I could shoehorn the new API into the existing key-value store. For example, I
could establish a convention that the state of every process i is stored in
some special key `process_{i}` in the key-value store. This felt roundabout.
[mcjax]: https://docs.jax.dev/en/latest/multi_process.html
[coordination_service]: https://github.com/openxla/xla/tree/main/xla/tsl/distributed_runtime/coordination
[kvs]: https://github.com/openxla/xla/blob/f5813dd522b9ef28eb58e638c45e60430b686d1a/xla/pjrt/c/pjrt_c_api.h#L324-L406
PiperOrigin-RevId: 783799013
copybara-service
bot
force-pushed
the
test_773072434
branch
from
e6cd06a to
d4292b4
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Added
PjrtClient::UpdateGlobalProcessInfomethod.Overview
Recall that a multi-controller JAX program involves multiple PjRt
clients running across multiple processes. These clients perform collective
operations, like AllReduce and AllGather, to execute a distributed program.
This commit adds an
UpdateGlobalProcessInfomethod that updates a client withinformation about all processes. For example, if there are four processes, we
might call
UpdateGlobalProcessInfoon process 0 with the information thatprocess 0, 1, and 2 are healthy but process 3 is dead.
Motivation
I am currently working on making multi-controller JAX fault tolerant. Part of
this work involves cancelling collectives where one of the participants of the
collective has failed. The
UpdateGlobalProcessInfomethod will allow a PjRtclient to notice when a peer process has failed and abort any collectives it is
performing with this failed peer.
Previously, I was using the coordination service to
determine when processes failed, but PjRt clients executed via C plugins do not
have access to the coordination service.
Future Work
This commit introduces the new
UpdateGlobalProcessInfomethod and pipes itthrough the C++ sandwich, but it doesn't actually implement it yet. Nothing is
calling the new method either. These things will come in future changes.
Alternatives
Rather than introducing a new
PjRtClientmethod, I could have piped acoordination service client through the C plugin API into
PjRtClients.However, this would be very complicated. The code to pipe a key-value store
client is complicated, and the API for the coordination service client is
significantly more complex.
I could shoehorn the new API into the existing key-value store. For example, I
could establish a convention that the state of every process i is stored in
some special key
process_{i}in the key-value store. This felt roundabout.