prov/ucx: Fix multi-threading support #11665
Merged
+25
−4
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The threading mode was set to FI_THREAD_SAFE, but the UCX threading mode was
set to UCS_THREAD_MODE_SINGLE which only allowed a single thread to access
UCX objects. Trying to access an EP (which hosts the UCX worker) from multiple
threads may cause the following error:
`ucp_ep.c:1808 Assertion 'ucs_async_check_owner_thread(&(worker)->async)' failed`
Changes made:
(1) Allow threading mode be set based on application hints.
(2) Use proper UCX threading mode based on OFI threading mode.
(3) Add a runtime parameter to force single thread mode. This is for the
purpose of evaluating the performance impact of supporting multiple
threads since the default setting has changed.
Signed-off-by: Jianxin Xiong <[email protected]>
aingerson
approved these changes
Nov 25, 2025
Contributor
Author
|
Intel CI failure is a known issue that is unrelated (tcp |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The threading mode was set to
FI_THREAD_SAFE, but the UCX threading mode was set toUCS_THREAD_MODE_SINGLEwhich only allowed a single thread to access UCX objects. Trying to access an EP (which hosts the UCX worker) from multiple threads may cause the following error:ucp_ep.c:1808 Assertion 'ucs_async_check_owner_thread(&(worker)->async)' failedChanges made:
(1) Allow threading mode be set based on application hints.
(2) Use proper UCX threading mode based on OFI threading mode.
(3) Add a runtime parameter to force single thread mode. This is for the
purpose of evaluating the performance impact of supporting multiple
threads since the default setting has changed.
Fix #11651