Feat/nccl #766
Draft
Feat/nccl #766
+265
−44
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
No Future here, that would be a overcomplicated abstraction ^^ |
| devices | ||
| } | ||
|
|
||
| pub fn linked_groups(split: Vec<f64>) -> Vec<Self> { |
There was a problem hiding this comment.
What is a split? Maybe add some docs in each public functions to help users
There was a problem hiding this comment.
In this Context you get a f64 for each device group you want do build. Each group gets devices assigned depending on the proportion of each f64 compared to the sum of Vec...
Just asking early on that frequently, because i want to get a felling if getting deeper in the code gets backslash, sorry for not pointing that out.
Since you did not comment the use of using CudaRuntime::Info i assume that using existing code is allowed as long as its behind the feature flag ...
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
So started with a NcclDevice behind a nccl feature flag. With some CudaDevice implementations which come with the feature to make it easier to set up groups.
CudaDevice is needed to be passed trough client so it comes trough .info() now with the feature on.
Next comes NcclOp with NcclOpBuilder. NcclOpBuilder will first translate the input Handel(s) to get ptr und size. Then it can be executed, which will build a
cudarc::nccl::sys::ncclComm_t, then starts the nccl group executes Futures on pointers step by step to then stop the nccl group. At least thats the plan for now.