Adding check to ANN MG tree merge mode #934
Conversation
cjnolet
added
bug
| @@ -605,6 +605,9 @@ void search(const raft::resources& clique, | |||
| n_neighbors, | |||
| n_batches); | |||
| } else if (merge_mode == TREE_MERGE) { | |||
| ASSERT(index.num_ranks_ % 2 == 0, | |||
There was a problem hiding this comment.
We discussed this briefly in our meeting, but I think we should find a better alternative here in the short term. I don't mind merging this change so that users at least have a warning / reason instead of segfaulting, but I don't think having an even number of GPUs is a fair requirement to levvy on our users. At the very least, we should be able to pair off the reductions using P2P, where there may be 1 GPU that only contributes in the final merge stage.
There was a problem hiding this comment.
Can you create an issue to follow-up with this (and link the issue in a comment in the code here so that any developers or users browsing the code know the plan)?
There was a problem hiding this comment.
Sure, created an issue over here : #979. Just to be clear, SNMG ANN sharded indexes can still be searched with an odd GPU count with the MERGE_ON_ROOT_RANK mode.
|
/merge |
Answers #904