Support for parallel Faiss CAGRA index builds on single GPU

[rchitale7]

Hi,

I am trying to build two GPU indexes of the same size in two parallel processes, using GpuIndexCagra and the python API for faiss. I've noticed that this takes roughly 2x the time to build a single GPU index in one process. But, if i set the GpuIndexCagraConfig.device setting to different values for each of the scripts, then I get roughly the same performance as building a single GPU index. However, I believe this device setting can only be changed on multi-GPU hardware. For hardware with a single GPU, is it expected behavior to see ~2x slowdown when building two GPU indexes in parallel? I would like to utilize the memory of my single GPU instance as much as possible, so curious if there's a way to get a performance boost when building multiple GPU indexes at the same time.

I am creating a 1,000,000 x 768 numpy array for this test. The script I used is here: https://gist.github.com/rchitale7/8e0995e8231eec5657f42627d2cc1228

My Setup

I used an EC2 g6.12xlarge machine (for multi GPU test) and g5.2xlarge (for single GPU test), with the Deep Learning Base OSS Nvidia Driver GPU AMI (Amazon Linux 2023) 20250912 AMI: https://docs.aws.amazon.com/dlami/latest/devguide/aws-deep-learning-base-gpu-ami-amazon-linux-2023.html

output of nvidia-smi:

Tue Sep 23 19:00:56 2025       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.172.08             Driver Version: 570.172.08     CUDA Version: 12.8     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA A10G                    On  |   00000000:00:1E.0 Off |                    0 |
|  0%   23C    P8             15W /  300W |       0MiB /  23028MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

Reproduction instructions

1. On a server with GPUs and conda installed, setup the following conda environment:

conda create -n faiss_test_new -c conda-forge -c pytorch -c nvidia -c rapidsai python=3.12 faiss-gpu-cuvs=1.12.0

2. Activate conda env:

conda activate faiss_test_new

3. Download the test script: https://gist.github.com/rchitale7/8e0995e8231eec5657f42627d2cc1228

4. Run the test script, supplying a device id (can only use 0 if there is one GPU on the instance):

python faiss_cagra_test.py 0

5. In a separate terminal session, activate the conda env and launch the same test script.

conda activate faiss_test_new && python faiss_cagra_test.py 0

6. Once both test scripts reach the pdb break point, at the same time, press c to continue the execution and run the scripts in parallel

7. After both scripts have completed, compare the results to running only one script at a time.

[trang-nm-nguyen]

@tarang-jain I wonder if you know there is a way to get a performance boost when building multiple GPU indexes at the same time

[cjnolet]

OpenSearch team has provided the following Dockerfile which they are using: https://github.com/opensearch-project/remote-vector-index-builder/blob/main/base_image/build_scripts/Dockerfile#L44

Prior version tested was 24.12, so it's possible a lot has changed in this impl in the past 8 months.

[rchitale7]

@cjnolet I performed this test with faiss-gpu-cuvs=1.10.0, and I still see the same behavior: ~2x slowdown when the indexes are built in parallel.

Confirmed that faiss-gpu-cuvs=1.10.0 uses libcuvs=24.12:

conda list | grep libcuvs

Output:

libcuvs                        24.12.00         cuda12_241211_g0ce6a03_0      rapidsai

[cjnolet]

@navneet1v any thoughts here? You mentioned this issue was not present on that version. Are you doing something differently now than you had been prior?

[navneet1v]

@navneet1v any thoughts here? You mentioned this issue was not present on that version. Are you doing something differently now than you had been prior?

We are not doing anything different earlier, I can dig through my runs 1 more time to see if I can find something like what version of cuvs I was using.

The things which are different for sure is cuda versions, cuvs vs raft. We can try running these in the older versions of cuda to see if 2x regression is there.

[navneet1v]

@cjnolet one thing I remember was back in early 2024, instead of CUVS, the Cagra code was coming from raft. And we have seen issues in memory spikes in cuvs but not in rapids. ref: #4274 .

[tarang-jain]

I was able to reproduce this. Investigating further!

[tarang-jain]

Even with a single index build, the sm% goes up to as much as 99%.
This can be tracked with:
nvidia-smi dmon -s pucm
What this means is that trying to build two parallel indexes with different streams is possibly not helping. The streams are contending for the same resources (threads), which is in fact slowing them down. This should explain the difference. In a few places, there is maximization of resource utilization if things can fit in device memory. For example, the CAGRA pruning kernel here is instantiating several small batch kernels

[cjnolet]

@navneet1v can you and @rchitale7 please try to reproduce your parallel index build experiment from the prior version of raft/cuvs/faiss? I'm really beginning to wonder if there were a specific set of parameters, specifically with low n_probes, which may have been displaying this behavior.

Full disclosure- each probe during IVFPQ search is scheduled on a different CUDA block to maximize the paralleism of the search. If you set low n_probes, it'll leave more room to schedule more CAGRA index builds. Large number of probes will saturate more of the GPU and leave less room for overlap.

It's worth pointing out that at some point we set heuristic defaults of the IVFPQ graph build params in cuVS, but those were just defaults. Are you guys using the defaults or are you setting the ivfpq params explicitly?

[navneet1v]

It's worth pointing out that at some point we set heuristic defaults of the IVFPQ graph build params in cuVS, but those were just defaults. Are you guys using the defaults or are you setting the ivfpq params explicitly?

I think we were setting the values explicitly. We are actually doing more test on our side to see what is happening. Will post some update on this once we have a concrete deep-dive

[mnorris11]

(assigned to @navneet1v for tracking purposes)