Support CAGRA in FAISS C API

[QingyangYinUber]

Description

This pull request introduces support for the CAGRA index in the FAISS C API. This enables downstream projects that use the C API, such as Lucene, to leverage GPU-accelerated indexing with CAGRA. The key changes include:

• New C interface for CAGRA: Exposes CAGRA index functionality through the C API.

• CPU/GPU Index Conversion: Adds C interfaces to convert a GPU-based CAGRA index to a CPU-based HNSW index and vice-versa. This is necessary because CAGRA indexes cannot be directly saved to disk. The current workaround is to convert the index to HNSW for saving and convert it back to CAGRA upon loading.

Motivation

The primary motivation is to unblock the use of CAGRA in Lucene. A recently merged PR in Lucene (apache/lucene#14178) introduced a FAISS codec for k-NN search using the Panama FFI. However, it cannot use CAGRA because the FAISS C API lacks the necessary support, which this PR now provides.

How to Test

The new functionality can be tested using the newly added C++ example: tests/test_gpu_cagra_c_api.cpp. To run the project, first create a Conda environment and install the required packages. Then, build the project and execute the tests as shown below.

conda create -n faiss -y
conda activate faiss
conda install -c conda-forge -c rapidsai -c nvidia openblas cuda-compiler libcuvs=24.12 libnvjitlink 'cuda-version>=12.0,<=12.5' 'gxx_linux-64<12' gflags -y

git clone https://github.com/QingyangYinUber/faiss.git
cd faiss
git checkout cagra-c
cmake -B build -S . \
  -DFAISS_ENABLE_GPU=ON \
  -DFAISS_ENABLE_PYTHON=OFF \
  -DFAISS_ENABLE_C_API=ON \
  -DFAISS_ENABLE_CUVS=ON \
  -DBUILD_TESTING=ON \
  -DFAISS_ENABLE_EXTRAS=ON \
  -DBUILD_SHARED_LIBS=ON \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_C_STANDARD=17 \
  -DCMAKE_CXX_STANDARD=17 \
  -DCMAKE_CUDA_ARCHITECTURES=80

make -C build -j$(nproc)

build/tests/faiss_test --gtest_filter="*GpuCagraCAPITest*"

[meta-cla]

Hi @QingyangYinUber!

Thank you for your pull request and welcome to our community.

Action Required

In order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at [email protected]. Thanks!

[Enet4]

What would be the difference between faiss_index_gpu_to_cpu_new and faiss_index_gpu_to_cpu? If these function are to ever have a different behavior, then that should probably be documented.

[mnorris11]

Hi @QingyangYinUber , seems like the build has errors, are you able to see the logs and update?

[QingyangYinUber]

Hi @mnorris11 , I have updated the code to follow the clang-format. For the build error, I modified the "How to Test" section, adding the conda environment and the needed packages to build.

[mnorris11]

If it is nullptr, this just casts null to null right? Is there a problem with it on your consumer side?

[mnorris11]

I don't quite understand why we need the _new if it just wraps this function?

[mnorris11]

This is the cmake run by the first initial job:

  cmake -B build \
        -DBUILD_TESTING=ON \
        -DBUILD_SHARED_LIBS=ON \
        -DFAISS_ENABLE_GPU=OFF \
        -DFAISS_ENABLE_CUVS=OFF \
        -DFAISS_ENABLE_ROCM=OFF \
        -DFAISS_OPT_LEVEL=generic \
        -DFAISS_ENABLE_C_API=ON \
        -DPYTHON_EXECUTABLE=$CONDA/bin/python \
        -DCMAKE_BUILD_TYPE=Release \
        -DBLA_VENDOR=Intel10_64_dyn \
        -DCMAKE_CUDA_FLAGS="-gencode arch=compute_75,code=sm_75" \
        .
  make -k -C build -j$(nproc)

It fails because it tries to build the c api but there is no cublas as FAISS_ENABLE_GPU is OFF

Can tests/test_gpu_cagra_c_api.cpp instead be under the gpu/tests directory? You can try it and see if it builds