Integration of Intel Scalable Vector Search (SVS) with FAISS #4450
Description
This issue aims to track the progress and facilitate discussions regarding the integration of the Scalable Vector Search (SVS) repository library with FAISS.
The integration will enhance FAISS's capabilities by incorporating SVS indices and functionalities.
Below is a preliminary list of tasks that need to be addressed for successful integration:
- Provide thin wrapper around SVS
- Map
train(),add(),search()to corresponding SVS functions. - Update CMake configurations to pull in the prebuilt SVS library.
- Ensure seamless integration with both C++ and Python interfaces.
- UPDATE: Completed.
- Map
- Enable SVS I/O and Align with FAISS Output
- Read/write via
read_index()/write_index(). - UPDATE: Completed.
- Read/write via
- Threading: FAISS OpenMP vs SVS Native Solution
- Evaluate threading models: compare FAISS's OpenMP implementation with SVS's native threading solution.
- Determine the optimal threading approach for compatibility.
- UPDATE: Completed. SVS now accepts a FAISS-owned OpenMP threadpool.
- Template Instantiations with SWIG
- Implement template instantiations using SWIG to facilitate seamless integration and usage of SVS within FAISS.
- UPDATE: Currently not required.
- Add Dynamic Support for SVS Flat Index
- Integrate dynamic support for SVS Flat index within FAISS.
- Investigate and compare the performance of SVS implementation against FAISS's flat index.
- UPDATE: Pending a change on SVS side. Fixed-size flat indices added. (Second call to
add()will currently throw.)
- Building via factory_string, cloning via clone_index
- Convenience functions. Will be added once core functionality is there.
- UPDATE: Completed.
Discussion Points:
- General feedback and suggestions for integration.
- Requests for additional features or improvements.
- Collaboration opportunities and resource sharing.
Please feel free to contribute to the discussion, provide insights, or raise any concerns regarding the integration process.