Mapping gets out of sync when deleting an ID from IVF + PQ with an IndexIDMap2 wrapper

[huutuanuet]

Hello,
In C++17, with FAISS 1.11.0, environment Raspberry Pi4, I have the following schema:

1. Step to reproduce

• Step 1: I add 2 times for a data set including 10 vectors, now the ID will be ID[1...20]
• Step 2: After searching with data as embedding of ID 1, the result will have ID 1 and ID 11.
• Step 3: I will use remove_ids to remove ID 1.
• Step 4: I will search again with data as embedding of ID 1, the result will be ID 2 and ID 12 with the same score as step 2.

You can see the pictures to visualize what I mean better. I am new to this field so the terminology used may not be accurate.
-> I am suspecting that after deleting ID 1, the IDs behind have been pushed up, leading to this mapping error.
The removal operation shifts the ids of vectors above the removed vector id.

2. Config

	"dimension": 768,
	"metric": INNER_PRODUCT,
	"ivf": {
	"nlist": 2048,
	"nprobe": 32
	},
	"pq": {
	"m": 96,
	"nbits": 8
	},

3. Source code

a. Create Index

auto quantizer = std::make_unique\<faiss::IndexFlat\>(config.dimension, metric);
auto ivf_index =
		std::make_unique<faiss::IndexIVFPQ>(quantizer.release(), config.dimension, ncentroids, m, nbits, metric);
ivf_index->nprobe = nprobe;
auto idmap_index = std::make_unique<faiss::IndexIDMap2>(ivf_index.release());

b. Add_with_ids

std::vector<float> processed_vector = embedding;
if (config.metric == MetricType::COSINE) {
	  processed_vector = NormalizeVector(embedding);
}
index->add_with_ids(1, processed_vector.data(), &custom_id);

c. Remove_ids

// Input parameter: 
const std::vector<int64_t>& faiss_ids
// Create ID selector for the IDs to remove
faiss::IDSelectorBatch selector(faiss_ids.size(), faiss_ids.data());
// Remove vectors using remove_ids
size_t removed = index->remove_ids(selector);

This is the test result:

[huutuanuet]

We have now updated to version 1.12.0 but the error still occurs.

[huutuanuet]

Since IndexIVFFlat and IndexIVFPQ already have add_with_ids() and remove_ids, there is no need to wrap with IndexIDMap2.
After unwarping with IndexIDMap2, the error no longer occurs.