Hard to achieve high recall rate(recall@R) for huge workload

[EvagelineFEI]

Hello team,
I am using faiss.IndexHNSWFlat and my dataset is sift1b from http://corpus-texmex.irisa.fr/.
I add 10M data into the index and then I adjust the parameters(M, efc, efs)to improve the recall rate(recall@100).
I have tried many pairs,(M from 5 to 500) but my best result stuck at 0.93.
It doesn't seem that my precision calculation code has error. And it goes better for smaller dataset(like 1M,2M, where the precision can reach 0.96 or higher). What can be the problem?
Some of my machine info:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 46 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 80
On-line CPU(s) list: 0-79
Vendor ID: GenuineIntel
Model name: Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
CPU family: 6
Model: 79
Thread(s) per core: 2
Core(s) per socket: 20
Socket(s): 2
Stepping: 1
CPU max MHz: 3600.0000
CPU min MHz: 1200.0000
BogoMIPS: 4600.00

[satymish]

Hi @EvagelineFEI what's the data distribution like for the smaller datasets as well as the larger datasets? Is the data randomly sampled and provide uniform data distribution?

[EvagelineFEI]

Thank you @satymish, for the smaller datasets, these subsets are the n first vectors of the bigann_base.bvecs file (n=1M,2M,5M,10M,20M,50M,100M,200M,500M,1B). (as mentioned in the official readme http://corpus-texmex.irisa.fr).
And for the data distribution of the larger datasets, are you referring to whether the dataset has distribution characteristics such as clustering, dimension correlation, sparsity, and intrinsic dimensionality?

[mnorris11]

@EvagelineFEI how many query vectors are you using? All 10k from sift1b? And can you paste how you are computing recall@100?

[mnorris11]

sanity check: can you paste a minimal repro of the entire thing, like how to obtain self.true_neighbors? Was this from an IndexFlat search?

I also didn't quite get it from your response; you have 10M database vectors? And how many query vectors?

Do you still see the same recall with different numbers of database vectors?

[EvagelineFEI]

Hi @mnorris11.For the sanity check: I did not use IndexFlat search to obtain self.true_neighbors, cause the ground-truth data is provided in the dataset so I think I can just read from the files？(like idx_10M.ivecs)(as can be seen in the figure below)🧐

Sorry I made it vague last time😿. Yes, I have 10M database vectors, and for the query vectors I use all 10k from sift1b.

I get different recall rates. For smaller db vectors, like 1M, 2M, I can get higher like 0.95 and 0.96 by adjusting the parameters.

[mnorris11]

Do you get higher recall with another HNSW implementation?

1B is relatively larger so pushing higher than 0.9 recall may require other approaches.

Did you also try IVF based indexes?

[EvagelineFEI]

Hi @mnorris11, right, I think for huge workload, the search space gets complex. I try to divide the workload into pieces, build several indices and merge the query to compute recall. I find the recall gets higher, however, the query latency gets longer☹️

[mnorris11]

Sorry for the delay;

One thing to try is comparing Faiss HNSW vs some other implementation. If you see that other implementations are getting higher recall etc, we can open a bug issue to figure out why.

To get higher recall, you may just have to make the tradeoff for higher latency.

[onestardao]

This actually looks like a classic case of what we call the High-Recall Tradeoff Loop (see WFGY ProblemMap No.6). Once you push HNSW toward the 0.93–0.95 recall ceiling on large-scale datasets (like 10M vectors), you often enter a zone where:

• Latency increases exponentially with efSearch, but
• Recall only improves asymptotically
• And worse, different efSearch values yield inconsistent outcomes due to non-uniform graph topologies

We've encountered this in several benchmark cases. A few suggestions that helped us break through:

1. Vector Canonicalization: Preprocess the vector set via PCA whitening or L2 re-projection to reduce structural noise and improve distance separability.
2. Multi-pass reranking: Do a fast pass with HNSWFlat, then rerank top-K using IndexFlatIP or IndexFlatL2 locally for precision boost.
3. Entropy-aligned segmenting: Instead of slicing the dataset randomly, group vectors by entropy buckets (based on cluster compactness), then build local HNSW indexes per segment. This often stabilizes recall > 0.96 with less latency jump.

Let me know if you're curious. We've actually open-sourced a semantic controller that handles this under an MIT license. Happy to share more if needed.