Different results by retriever.encode_and_retrieve and retriever.retrieve

[liyongkang123]

Hi, I encountered an issue while evaluating the performance of contriever-msmarco on the Arguana dataset using the official example.

When running the following code:

results = retriever.retrieve(corpus, queries)
results = retriever.encode_and_retrieve(corpus, queries)
ndcg, _map, recall, precision = retriever.evaluate(qrels, results, retriever.k_values, ignore_identical_ids=False)

I noticed that the results output might differ depending on which method (retrieve or encode_and_retrieve) is used. Specifically, encode_and_retrieve may include document IDs that are the same as the query IDs.

I always set ignore_identical_ids=False, and when using retrieve, I get a normal ndcg@10=44 by retrieve. However, when using encode_and_retrieve, the ndcg@10 is much lower at only 33.4. After comparing the results from both methods, I found that encode_and_retrieve includes query-document similarity with the query itself, which causes the issue.

I would like to know how I can fix this problem, as I intend to save the embeddings and use them.

Thank you in advance!

[thakur-nandan]

Hi @liyongkang123,

Thanks for catching the error; you are correct. I missed implementing the logic to remove query_id that are identical to itself in the document list. This is why your scores are coming out to be lower than expected.

I'll write a bugfix script to resolve this by tomorrow.

Regards,
Nandan

[liyongkang123]

Hi Nandan, I’m not sure if this is the right approach. If there are cases where a doc_id is the same as a query_id in the corpus, I wonder whether this should be handled under the ignore_identical_ids parameter?

Perhaps we could compare the text of the doc_id and the query_id, and if they are the same, we could remove it.

[liyongkang123]

Hi Nandan, I also found the following difference:

retriever = EvaluateRetrieval(model, score_function=score_function, k_values=[1, 5, 10, 50, 100, 1000])  # "dot" for dot product cos_sim
results = retriever.encode_and_retrieve(corpus, queries, encode_output_path=embedding_save_path, overwrite=True)
results = retriever.retrieve(corpus, queries)

If we set a score_function for the retriever, it will only affect the retriever.retrieve function and not retriever.encode_and_retrieve.

This is because retriever.encode_and_retrieve first encodes and saves the embeddings, while in the search_from_files function, it directly calls faiss.IndexFlatIP for inner product-based indexing. In short, if you use retriever.encode_and_retrieve, you must normalize the embeddings directly when the model outputs them if you intend to compute cosine similarity.

[thakur-nandan]

Hi @liyongkang123,

This is really useful. Will you be able to create a PR to include the score function within the retriever.encode_and_retrieve function?

[liyongkang123]

Hi Nandan, I can try doing it this way. However, I'm not sure when the normalization is performed. We have two strategies:

1. Perform normalization before saving the embedding, then continue using faiss.IndexFlatIP.
2. Save the embedding with user-controlled model output, and normalize all embeddings before using faiss.IndexFlatIP.
   Do you have a preference?

[thakur-nandan]

If we would like to do cosine-similarity, then (1), then normalizing before saving the embedding makes sense to me.