One-stage rerank results are suspiciously poor

[Steel-faced-Duke]

Describe the bug

I just annotate stage 1 code to get rerank results by CrossEncoder directly, but the rerank results are very poor, code is like this:

from mteb import MTEB
import mteb
from sentence_transformers import CrossEncoder, SentenceTransformer

dual_encoder = SentenceTransformer("all-MiniLM-L6-v2")

cross_encoder = CrossEncoder("cross-encoder/ms-marco-TinyBERT-L-2-v2")

tasks = mteb.get_tasks(tasks=["NFCorpus"], languages=["eng"])

subset = "default" # subset name used in the NFCorpus dataset
eval_splits = ["test"]

evaluation = MTEB(tasks=tasks)
# evaluation.run(#no stage 1
#    dual_encoder,
#    eval_splits=eval_splits,
#    save_predictions=True,
#    output_folder="results/stage1",
# )
evaluation.run(
    cross_encoder,
    eval_splits=eval_splits,
    # top_k=500,
    save_predictions=True,
    output_folder="results/stage2",
    # previous_results=f"results/stage1/NFCorpus_{subset}_predictions.json", # get results directly by CrossEncoder
)

here is some of the stage 1 results when I made this change before:

        "recall_at_1": 0.04323,
        "recall_at_3": 0.09053,
        "recall_at_5": 0.11959,
        "recall_at_10": 0.15499,
        "recall_at_20": 0.18891,
        "recall_at_100": 0.31151,
        "recall_at_1000": 0.63211,
        "precision_at_1": 0.41486,
        "precision_at_3": 0.34881,
        "precision_at_5": 0.30279,
        "precision_at_10": 0.24334,
        "precision_at_20": 0.17817,
        "precision_at_100": 0.07981,
        "precision_at_1000": 0.02096,

and this is the result directly by CrossEncoder:

        "recall_at_1": 0.00784,
        "recall_at_3": 0.01523,
        "recall_at_5": 0.01939,
        "recall_at_10": 0.02783,
        "recall_at_20": 0.03382,
        "recall_at_100": 0.05049,
        "recall_at_1000": 0.12993,
        "precision_at_1": 0.13313,
        "precision_at_3": 0.11765,
        "precision_at_5": 0.10774,
        "precision_at_10": 0.08545,
        "precision_at_20": 0.0596,
        "precision_at_100": 0.02201,
        "precision_at_1000": 0.00699,

Running a single CrossEncoder , the results should be much better than a single SentenceTransformer,but it is not, so what is the problem?

Here is my environment:
sentence-transformers 5.0.0
mteb 1.38.35

To reproduce

I just use the official rerank code and annotate the stage 1(as the discription) to reproduce，here is the official rerank code:

from mteb import MTEB
import mteb
from sentence_transformers import CrossEncoder, SentenceTransformer

cross_encoder = CrossEncoder("cross-encoder/ms-marco-TinyBERT-L-2-v2")
dual_encoder = SentenceTransformer("all-MiniLM-L6-v2")

tasks = mteb.get_tasks(tasks=["NFCorpus"], languages=["eng"])

subset = "default" # subset name used in the NFCorpus dataset
eval_splits = ["test"]

evaluation = MTEB(tasks=tasks)
evaluation.run(
    dual_encoder,
    eval_splits=eval_splits,
    save_predictions=True,
    output_folder="results/stage1",
)
evaluation.run(
    cross_encoder,
    eval_splits=eval_splits,
    top_k=5,
    save_predictions=True,
    output_folder="results/stage2",
    previous_results=f"results/stage1/NFCorpus_{subset}_predictions.json",
)

Additional information

No response

Are you interested to contribute a fix for this bug?

No

[Samoed]

Running a single CrossEncoder , the results should be much better than a single SentenceTransformer,but it is not, so what is the problem?

There is no problem. In fact, some cross-encoders might lead to a decrease in performance due to their inaccurate predictions. To improve performance, you could try more advanced cross-encoders, e.g. https://huggingface.co/mixedbread-ai/mxbai-rerank-large-v2

[Steel-faced-Duke]

Running a single CrossEncoder , the results should be much better than a single SentenceTransformer,but it is not, so what is the problem?

There is no problem. In fact, some cross-encoders might lead to a decrease in performance due to their inaccurate predictions. To improve performance, you could try more advanced cross-encoders, e.g. https://huggingface.co/mixedbread-ai/mxbai-rerank-large-v2

I have tried some advanced models:

cross-encoder/ms-marco-TinyBERT-L-2-v2
Qwen/Qwen3-Reranker-0.6B
BAAI/bge-reranker-v2-m3
jinaai/jina-reranker-v2-base-multilingual
Alibaba-NLP/gte-multilingual-reranker-base
mixedbread-ai/mxbai-rerank-base-v2

all of them preform the same: While using the stage1 "previous_results" , It performs very well. And performs poor while not.

[KennethEnevoldsen]

Thanks for the issue @gangmiangongjue. CrossEncoders are not guaranteed to lead to an improvement and def. not guaranteed to do so on domain-specific benchmarks, such as medical retrieval.

Maybe a better dataset to test it out on would be WikipediaRetrievalMultilingual?

I removed the bug label from this issue for now.

@orionw would love your eyes on this as well

[orionw]

Yeah +1 to @KennethEnevoldsen and @Samoed's comments.

You'll also likely have worse performance using a much smaller cross-encoder after a bigger bi-encoder. In this case your first stage is 22M parameters and your reranker is only 4M. You'd have better performance if you use a comparatively sized reranker.

all of them preform the same: While using the stage1 "previous_results" , It performs very well. And performs poor while not.

I don't fully understand this, you should have greatly different scores for each model.

Perhaps you are looking at metrics beyond 5 - since you are using top_k of 5, every metric past 5 should remain the same (e.g. same recall@10). Reranking a larger number will also show greater differences in rerankers. It would also probably help to use a bigger first stage model as that would surface more positives that your rerankers could rerank.

[Steel-faced-Duke]

Yeah +1 to @KennethEnevoldsen and @Samoed's comments.

You'll also likely have worse performance using a much smaller cross-encoder after a bigger bi-encoder. In this case your first stage is 22M parameters and your reranker is only 4M. You'd have better performance if you use a comparatively sized reranker.

all of them preform the same: While using the stage1 "previous_results" , It performs very well. And performs poor while not.

I don't fully understand this, you should have greatly different scores for each model.

Perhaps you are looking at metrics beyond 5 - since you are using top_k of 5, every metric past 5 should remain the same (e.g. same recall@10). Reranking a larger number will also show greater differences in rerankers. It would also probably help to use a bigger first stage model as that would surface more positives that your rerankers could rerank.

Thanks for help @Samoed @KennethEnevoldsen @orionw , I made some changes:

1. change dataset to WikipediaRetrievalMultilingual
2. run 4 cross-encoders to compare with the sentence-transformer
   The cross-encoder results still show a poor rate, and this is my code:

from mteb import MTEB
import mteb
from sentence_transformers import CrossEncoder, SentenceTransformer

dual_encoder = SentenceTransformer("all-MiniLM-L6-v2")

cross_model_name = ['cross-encoder/ms-marco-TinyBERT-L-2-v2','BAAI/bge-reranker-v2-m3','jinaai/jina-reranker-v2-base-multilingual','Alibaba-NLP/gte-multilingual-reranker-base']

tasks = mteb.get_tasks(tasks=["WikipediaRetrievalMultilingual"], languages=["eng"])

subset = "default" # subset name used in the NFCorpus dataset
eval_splits = ["test"]

evaluation = MTEB(tasks=tasks)
evaluation.run(
    dual_encoder,
    eval_splits=eval_splits,
    save_predictions=True,
    output_folder="results/only_embedding",
)
for cross_model in cross_model_name:
    cross_encoder = CrossEncoder(cross_model,trust_remote_code=True)
    evaluation.run(
        cross_encoder,
        eval_splits=eval_splits,
        save_predictions=True,
        output_folder=" results/only_rerank_{}".format(cross_model),
        # previous_results=f"results/stage1/WikipediaRetrievalMultilingual_en_predictions.json", #run cross-encoder directly
    )

And here is all the result:
results/only_embedding:

        "recall_at_1": 0.78267,
        "recall_at_3": 0.91867,
        "recall_at_5": 0.95533,
        "recall_at_10": 0.98267,
        "recall_at_20": 0.986,
        "recall_at_100": 0.99667,
        "recall_at_1000": 0.99933,
        "precision_at_1": 0.78267,
        "precision_at_3": 0.30622,
        "precision_at_5": 0.19107,
        "precision_at_10": 0.09827,
        "precision_at_20": 0.0493,
        "precision_at_100": 0.00997,
        "precision_at_1000": 0.001,

results/only_rerank_cross-encoder/ms-marco-TinyBERT-L-2-v2

        "recall_at_1": 0.06733,
        "recall_at_3": 0.072,
        "recall_at_5": 0.074,
        "recall_at_10": 0.074,
        "recall_at_20": 0.074,
        "recall_at_100": 0.074,
        "recall_at_1000": 0.074,
        "precision_at_1": 0.06733,
        "precision_at_3": 0.024,
        "precision_at_5": 0.0148,
        "precision_at_10": 0.0074,
        "precision_at_20": 0.0037,
        "precision_at_100": 0.00074,
        "precision_at_1000": 7e-05,

results/Alibaba-NLP/gte-multilingual-reranker-base

        "recall_at_1": 0.07133,
        "recall_at_3": 0.07333,
        "recall_at_5": 0.07333,
        "recall_at_10": 0.074,
        "recall_at_20": 0.074,
        "recall_at_100": 0.074,
        "recall_at_1000": 0.074,
        "precision_at_1": 0.07133,
        "precision_at_3": 0.02444,
        "precision_at_5": 0.01467,
        "precision_at_10": 0.0074,
        "precision_at_20": 0.0037,
        "precision_at_100": 0.00074,
        "precision_at_1000": 7e-05,

results/BAAI/bge-reranker-v2-m3

        "recall_at_1": 0.03533,
        "recall_at_3": 0.03733,
        "recall_at_5": 0.03733,
        "recall_at_10": 0.03733,
        "recall_at_20": 0.03733,
        "recall_at_100": 0.03733,
        "recall_at_1000": 0.03733,
        "precision_at_1": 0.03533,
        "precision_at_3": 0.01244,
        "precision_at_5": 0.00747,
        "precision_at_10": 0.00373,
        "precision_at_20": 0.00187,
        "precision_at_100": 0.00037,
        "precision_at_1000": 4e-05,

results/jinaai/jina-reranker-v2-base-multilingual

        "recall_at_1": 0.06933,
        "recall_at_3": 0.07267,
        "recall_at_5": 0.074,
        "recall_at_10": 0.074,
        "recall_at_20": 0.074,
        "recall_at_100": 0.074,
        "recall_at_1000": 0.074,
        "precision_at_1": 0.06933,
        "precision_at_3": 0.02422,
        "precision_at_5": 0.0148,
        "precision_at_10": 0.0074,
        "precision_at_20": 0.0037,
        "precision_at_100": 0.00074,
        "precision_at_1000": 7e-05,

As we can see, directly running cross-encoder lead to a poor rate

[orionw]

Hmm, that does seem like an issue. What version of MTEB are you using? I suspect it may be fixed on the v2 branch...

[Steel-faced-Duke]

Hmm, that does seem like an issue. What version of MTEB are you using? I suspect it may be fixed on the v2 branch...

sentence-transformers 5.0.0
mteb 1.38.35

[KennethEnevoldsen]

Let me run this on both the latest version of main and v2.0.0

[KennethEnevoldsen]

So, I rewrote the script slightly (shouldn't influence performance, but if it does, that would be revealing):

from __future__ import annotations

import mteb
from mteb import MTEB

dual_encoder = mteb.get_model("sentence-transformers/all-MiniLM-L6-v2")

tasks = mteb.get_tasks(tasks=["WikipediaRetrievalMultilingual"], languages=["eng"])

evaluation = MTEB(tasks=tasks)
evaluation.run(
    dual_encoder,
    save_predictions=True,
    output_folder="results/only_embedding",
)

cross_model_name = "jinaai/jina-reranker-v2-base-multilingual" # using just one model should be enough for debugging
# cross_encoder = CrossEncoder(cross_model_name, trust_remote_code=True)
cross_encoder = mteb.get_model(cross_model_name)
evaluation.run(
    cross_encoder,
    save_predictions=True,
    output_folder=f" results/only_rerank_{cross_model_name.split('/')[-1]}",
    # previous_results=f"results/stage1/WikipediaRetrievalMultilingual_en_predictions.json", #run cross-encoder directly
)

First, let's see if the scores for the same for all-MiniLM-L6-v2:

# using:
# v2.0.0: sentence-transformers/all-MiniLM-L6-v2
{
    "dataset_revision": "5f6c91d21f2f5b9afb663858d19848fbd223c775",
    "task_name": "WikipediaRetrievalMultilingual",
    "scores": {
        "test": [
            {
                # [...]
                "recall_at_1": 0.78267,
                "recall_at_3": 0.91867,
                "recall_at_5": 0.95533,
                "recall_at_10": 0.98267,
                "recall_at_20": 0.986,
                "recall_at_100": 0.99667,
                "recall_at_1000": 0.99933,
                "precision_at_1": 0.78267,
                "precision_at_3": 0.30622,
                "precision_at_5": 0.19107,
                "precision_at_10": 0.09827,
                "precision_at_20": 0.0493,
                "precision_at_100": 0.00997,
                "precision_at_1000": 0.001,
                # [...]
            }
        ]
    },
}

Great, they are!

So what about the reranking:
This produces slightly different results, but results which also seem concerning:

# using:
# v2.0.0: jinaai/jina-reranker-v2-base-multilingual
{
    "dataset_revision": "5f6c91d21f2f5b9afb663858d19848fbd223c775",
    "task_name": "WikipediaRetrievalMultilingual",
    "scores": {
        "test": [
            {
                # [...]
                "recall_at_1": 0.0,
                "recall_at_3": 0.0,
                "recall_at_5": 0.00067,
                "recall_at_10": 0.00133,
                "recall_at_20": 0.002,
                "recall_at_100": 0.00667,
                "recall_at_1000": 0.074,
                "precision_at_1": 0.0,
                "precision_at_3": 0.0,
                "precision_at_5": 0.00013,
                "precision_at_10": 0.00013,
                "precision_at_20": 0.0001,
                "precision_at_100": 7e-05,
                "precision_at_1000": 7e-05,
                # [...]
            }
        ]
    },
}

Haven't had the chance to rerun it in the latest version on main yet