Qwen2.5-Coder-instruct-3b only achieves 45.12 pass@1 on humaneval using Opencompass

[TsingZ0]

Qwen2.5-Coder-instruct-3B only achieves a pass@1 score of 45.12 on HumanEval and 30.20 on MBPP using the well-known benchmark platform OpenCompass, which is significantly lower than the reported 84.1 and 73.6 in the Qwen2.5-Coder technical report (Table 16).
The logs are attached.

Could you clarify the possible reasons for this discrepancy?

run-humaneval-0.0-Qwen2.5-Coder-3B-Instruct.txt
run-mbpp-0.0-Qwen2.5-Coder-3B-Instruct.txt

Commands:

nohup python -u run.py --datasets humaneval_passk_gen_8e312c --hf-type base --hf-path ../models/Qwen2.5-Coder-3B-Instruct --max-out-len 512 --generation-kwargs do_sample=False temperature=0.0 --debug > run-humaneval-0.0-Qwen2.5-Coder-3B-Instruct.out 2>&1 &

nohup python -u run.py --datasets mbpp_passk_gen_830460 --hf-type base --hf-path ../models/Qwen2.5-Coder-3B-Instruct --max-out-len 512 --generation-kwargs do_sample=False temperature=0.0 --debug > run-mbpp-0.0-Qwen2.5-Coder-3B-Instruct.out 2>&1 &

[TsingZ0]

Additionally, I noticed that Qwen2.5-Coder applies various dataset-specific processing steps. Could this be the reason why the results in the technical report are significantly better?

See the specific EOS examples for humanEval and mbpp: https://github.com/QwenLM/Qwen2.5-Coder/blob/main/qwencoder-eval/instruct/eval_plus/model.py. Are these tricks common in the community?

[TsingZ0]

It appears that this evaluation setup is inherited from EvalPlus: https://github.com/evalplus/evalplus/blob/master/evalplus/provider/utility.py

[SebastianSchiefer]

I think there might be something wrong with your evaluation setup. I can replicate the results from the technical report with my own local setup for the humaneval dataset:

• llama.cpp for inference, a release from ~8 days ago
• self-written code to query the backend + run the generated code in a sandbox
• the exact same prompts as here
• sampling parameters: temperature: 0, n_predict: 512, seed: 0
• no EOS strings used, i achieve similar results by filtering the model answers with regexes(probably inefficient, but should not matter since i do not reuse the context)

i get

• 88.4%(145/164) with 7b-instruct-q8-0
• 84.15%(138/164) with 3b-instruct-q8_0

I don't know how to figure out the difference between your setup and the one used in the tech-report. I just want to point out that IMO the issue is in your setup and not the author's results.

Good Luck Sir :)

[TsingZ0]

@SebastianSchiefer Thank you. I’ve successfully reproduced the results using EvalPlus with specific EOS tokens and post-processing tailored to each dataset. However, when using a general prompt without such dataset-specific handling—like in OpenCompass—the performance drops significantly.

This tailored approach seems more like a trick to report better results, which might be common in the community, but it doesn't reflect the model’s actual capability in a real-world or product environment.

As mentioned, you also apply filtering.