[EVAL] Clarification on Reproducing DeepSeek R1 Results with do_sampling=True

[Louym]

Evaluation short description

In the technical report of DeepSeek R1, detailed reasoning results are provided for datasets such as Math-500, GPQA, and AIME. It’s exciting to see that lighteval supports all of them—thank you for this fantastic work!

However, while attempting to reproduce these results, I encountered some issues. The report specifies using do_sampling=True, but I couldn’t find clear instructions on how to configure this setting. To address this, I modified the initialization function of generation_config in the transformers package. Does this approach make sense?

Additionally, I noticed that generation_config is initialized six times—three times before loading weights and three times after. Could you clarify the purpose of these multiple initializations?

I’d really appreciate any insights. Thanks again for your work!
@clefourrier @NathanHB @hynky1999
(Not sure who to tag, so I randomly @ some contributors. Please feel free to ignore if this is not relevant to you!)

Evaluation metadata

Provide all available

• Paper url: https://arxiv.org/pdf/[2501.12948](https://arxiv.org/pdf/2501.12948)

[NathanHB]

hi ! you can set generation parameters using a config file, see here at the bottom of the page.

[Louym]

@NathanHB Thank you so much for your reply! I still have one question. If I just want to evaluate the model locally with vllm as the backend, how can I use sampling?

[NathanHB]

the same way, if you use temperature, it will activate sampling

[Louym]

Thank you very much! Just one more question: can sampling be performed when using Accelerate?

[NathanHB]

yes ! But we advise on using vllm especially for bigger models

[Louym]

Thank you so much!
When using vllm, I encountered another issue, since the reasoning benchmark requires many tokens.
I wonder where to modify the self.max_length.

[2025-03-24 09:46:45,253] [ WARNING]: context_size + max_new_tokens=11004 which is greater than self.max_length=8192. Truncating context to 0 tokens. (vllm_model.py:266)

[NathanHB]

self.max_length is modified using max_model_length model arg.

[Louym]

Thank you so much! If i want get the results of AIME 2024 cons@64, should I write this in the yaml file?
metric_options: # Optional metric arguments
exact_match@1: 16
num_samples: 16

[NathanHB]

yes !

[NathanHB]

though @plaguss will know more than me about this :)

[Louym]

@NathanHB Thank you for your feedback! I really appreciate you taking the time to investigate this.

Regarding pass@K, I've implemented it as shown here, but the runtime appears unchanged. @plaguss Would you have any insights on how we might better capture/interpret the results?

Thank you so much! If i want get the results of AIME 2024 cons@64, should I write this in the yaml file? metric_options: # Optional metric arguments exact_match@1: 16 num_samples: 16

[rawsh]

This works well for me: #618 (comment)

I've been able to reproduce the DeepScaleR pass@1 n=16 results:

@plaguss @NathanHB Wonder what y'all think about support for reporting multiple metrics? E.g. when sampling 16 times you should be able to report pass@1, pass@16, and consensus@16 from the same data. Would also love more control over sampling so it's easier to implement custom "reranking" setups like Best-of-N

[ZepinLi]

@rawsh
I'm also working on reproducing DeepScaleR pass@1 n=16 results. Could you please share with me the detailed pipeline of using lighteval to do this? It seems like that lighteval uses n=1 for default setting.

[ZepinLi]

hi @NathanHB, I'm testing AIME pass@1 (n>1) score with script like: https://github.com/huggingface/open-r1?tab=readme-ov-file#aime-2024. Could you provide some insights of integrating them all together -- configs of vllm, openr1 and lighteval?

thanks you.