can't reproduce AE-LC numbers in hf ckpt(Llama-3-8b-SFT-DPO, Llama-3-8b-SFT-SimPO))

[schrieffer-z]

1. vllm under any version mismatch with current env and if you separate eval and train. I still need a version of vllm
2. in separate env, using the command below, I try different engine (4o and 4-turbo) and get some numbers dose not make sense. Have you ever try different annotators when use 4o, it give me a result where DPO>SimPO, while 4-turbo gives the opposite
   

alpaca_eval evaluate_from_model
--model_configs /mnt/vepfs/fs_users/***/xAI-RLHF/***/SimPO/eval/alpacaeval2/configs/Llama-3-Base-8B-SFT-SimPO.yaml
--annotators_config weighted_alpaca_eval_gpt4_turbo\

[schrieffer-z]

repo's AE-LC number:
simpo - 22%
dpo - 18.2%

[lancerts]

Had a similar issue with ckpt [princeton-nlp/Llama-3-Instruct-8B-SimPO-v0.2](https://huggingface.co/princeton-nlp/Llama-3-Instruct-8B-SimPO-v0.2)
3 AE runs gives consistent LC/WR (48/44) where the number reported is 53.7/47.5.
One potential issue could be #75 and there is no vllm version specified in https://github.com/princeton-nlp/SimPO/blob/main/environment.yml

[visionxyz]

What vllm version should we use? I can't even run the alpaca eval now. Struggling on beam search related problem.

[schrieffer-z]

What vllm version should we use? I can't even run the alpaca eval now. Struggling on beam search related problem.

I used anther env to run evaluation. And it works fine expect that results are close but less than the reported numer by 1-2 points

[visionxyz]

I think it is already good enough. I tried to run Zephyr in the config of Alpaca eval, it is also 1-2 points worse than the recorded result.

[xiamengzhou]

Hi sorry for the late reply! At the time of development, we used vllm==0.5.4, but I think the newest version should work just fine!