Search results

Cannot get length controlled win rate

Status: Open.

#453 In tatsu-lab/alpaca_eval;

· FocusLiwen opened on Jun 20, 2025

What models are used when GPT-4-1106-preview is inaccessible?

Status: Open.

#452 In tatsu-lab/alpaca_eval;

· fulQuan opened on Jun 10, 2025

How to determine the possibility of another LLM as a judge model?

Status: Open.

#450 In tatsu-lab/alpaca_eval;

· AmberXu98 opened on Jun 6, 2025

When fn_completions is set to huggingface_local_completions, the model will be reloaded for each chunk causing OOM error

Status: Open.

#449 In tatsu-lab/alpaca_eval;

· arushibear opened on Jun 3, 2025

Why is the output “[\n {‘model’: ‘model_1’, ‘rank’: 1},\n {‘model’: ‘model_2’, ‘rank’: 2}\n]” and the final result is “preference”: 2.0, when these two do not correspond? What is the reason for this?

Status: Open.

#447 In tatsu-lab/alpaca_eval;

· FHhui opened on May 12, 2025

"You must manually fix the score pair" Error While Using Custom Evaluator

Status: Open.

#446 In tatsu-lab/alpaca_eval;

· tommycwh opened on May 8, 2025

ValueError: cannot reindex on an axis with duplicate labels

Status: Open.

#445 In tatsu-lab/alpaca_eval;

· Liesy opened on May 1, 2025

when model output is like deepseek-r1 ,wondering if it is better to omit content inside <think>

Status: Open.

#442 In tatsu-lab/alpaca_eval;

· jijivski opened on Mar 22, 2025

Allow specifying model revision

Status: Open.

#441 In tatsu-lab/alpaca_eval;

· tomtseng opened on Feb 8, 2025

Can you realize deepseek as a judge?

Status: Open.

#440 In tatsu-lab/alpaca_eval;

· NuoJohnChen opened on Feb 8, 2025

openai.BadRequestError

Status: Open.

#436 In tatsu-lab/alpaca_eval;

· EnlightenedAI opened on Jan 24, 2025

Question about best_of when use_beam_search=True

Status: Open.

#435 In tatsu-lab/alpaca_eval;

· ZeguanXiao opened on Jan 18, 2025