2 Pull requests merged by 2 people

• Improve test

  #674 merged Apr 16, 2025

• Nathan unify modelargs

  #609 merged Apr 14, 2025

7 Pull requests opened by 3 people

• fix wrong 'custom_task_directory' in python api doc

  #671 opened Apr 14, 2025

• Adds multimodal support

  #675 opened Apr 15, 2025

• adds wandb loging of metrics

  #676 opened Apr 15, 2025

• hardcode configs to not make lighteval crash if lcb repo unavailable

  #677 opened Apr 17, 2025

• Inference provider model config to inherit from ModelConfig base class

  #678 opened Apr 17, 2025

• Update quicktour command

  #679 opened Apr 17, 2025

• adds simpleqa

  #680 opened Apr 17, 2025

4 Issues closed by 2 people

• [BUG] Use sampling for greedy mode

  #559 closed Apr 17, 2025

• [BUG] AttributeError: 'NoneType' object has no attribute 'builder_configs'

  #672 closed Apr 17, 2025

• [CORE] Make parsing of model args and generation parameters robust

  #579 closed Apr 15, 2025

• [BUG] The word `pretrained` is not accepted in `model_args` when using VLLM

  #673 closed Apr 14, 2025

7 Unresolved conversations

Sometimes conversations happen on old items that aren’t yet closed. Here is a list of all the Issues and Pull Requests with unresolved conversations.

• [BUG] encounter an ArrowInvalid error while saving experiment tracker

  #660 commented on Apr 15, 2025 • 0 new comments

• [BUG] can't pickle mappingproxy objects

  #668 commented on Apr 17, 2025 • 0 new comments

• [BUG] Transformers model padding should be to "longest"

  #663 commented on Apr 17, 2025 • 0 new comments

• [BUG] vLLM backend hangs with DDP

  #670 commented on Apr 17, 2025 • 0 new comments

• Fix `TGI` (Text Generation Inference) Endpoint Inference and TGI JSON Grammar Generation

  #502 commented on Apr 17, 2025 • 0 new comments

• Add G-Pass@k Metric

  #589 commented on Apr 17, 2025 • 0 new comments

• tokenizer 'padding' param is not correct.

  #669 commented on Apr 17, 2025 • 0 new comments