Skip to content

Issues: huggingface/lighteval

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

95 Open 157 Closed
95 Open 157 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

[BUG] can't pickle mappingproxy objects bug Something isn't working
#668 opened Apr 9, 2025 by anilravuri
1
[BUG] Transformers model padding should be to "longest" bug Something isn't working
#663 opened Apr 7, 2025 by pingzhili
[FT] Log progress bar on main process feature request New feature/request
#662 opened Apr 7, 2025 by lewtun
[BUG] encounter an ArrowInvalid error while saving experiment tracker bug Something isn't working
#660 opened Apr 6, 2025 by DIaacKr
[BUG] Pipeline does not work with GenerationConfig bug Something isn't working
#657 opened Apr 4, 2025 by libeanim
[BUG] Very slow livecodebench scoring bug Something isn't working
#650 opened Mar 27, 2025 by rawsh
2
[BUG] When using accelerator, current model_args parser won't correctly parse dict-like arguments like generation_parameters bug Something isn't working
#646 opened Mar 27, 2025 by Ignoramus0817
[FT] Build in a way to specify specific IDs/Lines in Dataset to use as few-shot examples in the same split feature request New feature/request
#634 opened Mar 21, 2025 by EdwardSJ151
[BUG] ImportError: cannot import name 'T_co' from 'torch.utils.data.distributed' bug Something isn't working
#633 opened Mar 21, 2025 by blap
2
[BUG] OSError: [Errno 22] Invalid argument bug Something isn't working
#632 opened Mar 21, 2025 by blap
[EVAL] Clarification on Reproducing DeepSeek R1 Results with do_sampling=True new task
#631 opened Mar 20, 2025 by Louym
14
[BUG] core dump when using endpoint litellm bug Something isn't working
#623 opened Mar 19, 2025 by NiushanDong
[BUG] Num samples not respected bug Something isn't working
#618 opened Mar 17, 2025 by rawsh
5
[EVAL] Big-Bench Extra Hard (BBEH) new task
#600 opened Mar 3, 2025 by lewtun
[EVAL] SimpleQA new task
#599 opened Mar 3, 2025 by lewtun
[FT] Assistant Response Prefilling feature request New feature/request
#591 opened Feb 28, 2025 by matmult
Sth wrong in the parser for generation_parameters in main_sglang.py bug Something isn't working
#590 opened Feb 27, 2025 by HancCui
[CORE] Make parsing of model args and generation parameters robust prio
#579 opened Feb 21, 2025 by lewtun
[EVAL] Add TUMLU benchmark new task
#577 opened Feb 19, 2025 by gaydmi
10
Try to compute Thai finetasks score of llama 3.2 1B
#569 opened Feb 17, 2025 by Nevermetyou65
[FT] LiteLLM concurrency parameters hard-coded feature request New feature/request
#567 opened Feb 16, 2025 by lhl
[BUG] Use sampling for greedy mode bug Something isn't working
#559 opened Feb 13, 2025 by tonysy
1
[BUG] Nanotron runner imports non-existant bug Something isn't working
#555 opened Feb 13, 2025 by jquesnelle
3
how can i use this "community|alghafa:meta_ar_dialects " as a task
#554 opened Feb 12, 2025 by pratim808
couldn't find it in the cached files and it looks like Elron/bleurt-tiny-512, how to set the model path?
#545 opened Feb 8, 2025 by bannima
1
ProTip! What’s not been updated in a month: updated:<2025-03-10.