Skip to content

Conversation

@safikhanSoofiyani
Copy link

Fix issue number: #3435

This pull request fixes a cosmetic bug in get_task_dict / pretty_print_task in lm_eval/tasks/__init__.py, where lm_eval would crash when loading custom task configs that live outside the lm_eval/tasks/ directory.

The root cause is the following line:

relative_yaml_path = yaml_path.relative_to(lm_eval_tasks_path)

This assumes that every task's yaml_path is a subpath of lm_eval/tasks/. For external/custom configs in user project directories, this raises:

ValueError: '<path/to/custom.yaml>' is not in the subpath of '<path/to/lm_eval/tasks>' OR one path is relative and the other is absolute.

Since this is only used for logging/pretty-printing selected tasks, the crash is unnecessary and breaks previously valid usage patterns.

What this PR changes

Makes pretty_print_task robust to external YAML paths by falling back to the original path when relative_to() fails.

@CLAassistant
Copy link

CLAassistant commented Nov 28, 2025

CLA assistant check
All committers have signed the CLA.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants