Fix pretty_print_task for external custom task configs #3436
+4
−1
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fix issue number: #3435
This pull request fixes a cosmetic bug in
get_task_dict/pretty_print_taskinlm_eval/tasks/__init__.py, wherelm_evalwould crash when loading custom task configs that live outside thelm_eval/tasks/directory.The root cause is the following line:
This assumes that every task's yaml_path is a subpath of
lm_eval/tasks/. For external/custom configs in user project directories, this raises:Since this is only used for logging/pretty-printing selected tasks, the crash is unnecessary and breaks previously valid usage patterns.
What this PR changes
Makes
pretty_print_taskrobust to external YAML paths by falling back to the original path whenrelative_to()fails.