Open
Description
MMLU Evaluation currently has the following code block:
def __init__(
self,
model_path,
tasks_dir: Optional[str],
tasks: list[str],
model_dtype="bfloat16",
few_shots: int = 5,
batch_size: Optional[Union[int, str]] = "auto",
device: str = ("cuda" if torch.cuda.is_available() else "cpu"),
) -> None:
However; since device
is taking a default which relies on a dynamic value, it would best to instead set it to None
and compute the default when __init__
is called.
Metadata
Metadata
Assignees
Labels
No labels