-
Notifications
You must be signed in to change notification settings - Fork 56
fix a few benchmark such that importing any of them works properly #127
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
The linter didn't pass for some files that I didn't change. I linted them but now it obfuscates the PR a bit. I recommend looking at the first commit only. |
Amazing! Thanks @jmercat ! |
@@ -67,7 +66,7 @@ def __init__( | |||
""" | |||
super().__init__(logger=logger, system_instruction=system_instruction) | |||
self.debug = debug | |||
self.max_new_tokens = max_tokens if max_tokens is not None else 32768 # set higher to avoid truncation for reasoning models | |||
self.max_new_tokens = max_tokens |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Don't we still need to check if max_tokens is not None ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
oh ok that's not the best way to handle it I think. I'll revert for now but we should probably not send an argument as none if we don't want it to be none.
When trying to load all benchmarks I found a few issues that this should fix.
Here is an import code that reveals the issues that this PR fixes: