System Info
Hi @McPatate, @remi-or,
I noticed a few things in BenchmarkConfig:
-
In check_validity, for continuous_batching you check sdpa_backend is not None and then set it to "math", but earlier it’s already set to "flash_attention" if None. The warning says it should be None, but it’s set to "math".
-
Level 3/4 in get_config_by_level generates all configs but ignores "paged|sdpa", which will be included in level 2.
Thanks!