Add benchmark sweeper configuration and auto-override for benchmark type

[tadiwa-aizen]

What changed and why?

Fixed an issue where benchmark parameters for unselected benchmark types were incorrectly included in multirun sweeps. Previously, all benchmark parameters were defined in the common sweeper config, causing irrelevant parameters to be swept even when running specific benchmark types.

Changes:

• Added auto-detection of benchmark_type parameter to automatically select appropriate sweeper config
• Split benchmark-specific sweep parameters into separate config files (fio.yaml, prefetch.yaml, client-bp.yaml, etc.)
• Moved common parameters to base.yaml sweeper config
• Now only relevant parameters are swept for each benchmark type

Example: Running benchmark_type=fio now only sweeps FIO-specific parameters instead of including unrelated prefetch or client-backpressure parameters.

Does this change impact existing behavior?

No breaking changes. Existing commands work as before, but now sweep only relevant parameters for the specified benchmark type.

Does this change need a changelog entry? Does it require a version change?

None needed

---

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and I agree to the terms of the Developer Certificate of Origin (DCO).

[mansi153]

So we don't miss it later, can we pls remove 128 app workers from here as discussed?

[mansi153]

I could be misunderstanding something but is it possible to move all this default configuration (upto line 31) into base.yml too and refer to that as @hydra.main(version_base=None, config_path="hydra/sweeper", config_name="base") or so?

We could also then move the benchmark specific static/default configs to their respective config files, for e.g. mountpoint.mountpoint_max_background: !!null # or say, 64 can move into the Fio yml file so it's easier to see what all defaults we're configuring and which all are then being overridden in the sweeper

[mansi153]

I don't think this should be here. It should either be in fio.yml for now or a separate mountpoint.yml which the Fio one (and others in the future) can inherit this configuration from

[mansi153]

Should this mem_limiter config really be turned on by default?

[mansi153]

While we're at it, let's also think about what reasonable values we would want here. Let's not put 8MB as a test, as it's too small and might not be too meaningful. I think 2GB, 8GB and maybe 16GB are the values I'd suggest testing here.

[mansi153]

Can we pls make this default to 10 (or something >1) here?

[mansi153]

Do we stay backward compatible and default to benchmark_type=fio here as well so we don't miss sweeping the extra params, given we already silently default to fio benchmark with benchmark_type: "fio"?

[mansi153]

I'm not sure I understand. Do you mind elaborating how this works or a sample usage? Where is the hydra/benchmark_sweeper= value coming from?

Can we do something like below, or maybe I'm missing something?

def get_config_name():
    env = os.getenv("benchmark_type", "fio")
    config_map = {
        "fio": "config_fio",
        "prefetch": "config_prefetch", 
        "client_bp": "config_client_bp",
        "client": "config_client",
        "crt": "config_crt"
    }
    return config_map.get(env, "config")

@hydra.main(version_base=None, config_path="conf/hydra/sweeper", config_name=get_config_name())
def run_experiment(cfg: DictConfig) -> None:
    blah blah