Description
It would be great if Tango provided a simple yet general mechanism for doing hyperparameter searches. Here's an outline of how that could look 👇
We provide a new subcommand: tango sweep
. This command takes
- a "sweep" config,
- a regular experiment config,
- and a target step name.
For example:
tango sweep sweep-config.jsonnet target-config.jsonnet step-name
step-name
should correspond to the main step of interest in target-config.jsonnet
that provides the results we are trying to optimize for (might require #142). For example, this could be a validation/eval step that spits out some metrics of your model on a dataset.
The sweep-config.jsonnet
would define which hyperparameters to search and how to search over them. By "hyperparemeters" I really just mean any fields in target-config.jsonnet
. There are many ways we could do the search, and this is an active area of research. So I think it would be ideal if we were able to integrate with existing hyperparameter sweep frameworks / platforms, like W&B, Optuna, etc. These integrations should be optional, however, and I think we should provide a simple default search method, which could just be grid search.
Under the hood tango sweep
could use the tango run
subcommand with the --overrides
parameter to select hyperparameter values. We should also be able to run the search in parallel.