It would be great if Tango provided a simple yet general mechanism for doing hyperparameter searches. Here's an outline of how that could look 👇

We provide a new subcommand: tango sweep. This command takes

• a "sweep" config,
• a regular experiment config,
• and a target step name.

For example:

tango sweep sweep-config.jsonnet target-config.jsonnet step-name

step-name should correspond to the main step of interest in target-config.jsonnet that provides the results we are trying to optimize for (might require #142). For example, this could be a validation/eval step that spits out some metrics of your model on a dataset.

The sweep-config.jsonnet would define which hyperparameters to search and how to search over them. By "hyperparemeters" I really just mean any fields in target-config.jsonnet. There are many ways we could do the search, and this is an active area of research. So I think it would be ideal if we were able to integrate with existing hyperparameter sweep frameworks / platforms, like W&B, Optuna, etc. These integrations should be optional, however, and I think we should provide a simple default search method, which could just be grid search.

Under the hood tango sweep could use the tango run subcommand with the --overrides parameter to select hyperparameter values. We should also be able to run the search in parallel.