Skip to content

Hyperparameter sweeps #143

Open
Open
@epwalsh

Description

@epwalsh

It would be great if Tango provided a simple yet general mechanism for doing hyperparameter searches. Here's an outline of how that could look 👇

We provide a new subcommand: tango sweep. This command takes

  • a "sweep" config,
  • a regular experiment config,
  • and a target step name.

For example:

tango sweep sweep-config.jsonnet target-config.jsonnet step-name

step-name should correspond to the main step of interest in target-config.jsonnet that provides the results we are trying to optimize for (might require #142). For example, this could be a validation/eval step that spits out some metrics of your model on a dataset.

The sweep-config.jsonnet would define which hyperparameters to search and how to search over them. By "hyperparemeters" I really just mean any fields in target-config.jsonnet. There are many ways we could do the search, and this is an active area of research. So I think it would be ideal if we were able to integrate with existing hyperparameter sweep frameworks / platforms, like W&B, Optuna, etc. These integrations should be optional, however, and I think we should provide a simple default search method, which could just be grid search.

Under the hood tango sweep could use the tango run subcommand with the --overrides parameter to select hyperparameter values. We should also be able to run the search in parallel.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions