kurobako is a command-line tool to benchmark black-box optimization algorithms.
Features:
- It has the capability to support wide range of optimization problems:
- various search sapce:
- Continuous numerical, discrete numerical and categorical
- Uniform distribution and log uniform distribution
- Conditional
- Constrainted problems
- Multi-objective problems
- various search sapce:
- Generating a markdown report and PNG plots from benchmarking results
- Easy to add user-defined optimization problems and solvers
- Simulating a concurrent environment in which an optimization process is executed by multiple workers simultaneously
- Reproducible
Precompiled binaries for Linux are available in the releases page.
$ curl -L https://github.com/optuna/kurobako/releases/download/${VERSION}/kurobako-${VERSION}.linux-amd64 -o kurobako
$ chmod +x kurobako
$ ./kurobako -hIf you have already installed Cargo, you can install kurobako by executing the following command:
$ cargo install kurobakoIf you want to use kurobako plot command, you need to have installed gnuplot package on your environment.
// Define solver.
$ kurobako solver random | tee solver.json
{"random":{}}
// Define problem.
$ curl -OL http://ml4aad.org/wp-content/uploads/2019/01/fcnet_tabular_benchmarks.tar.gz
$ tar xf fcnet_tabular_benchmarks.tar.gz && cd fcnet_tabular_benchmarks/
$ kurobako problem hpobench fcnet_protein_structure_data.hdf5 | tee problem.json
{"hpobench":{"dataset":"fcnet_protein_structure_data.hdf5"}}
// Run benchmark.
$ kurobako studies --solvers $(cat solver.json) --problems $(cat problem.json) | kurobako run > result.json
(ALL) [00:00:01] [STUDIES 10/10 100%] [ETA 0s] done
// Report the benchmark result.
$ cat result.json | kurobako report
...abbrev...
// Plot the benchmark result.
$ cat result.json | kurobako plot curve
(PLOT) [00:00:01] [1/1 100%] [ETA 0s] done (dir="images/curve/")Solvers:
Problems:
"kurobako" is a Japanese translation of "black box".