return multiple trials per suggestion for bayesian optimization

[zrcahyx]

Since training a neural network can be time-consuming, so it's efficient to train multiple model simultaneously. However Advisor's bayesian optimization currently returns only one trial per suggestion, hence we cannot do multiple trials at the same time. Due to this reason, I made a few changes to bayesian_optimization.py, so that it can return more than one trial per suggestion.

[tobegit3hub]

Great and thanks for contribution @zrcahyx 👍

I will have a look at this pull-request and test soon.

[zrcahyx]

@tobegit3hub The idea is simple, just return the top N x_tries which have highest acquisition function values. But since sample number are set to 100000 that is high, so returned top N trials are quite close to each other which can be indistinguishable. Do you think it's better to mix random search with bayesian optimization? For example, we just return the one-best trial via bayesian optimization and the rest number - 1 trials use random search. In this way, priori samples could be more distinguishable. But it's kind of strange, so if you have more elegant solutions for returning more than one bayesian optimization suggestions, please let me know.

[zrcahyx]

@tobegit3hub Hi, I have made a second commit to this pull-request that abandons the bound_dict to avoid iteration over a dict which can be dangerous and make bounds to be out-of-order. This is quite important. If not doing so, bayesian optimization over high dimensional features could lead to a failure.

[tobegit3hub]

I remember that the problem of retuning multiple trials is not about running the GaussianProgress in multiple times. If we don't give the new data to run BayesianOptimization, the acquisition function may return the same value. In short, if you get multiple trials from BayesianOptimization without new data, the return values may be similar.

[zrcahyx]

@tobegit3hub Sure, without new data, if you try to get suggestion from BayesianOptimization multiple times, returned trials will have parameters quite similar(because GaussianProgress fit the same completed trials). What I mean is to return multiple trials per suggestion, we do just one time GaussianProgress fit and return the topN trials which have topN highest acquisition values.

[tobegit3hub]

Yes, you know what I means @zrcahyx .

But the acquisition function is continuous which means that if we get the highest point, the second highest point is next to it. I'm not sure if we can get the TopN values in different reasonable points.

[zrcahyx]

Yes, that's the problem! @tobegit3hub

So that's why I asked you whether it's better to mix random search with bayesian optimization? Or other better solutions maybe. But it's sure that iterate over dict is dangerous.

[LichaoLiu666]

@zrcahyx , I found your code and planned to to test your parallel Bayesian Optimization algorithm. However, it seems that it relies on suggestion.algorithm.base_algorithm, which is not submitted through PR. Can you take a look?