Bayesian Optimization is a popular searching algorithm for hyper-parameters in the machine learning area.

There are also two popular Python libraries for this algorithm: Hyperopt and Optuna. So I have done some experiments on these two libraries. The trial is using LightGBM to classify tabular data, and the hyper-parameters and their ranges are:

- n_estimators (10~100000)
- learning_rate (0.0001~1.0)
- num_leaves (2, 2000)
- max_depth (2, 200)
- min_child_samples (1, 10000)
- reg_alpha (0.001, 10.0)
- reg_lambda (0.001, 10.0)

I run 10 trials each time for 10 times, by using different algorithms in Hyperopt and Optuna. The result for Hyperopt is:

And the result for Optuna is:

Seems that for my tabular data, TPE and Annealing works best, and Hyperopt works better than Optuna