import numpy as np
from math import inf
from spotpython.fun.objectivefunctions import Analytical
from spotpython.spot import spot
from scipy.optimize import shgo
from scipy.optimize import direct
from scipy.optimize import differential_evolution
from scipy.optimize import dual_annealing
from scipy.optimize import basinhopping
from spotpython.utils.init import fun_control_init, design_control_init, optimizer_control_init, surrogate_control_init
4 Sequential Parameter Optimization: Using scipy
Optimizers
As a default optimizer, spotpython
uses differential_evolution
from the scipy.optimize
package. Alternatively, any other optimizer from the scipy.optimize
package can be used. This chapter describes how different optimizers from the scipy optimize
package can be used on the surrogate. The optimization algorithms are available from https://docs.scipy.org/doc/scipy/reference/optimize.html
4.1 The Objective Function Branin
The spotpython
package provides several classes of objective functions. We will use an analytical objective function, i.e., a function that can be described by a (closed) formula. Here we will use the Branin function. The 2-dim Branin function is \[
y = a (x_2 - b x_1^2 + c x_1 - r) ^2 + s (1 - t) \cos(x_1) + s,
\] where values of \(a\), \(b\), \(c\), \(r\), \(s\) and \(t\) are: \(a = 1\), \(b = 5.1 / (4\pi^2)\), \(c = 5 / \pi\), \(r = 6\), \(s = 10\) and \(t = 1 / (8\pi)\).
It has three global minima: \(f(x) = 0.397887\) at \((-\pi, 12.275)\), \((\pi, 2.275)\), and \((9.42478, 2.475)\).
Input Domain: This function is usually evaluated on the square \(x_1 \in [-5, 10] \times x_2 \in [0, 15]\).
from spotpython.fun.objectivefunctions import Analytical
= np.array([-5,-0])
lower = np.array([10,15])
upper = Analytical(seed=123).fun_branin fun
4.2 The Optimizer
Differential Evolution (DE) from the scikit.optimize
package, see https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.differential_evolution.html#scipy.optimize.differential_evolution is the default optimizer for the search on the surrogate. Other optimiers that are available in spotpython
, see https://docs.scipy.org/doc/scipy/reference/optimize.html#global-optimization.
dual_annealing
direct
shgo
basinhopping
These optimizers can be selected as follows:
from scipy.optimize import differential_evolution
= differential_evolution optimizer
As noted above, we will use differential_evolution
. The optimizer can use 1000
evaluations. This value will be passed to the differential_evolution
method, which has the argument maxiter
(int). It defines the maximum number of generations over which the entire differential evolution population is evolved, see https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.differential_evolution.html#scipy.optimize.differential_evolution
Similar to the one-dimensional case, which is discussed in Section 7.5, we can use TensorBoard to monitor the progress of the optimization. We will use a similar code, only the prefix is different:
=fun_control_init(
fun_control= lower,
lower = upper,
upper = 20,
fun_evals = "04_DE_"
PREFIX
)=surrogate_control_init(
surrogate_control=len(lower)) n_theta
= spot.Spot(fun=fun,
spot_de =fun_control,
fun_control=surrogate_control)
surrogate_control spot_de.run()
spotpython tuning: 3.8004662117718677 [######----] 55.00%
spotpython tuning: 3.8004662117718677 [######----] 60.00%
spotpython tuning: 3.159024883515257 [######----] 65.00%
spotpython tuning: 3.133916697143885 [#######---] 70.00%
spotpython tuning: 2.8926749183116236 [########--] 75.00%
spotpython tuning: 0.4190219407803557 [########--] 80.00%
spotpython tuning: 0.401871440801683 [########--] 85.00%
spotpython tuning: 0.39926034519166187 [#########-] 90.00%
spotpython tuning: 0.39926034519166187 [##########] 95.00%
spotpython tuning: 0.39926034519166187 [##########] 100.00% Done...
<spotpython.spot.spot.Spot at 0x15538e240>
4.2.1 TensorBoard
If the prefix
argument in fun_control_init()
is not None
(as above, where the prefix
was set to 04_DE_
) , we can start TensorBoard in the background with the following command:
tensorboard --logdir="./runs"
We can access the TensorBoard web server with the following URL:
http://localhost:6006/
The TensorBoard plot illustrates how spotpython
can be used as a microscope for the internal mechanisms of the surrogate-based optimization process. Here, one important parameter, the learning rate \(\theta\) of the Kriging surrogate is plotted against the number of optimization steps.
4.3 Print the Results
spot_de.print_results()
min y: 0.39926034519166187
x0: 3.1509546500431656
x1: 2.298567899278217
[['x0', np.float64(3.1509546500431656)], ['x1', np.float64(2.298567899278217)]]
4.4 Show the Progress
=True) spot_de.plot_progress(log_y
spot_de.surrogate.plot()
4.5 Exercises
4.5.1 dual_annealing
- Describe the optimization algorithm, see scipy.optimize.dual_annealing.
- Use the algorithm as an optimizer on the surrogate.
We can run spotpython with the dual_annealing
optimizer as follows:
= spot.Spot(fun=fun,
spot_da =fun_control,
fun_control=dual_annealing,
optimizer=surrogate_control)
surrogate_control
spot_da.run()
spot_da.print_results()=True)
spot_da.plot_progress(log_y spot_da.surrogate.plot()
spotpython tuning: 3.800450053998908 [######----] 55.00%
spotpython tuning: 3.800450053998908 [######----] 60.00%
spotpython tuning: 3.158868878483762 [######----] 65.00%
spotpython tuning: 3.1342300163360486 [#######---] 70.00%
spotpython tuning: 2.893119335801825 [########--] 75.00%
spotpython tuning: 0.41578200014582833 [########--] 80.00%
spotpython tuning: 0.40201943256176875 [########--] 85.00%
spotpython tuning: 0.3991976498681318 [#########-] 90.00%
spotpython tuning: 0.3991976498681318 [##########] 95.00%
spotpython tuning: 0.3991976498681318 [##########] 100.00% Done...
min y: 0.3991976498681318
x0: 3.1505143339793134
x1: 2.2985181488234647
4.5.2 direct
- Describe the optimization algorithm
- Use the algorithm as an optimizer on the surrogate
We can run spotpython with the direct
optimizer as follows:
= spot.Spot(fun=fun,
spot_di =fun_control,
fun_control=direct,
optimizer=surrogate_control)
surrogate_control
spot_di.run()
spot_di.print_results()=True)
spot_di.plot_progress(log_y spot_di.surrogate.plot()
spotpython tuning: 3.812970247994418 [######----] 55.00%
spotpython tuning: 3.812970247994418 [######----] 60.00%
spotpython tuning: 3.162514679816068 [######----] 65.00%
spotpython tuning: 3.1189615135325983 [#######---] 70.00%
spotpython tuning: 2.6597698275013038 [########--] 75.00%
spotpython tuning: 0.3984917773445744 [########--] 80.00%
spotpython tuning: 0.3984917773445744 [########--] 85.00%
spotpython tuning: 0.3984917773445744 [#########-] 90.00%
spotpython tuning: 0.3984917773445744 [##########] 95.00%
spotpython tuning: 0.3984917773445744 [##########] 100.00% Done...
min y: 0.3984917773445744
x0: 3.137860082304525
x1: 2.3010973936899863
4.5.3 shgo
- Describe the optimization algorithm
- Use the algorithm as an optimizer on the surrogate
We can run spotpython with the direct
optimizer as follows:
= spot.Spot(fun=fun,
spot_sh =fun_control,
fun_control=shgo,
optimizer=surrogate_control)
surrogate_control
spot_sh.run()
spot_sh.print_results()=True)
spot_sh.plot_progress(log_y spot_sh.surrogate.plot()
spotpython tuning: 3.800455654373023 [######----] 55.00%
spotpython tuning: 3.800455654373023 [######----] 60.00%
spotpython tuning: 3.1589998607289376 [######----] 65.00%
spotpython tuning: 3.134192072951082 [#######---] 70.00%
spotpython tuning: 2.896589509267746 [########--] 75.00%
spotpython tuning: 0.433945961604028 [########--] 80.00%
spotpython tuning: 0.40260825513499476 [########--] 85.00%
spotpython tuning: 0.39928786197917887 [#########-] 90.00%
spotpython tuning: 0.39928786197917887 [##########] 95.00%
spotpython tuning: 0.39928786197917887 [##########] 100.00% Done...
min y: 0.39928786197917887
x0: 3.1517408652058885
x1: 2.2972000572204814
4.5.4 basinhopping
- Describe the optimization algorithm
- Use the algorithm as an optimizer on the surrogate
We can run spotpython with the direct
optimizer as follows:
= spot.Spot(fun=fun,
spot_bh =fun_control,
fun_control=basinhopping,
optimizer=surrogate_control)
surrogate_control
spot_bh.run()
spot_bh.print_results()=True)
spot_bh.plot_progress(log_y spot_bh.surrogate.plot()
spotpython tuning: 3.8004536838842267 [######----] 55.00%
spotpython tuning: 3.8004536838842267 [######----] 60.00%
spotpython tuning: 3.1590078408373294 [######----] 65.00%
spotpython tuning: 3.1342008291295436 [#######---] 70.00%
spotpython tuning: 2.8939639248159903 [########--] 75.00%
spotpython tuning: 0.4191913455845153 [########--] 80.00%
spotpython tuning: 0.4019076839214648 [########--] 85.00%
spotpython tuning: 0.3992176203377902 [#########-] 90.00%
spotpython tuning: 0.3992176203377902 [##########] 95.00%
spotpython tuning: 0.3992176203377902 [##########] 100.00% Done...
min y: 0.3992176203377902
x0: 3.150847266242321
x1: 2.2981098602302974
4.5.5 Performance Comparison
Compare the performance and run time of the 5 different optimizers:
differential_evolution
dual_annealing
direct
shgo
basinhopping
.
The Branin function has three global minima:
- \(f(x) = 0.397887\) at
- \((-\pi, 12.275)\),
- \((\pi, 2.275)\), and
- \((9.42478, 2.475)\).
- Which optima are found by the optimizers?
- Does the
seed
argument infun = Analytical(seed=123).fun_branin
change this behavior?
4.6 Jupyter Notebook
- The Jupyter-Notebook of this chapter is available on GitHub in the Hyperparameter-Tuning-Cookbook Repository