import numpy as np
from math import inf
from spotpython.fun.objectivefunctions import analytical
from spotpython.spot import spot
from spotpython.utils.init import fun_control_init, surrogate_control_init
="003" PREFIX
9 Isotropic and Anisotropic Kriging
This chapter illustrates the difference between isotropic and anisotropic Kriging models. The difference is illustrated with the help of the spotpython
package. Isotropic Kriging models use the same theta
value for every dimension. Anisotropic Kriging models use different theta
values for each dimension.
9.1 Example: Isotropic Spot
Surrogate and the 2-dim Sphere Function
9.1.1 The Objective Function: 2-dim Sphere
The spotpython
package provides several classes of objective functions. We will use an analytical objective function, i.e., a function that can be described by a (closed) formula:
\[
f(x, y) = x^2 + y^2
\] The size of the lower
bound vector determines the problem dimension. Here we will use np.array([-1, -1])
, i.e., a two-dimensional function.
= analytical().fun_sphere
fun = fun_control_init(PREFIX=PREFIX,
fun_control = np.array([-1, -1]),
lower = np.array([1, 1])) upper
Although the default spot
surrogate model is an isotropic Kriging model, we will explicitly set the n_theta
parameter to a value of 1
, so that the same theta value is used for both dimensions. This is done to illustrate the difference between isotropic and anisotropic Kriging models.
=surrogate_control_init(n_theta=1) surrogate_control
= spot.Spot(fun=fun,
spot_2 =fun_control,
fun_control=surrogate_control)
surrogate_control
spot_2.run()
spotpython tuning: 2.217297132344163e-05 [#######---] 73.33%
spotpython tuning: 2.217297132344163e-05 [########--] 80.00%
spotpython tuning: 2.217297132344163e-05 [#########-] 86.67%
spotpython tuning: 2.217297132344163e-05 [#########-] 93.33%
spotpython tuning: 2.217297132344163e-05 [##########] 100.00% Done...
<spotpython.spot.spot.Spot at 0x34abd9130>
9.1.2 Results
spot_2.print_results()
min y: 2.217297132344163e-05
x0: 0.001637745619980198
x1: 0.004414834154039912
[['x0', 0.001637745619980198], ['x1', 0.004414834154039912]]
=True) spot_2.plot_progress(log_y
9.2 Example With Anisotropic Kriging
As described in Section 9.1, the default parameter setting of spotpython
’s Kriging surrogate uses the same theta
value for every dimension. This is referred to as “using an isotropic kernel”. If different theta
values are used for each dimension, then an anisotropic kernel is used. To enable anisotropic models in spotpython
, the number of theta
values should be larger than one. We can use surrogate_control=surrogate_control_init(n_theta=2)
to enable this behavior (2
is the problem dimension).
= surrogate_control_init(n_theta=2)
surrogate_control = spot.Spot(fun=fun,
spot_2_anisotropic =fun_control,
fun_control=surrogate_control)
surrogate_control spot_2_anisotropic.run()
spotpython tuning: 1.5904060546935205e-05 [#######---] 73.33%
spotpython tuning: 1.5904060546935205e-05 [########--] 80.00%
spotpython tuning: 1.5904060546935205e-05 [#########-] 86.67%
spotpython tuning: 1.5904060546935205e-05 [#########-] 93.33%
spotpython tuning: 1.4886245843852055e-05 [##########] 100.00% Done...
<spotpython.spot.spot.Spot at 0x34b1eccb0>
The search progress of the optimization with the anisotropic model can be visualized:
=True) spot_2_anisotropic.plot_progress(log_y
spot_2_anisotropic.print_results()
min y: 1.4886245843852055e-05
x0: -0.003850983818095356
x1: -0.00023700100552480994
[['x0', -0.003850983818095356], ['x1', -0.00023700100552480994]]
spot_2_anisotropic.surrogate.plot()
9.2.1 Taking a Look at the theta
Values
9.2.1.1 theta
Values from the spot
Model
We can check, whether one or several theta
values were used. The theta
values from the surrogate can be printed as follows:
spot_2_anisotropic.surrogate.theta
array([-0.32858908, -0.13977864])
- Since the surrogate from the isotropic setting was stored as
spot_2
, we can also take a look at thetheta
value from this model:
spot_2.surrogate.theta
array([-0.17881419])
9.2.1.2 TensorBoard
Now we can start TensorBoard in the background with the following command:
tensorboard --logdir="./runs"
We can access the TensorBoard web server with the following URL:
http://localhost:6006/
The TensorBoard plot illustrates how spotpython
can be used as a microscope for the internal mechanisms of the surrogate-based optimization process. Here, one important parameter, the learning rate \(\theta\) of the Kriging surrogate is plotted against the number of optimization steps.
9.3 Exercises
9.3.1 1. The Branin Function fun_branin
- Describe the function.
- The input dimension is
2
. The search range is \(-5 \leq x_1 \leq 10\) and \(0 \leq x_2 \leq 15\).
- The input dimension is
- Compare the results from
spotpython
run a) with isotropic and b) anisotropic surrogate models. - Modify the termination criterion: instead of the number of evaluations (which is specified via
fun_evals
), the time should be used as the termination criterion. This can be done as follows (max_time=1
specifies a run time of one minute):
from math import inf
= fun_control_init(
fun_control =inf,
fun_evals=1) max_time
9.3.2 2. The Two-dimensional Sin-Cos Function fun_sin_cos
- Describe the function.
- The input dimension is
2
. The search range is \(-2\pi \leq x_1 \leq 2\pi\) and \(-2\pi \leq x_2 \leq 2\pi\).
- The input dimension is
- Compare the results from
spotpython
run a) with isotropic and b) anisotropic surrogate models. - Modify the termination criterion (
max_time
instead offun_evals
) as described forfun_branin
.
9.3.3 3. The Two-dimensional Runge Function fun_runge
- Describe the function.
- The input dimension is
2
. The search range is \(-5 \leq x_1 \leq 5\) and \(-5 \leq x_2 \leq 5\).
- The input dimension is
- Compare the results from
spotpython
run a) with isotropic and b) anisotropic surrogate models. - Modify the termination criterion (
max_time
instead offun_evals
) as described forfun_branin
.
9.3.4 4. The Ten-dimensional Wing-Weight Function fun_wingwt
- Describe the function.
- The input dimension is
10
. The search ranges are between 0 and 1 (values are mapped internally to their natural bounds).
- The input dimension is
- Compare the results from
spotpython
run a) with isotropic and b) anisotropic surrogate models. - Modify the termination criterion (
max_time
instead offun_evals
) as described forfun_branin
.
9.3.5 5. The Two-dimensional Rosenbrock Function fun_rosen
- Describe the function.
- The input dimension is
2
. The search ranges are between -5 and 10.
- The input dimension is
- Compare the results from
spotpython
run a) with isotropic and b) anisotropic surrogate models. - Modify the termination criterion (
max_time
instead offun_evals
) as described forfun_branin
.
9.4 Selected Solutions
9.4.1 Solution to Exercise Section 9.3.5: The Two-dimensional Rosenbrock Function fun_rosen
9.4.1.1 The Two Dimensional fun_rosen
: The Isotropic Case
import numpy as np
from spotpython.fun.objectivefunctions import analytical
from spotpython.utils.init import fun_control_init, surrogate_control_init
from spotpython.spot import spot
The spotpython
package provides several classes of objective functions. We will use the fun_rosen
in the analytical
class [SOURCE].
= analytical().fun_rosen fun_rosen
Here we will use problem dimension \(k=2\), which can be specified by the lower
bound arrays. The size of the lower
bound array determines the problem dimension.
The prefix is set to "ROSEN"
to distinguish the results from the one-dimensional case. Again, TensorBoard can be used to monitor the progress of the optimization.
= fun_control_init(
fun_control ="ROSEN",
PREFIX= np.array([-5, -5]),
lower = np.array([10, 10]),
upper =True)
show_progress= surrogate_control_init(n_theta=1)
surrogate_control = spot.Spot(fun=fun_rosen,
spot_rosen =fun_control,
fun_control=surrogate_control)
surrogate_control spot_rosen.run()
spotpython tuning: 52.876358299143206 [#######---] 73.33%
spotpython tuning: 52.36204755139067 [########--] 80.00%
spotpython tuning: 52.36204755139067 [#########-] 86.67%
spotpython tuning: 43.44294819789333 [#########-] 93.33%
spotpython tuning: 12.275982031887748 [##########] 100.00% Done...
<spotpython.spot.spot.Spot at 0x34b64bf80>
Now we can start TensorBoard in the background with the following command:
tensorboard --logdir="./runs"
and can access the TensorBoard web server with the following URL:
http://localhost:6006/
9.4.1.1.1 Results
= spot_rosen.print_results() _
min y: 12.275982031887748
x0: -2.3708319530278255
x1: 5.923081469551754
spot_rosen.plot_progress()
9.4.1.1.2 A Contour Plot
We can select two dimensions, say \(i=0\) and \(j=1\), and generate a contour plot as follows.
= None
min_z = None
max_z =0, j=1, min_z=min_z, max_z=max_z) spot_rosen.plot_contour(i
- The variable importance cannot be calculated, because only one
theta
value was used.
9.4.1.1.3 TensorBoard
TBD
9.4.1.2 The Two Dimensional fun_rosen
: The Anisotropic Case
import numpy as np
from spotpython.fun.objectivefunctions import analytical
from spotpython.utils.init import fun_control_init, surrogate_control_init
from spotpython.spot import spot
The spotpython
package provides several classes of objective functions. We will use the fun_rosen
in the analytical
class [SOURCE].
= analytical().fun_rosen fun_rosen
Here we will use problem dimension \(k=2\), which can be specified by the lower
bound arrays. The size of the lower
bound array determines the problem dimension.
We can also add interpreable labels to the dimensions, which will be used in the plots.
= fun_control_init(
fun_control ="ROSEN",
PREFIX= np.array([-5, -5]),
lower = np.array([10, 10]),
upper =True)
show_progress= surrogate_control_init(n_theta=2)
surrogate_control = spot.Spot(fun=fun_rosen,
spot_rosen =fun_control,
fun_control=surrogate_control)
surrogate_control spot_rosen.run()
spotpython tuning: 90.78805160669465 [#######---] 73.33%
spotpython tuning: 1.0171930962259077 [########--] 80.00%
spotpython tuning: 1.0171930962259077 [#########-] 86.67%
spotpython tuning: 1.0171930962259077 [#########-] 93.33%
spotpython tuning: 1.0171930962259077 [##########] 100.00% Done...
<spotpython.spot.spot.Spot at 0x1413172f0>
Now we can start TensorBoard in the background with the following command:
tensorboard --logdir="./runs"
and can access the TensorBoard web server with the following URL:
http://localhost:6006/
9.4.1.2.1 Results
= spot_rosen.print_results() _
min y: 1.0171930962259077
x0: 0.002781302621691875
x1: -0.047687091984116046
spot_rosen.plot_progress()
9.4.1.2.2 A Contour Plot
We can select two dimensions, say \(i=0\) and \(j=1\), and generate a contour plot as follows.
= None
min_z = None
max_z =0, j=1, min_z=min_z, max_z=max_z) spot_rosen.plot_contour(i
- The variable importance can be calculated as follows:
= spot_rosen.print_importance() _
x0: 100.0
x1: 2.2275258776369804
spot_rosen.plot_importance()
9.4.1.2.3 TensorBoard
TBD
9.5 Jupyter Notebook
- The Jupyter-Notebook of this lecture is available on GitHub in the Hyperparameter-Tuning-Cookbook Repository