9  Isotropic and Anisotropic Kriging

This chapter illustrates the difference between isotropic and anisotropic Kriging models. The difference is illustrated with the help of the spotpython package. Isotropic Kriging models use the same theta value for every dimension. Anisotropic Kriging models use different theta values for each dimension.

9.1 Example: Isotropic Spot Surrogate and the 2-dim Sphere Function

import numpy as np
from math import inf
from spotpython.fun.objectivefunctions import analytical
from spotpython.spot import spot
from spotpython.utils.init import fun_control_init, surrogate_control_init
PREFIX="003"

9.1.1 The Objective Function: 2-dim Sphere

The spotpython package provides several classes of objective functions. We will use an analytical objective function, i.e., a function that can be described by a (closed) formula:

\[ f(x, y) = x^2 + y^2 \] The size of the lower bound vector determines the problem dimension. Here we will use np.array([-1, -1]), i.e., a two-dimensional function.

fun = analytical().fun_sphere
fun_control = fun_control_init(PREFIX=PREFIX,
                               lower = np.array([-1, -1]),
                               upper = np.array([1, 1]))

Although the default spot surrogate model is an isotropic Kriging model, we will explicitly set the n_theta parameter to a value of 1, so that the same theta value is used for both dimensions. This is done to illustrate the difference between isotropic and anisotropic Kriging models.

surrogate_control=surrogate_control_init(n_theta=1)
spot_2 = spot.Spot(fun=fun,
                   fun_control=fun_control,
                   surrogate_control=surrogate_control)

spot_2.run()
spotpython tuning: 2.217297132344163e-05 [#######---] 73.33% 
spotpython tuning: 2.217297132344163e-05 [########--] 80.00% 
spotpython tuning: 2.217297132344163e-05 [#########-] 86.67% 
spotpython tuning: 2.217297132344163e-05 [#########-] 93.33% 
spotpython tuning: 2.217297132344163e-05 [##########] 100.00% Done...

<spotpython.spot.spot.Spot at 0x16ae54a10>

9.1.2 Results

spot_2.print_results()
min y: 2.217297132344163e-05
x0: 0.001637745619980198
x1: 0.004414834154039912
[['x0', 0.001637745619980198], ['x1', 0.004414834154039912]]
spot_2.plot_progress(log_y=True)

9.2 Example With Anisotropic Kriging

As described in Section 9.1, the default parameter setting of spotpython’s Kriging surrogate uses the same theta value for every dimension. This is referred to as “using an isotropic kernel”. If different theta values are used for each dimension, then an anisotropic kernel is used. To enable anisotropic models in spotpython, the number of theta values should be larger than one. We can use surrogate_control=surrogate_control_init(n_theta=2) to enable this behavior (2 is the problem dimension).

surrogate_control = surrogate_control_init(n_theta=2)
spot_2_anisotropic = spot.Spot(fun=fun,
                    fun_control=fun_control,
                    surrogate_control=surrogate_control)
spot_2_anisotropic.run()
spotpython tuning: 1.5904060546935205e-05 [#######---] 73.33% 
spotpython tuning: 1.5904060546935205e-05 [########--] 80.00% 
spotpython tuning: 1.5904060546935205e-05 [#########-] 86.67% 
spotpython tuning: 1.5904060546935205e-05 [#########-] 93.33% 
spotpython tuning: 1.4886245843852055e-05 [##########] 100.00% Done...

<spotpython.spot.spot.Spot at 0x35b1a4800>

The search progress of the optimization with the anisotropic model can be visualized:

spot_2_anisotropic.plot_progress(log_y=True)

spot_2_anisotropic.print_results()
min y: 1.4886245843852055e-05
x0: -0.003850983818095356
x1: -0.00023700100552480994
[['x0', -0.003850983818095356], ['x1', -0.00023700100552480994]]
spot_2_anisotropic.surrogate.plot()

9.2.1 Taking a Look at the theta Values

9.2.1.1 theta Values from the spot Model

We can check, whether one or several theta values were used. The theta values from the surrogate can be printed as follows:

spot_2_anisotropic.surrogate.theta
[-0.32858908115378416, -0.13977864172679175]
  • Since the surrogate from the isotropic setting was stored as spot_2, we can also take a look at the theta value from this model:
spot_2.surrogate.theta
[-0.17881418596627952]

9.2.1.2 TensorBoard

Now we can start TensorBoard in the background with the following command:

tensorboard --logdir="./runs"

We can access the TensorBoard web server with the following URL:

http://localhost:6006/

The TensorBoard plot illustrates how spotpython can be used as a microscope for the internal mechanisms of the surrogate-based optimization process. Here, one important parameter, the learning rate \(\theta\) of the Kriging surrogate is plotted against the number of optimization steps.

TensorBoard visualization of the spotpython surrogate model.

9.3 Exercises

9.3.1 1. The Branin Function fun_branin

  • Describe the function.
    • The input dimension is 2. The search range is \(-5 \leq x_1 \leq 10\) and \(0 \leq x_2 \leq 15\).
  • Compare the results from spotpython run a) with isotropic and b) anisotropic surrogate models.
  • Modify the termination criterion: instead of the number of evaluations (which is specified via fun_evals), the time should be used as the termination criterion. This can be done as follows (max_time=1 specifies a run time of one minute):
from math import inf
fun_control = fun_control_init(
              fun_evals=inf,
              max_time=1)

9.3.2 2. The Two-dimensional Sin-Cos Function fun_sin_cos

  • Describe the function.
    • The input dimension is 2. The search range is \(-2\pi \leq x_1 \leq 2\pi\) and \(-2\pi \leq x_2 \leq 2\pi\).
  • Compare the results from spotpython run a) with isotropic and b) anisotropic surrogate models.
  • Modify the termination criterion (max_time instead of fun_evals) as described for fun_branin.

9.3.3 3. The Two-dimensional Runge Function fun_runge

  • Describe the function.
    • The input dimension is 2. The search range is \(-5 \leq x_1 \leq 5\) and \(-5 \leq x_2 \leq 5\).
  • Compare the results from spotpython run a) with isotropic and b) anisotropic surrogate models.
  • Modify the termination criterion (max_time instead of fun_evals) as described for fun_branin.

9.3.4 4. The Ten-dimensional Wing-Weight Function fun_wingwt

  • Describe the function.
    • The input dimension is 10. The search ranges are between 0 and 1 (values are mapped internally to their natural bounds).
  • Compare the results from spotpython run a) with isotropic and b) anisotropic surrogate models.
  • Modify the termination criterion (max_time instead of fun_evals) as described for fun_branin.

9.3.5 5. The Two-dimensional Rosenbrock Function fun_rosen

  • Describe the function.
    • The input dimension is 2. The search ranges are between -5 and 10.
  • Compare the results from spotpython run a) with isotropic and b) anisotropic surrogate models.
  • Modify the termination criterion (max_time instead of fun_evals) as described for fun_branin.

9.4 Selected Solutions

9.4.1 Solution to Exercise Section 9.3.5: The Two-dimensional Rosenbrock Function fun_rosen

9.4.1.1 The Two Dimensional fun_rosen: The Isotropic Case

import numpy as np
from spotpython.fun.objectivefunctions import analytical
from spotpython.utils.init import fun_control_init, surrogate_control_init
from spotpython.spot import spot

The spotpython package provides several classes of objective functions. We will use the fun_rosen in the analytical class [SOURCE].

fun_rosen = analytical().fun_rosen

Here we will use problem dimension \(k=2\), which can be specified by the lower bound arrays. The size of the lower bound array determines the problem dimension.

The prefix is set to "ROSEN" to distinguish the results from the one-dimensional case. Again, TensorBoard can be used to monitor the progress of the optimization.

fun_control = fun_control_init(
              PREFIX="ROSEN",
              lower = np.array([-5, -5]),
              upper = np.array([10, 10]),
              show_progress=True)
surrogate_control = surrogate_control_init(n_theta=1)
spot_rosen = spot.Spot(fun=fun_rosen,
                  fun_control=fun_control,
                  surrogate_control=surrogate_control)
spot_rosen.run()
spotpython tuning: 52.876358299143206 [#######---] 73.33% 
spotpython tuning: 52.36204755139067 [########--] 80.00% 
spotpython tuning: 52.36204755139067 [#########-] 86.67% 
spotpython tuning: 43.44294819789333 [#########-] 93.33% 
spotpython tuning: 12.275982031887748 [##########] 100.00% Done...

<spotpython.spot.spot.Spot at 0x35b5dad20>
Note

Now we can start TensorBoard in the background with the following command:

tensorboard --logdir="./runs"

and can access the TensorBoard web server with the following URL:

http://localhost:6006/
9.4.1.1.1 Results
_ = spot_rosen.print_results()
min y: 12.275982031887748
x0: -2.3708319530278255
x1: 5.923081469551754
spot_rosen.plot_progress()

9.4.1.1.2 A Contour Plot

We can select two dimensions, say \(i=0\) and \(j=1\), and generate a contour plot as follows.

min_z = None
max_z = None
spot_rosen.plot_contour(i=0, j=1, min_z=min_z, max_z=max_z)

  • The variable importance cannot be calculated, because only one theta value was used.
9.4.1.1.3 TensorBoard

TBD

9.4.1.2 The Two Dimensional fun_rosen: The Anisotropic Case

import numpy as np
from spotpython.fun.objectivefunctions import analytical
from spotpython.utils.init import fun_control_init, surrogate_control_init
from spotpython.spot import spot

The spotpython package provides several classes of objective functions. We will use the fun_rosen in the analytical class [SOURCE].

fun_rosen = analytical().fun_rosen

Here we will use problem dimension \(k=2\), which can be specified by the lower bound arrays. The size of the lower bound array determines the problem dimension.

We can also add interpreable labels to the dimensions, which will be used in the plots.

fun_control = fun_control_init(
              PREFIX="ROSEN",
              lower = np.array([-5, -5]),
              upper = np.array([10, 10]),
              show_progress=True)
surrogate_control = surrogate_control_init(n_theta=2)
spot_rosen = spot.Spot(fun=fun_rosen,
                  fun_control=fun_control,
                  surrogate_control=surrogate_control)
spot_rosen.run()
spotpython tuning: 90.78805160669465 [#######---] 73.33% 
spotpython tuning: 1.0171930962259077 [########--] 80.00% 
spotpython tuning: 1.0171930962259077 [#########-] 86.67% 
spotpython tuning: 1.0171930962259077 [#########-] 93.33% 
spotpython tuning: 1.0171930962259077 [##########] 100.00% Done...

<spotpython.spot.spot.Spot at 0x35b6ba0c0>
Note

Now we can start TensorBoard in the background with the following command:

tensorboard --logdir="./runs"

and can access the TensorBoard web server with the following URL:

http://localhost:6006/
9.4.1.2.1 Results
_ = spot_rosen.print_results()
min y: 1.0171930962259077
x0: 0.002781302621691875
x1: -0.047687091984116046
spot_rosen.plot_progress()

9.4.1.2.2 A Contour Plot

We can select two dimensions, say \(i=0\) and \(j=1\), and generate a contour plot as follows.

min_z = None
max_z = None
spot_rosen.plot_contour(i=0, j=1, min_z=min_z, max_z=max_z)

  • The variable importance can be calculated as follows:
_ = spot_rosen.print_importance()
x0:  100.0
x1:  2.2275258776369804
spot_rosen.plot_importance()

9.4.1.2.3 TensorBoard

TBD

9.5 Jupyter Notebook

Note