9  Isotropic and Anisotropic Kriging

This chapter illustrates the difference between isotropic and anisotropic Kriging models. The difference is illustrated with the help of the spotpython package. Isotropic Kriging models use the same theta value for every dimension. Anisotropic Kriging models use different theta values for each dimension.

9.1 Example: Isotropic Spot Surrogate and the 2-dim Sphere Function

import numpy as np
from math import inf
from spotpython.fun.objectivefunctions import Analytical
from spotpython.spot import spot
from spotpython.utils.init import fun_control_init, surrogate_control_init
PREFIX="003"

9.1.1 The Objective Function: 2-dim Sphere

The spotpython package provides several classes of objective functions. We will use an analytical objective function, i.e., a function that can be described by a (closed) formula:

\[ f(x, y) = x^2 + y^2 \] The size of the lower bound vector determines the problem dimension. Here we will use np.array([-1, -1]), i.e., a two-dimensional function.

fun = Analytical().fun_sphere
fun_control = fun_control_init(PREFIX=PREFIX,
                               lower = np.array([-1, -1]),
                               upper = np.array([1, 1]))

Although the default spot surrogate model is an isotropic Kriging model, we will explicitly set the n_theta parameter to a value of 1, so that the same theta value is used for both dimensions. This is done to illustrate the difference between isotropic and anisotropic Kriging models.

surrogate_control=surrogate_control_init(n_theta=1)
spot_2 = spot.Spot(fun=fun,
                   fun_control=fun_control,
                   surrogate_control=surrogate_control)

spot_2.run()
spotpython tuning: 1.9577055446455904e-05 [#######---] 73.33% 
spotpython tuning: 1.9577055446455904e-05 [########--] 80.00% 
spotpython tuning: 1.9577055446455904e-05 [#########-] 86.67% 
spotpython tuning: 1.9577055446455904e-05 [#########-] 93.33% 
spotpython tuning: 1.9577055446455904e-05 [##########] 100.00% Done...

<spotpython.spot.spot.Spot at 0x152ef3c80>

9.1.2 Results

spot_2.print_results()
min y: 1.9577055446455904e-05
x0: 0.00171841680372826
x1: 0.0040772661349389805
[['x0', np.float64(0.00171841680372826)],
 ['x1', np.float64(0.0040772661349389805)]]
spot_2.plot_progress(log_y=True)

9.2 Example With Anisotropic Kriging

As described in Section 9.1, the default parameter setting of spotpython’s Kriging surrogate uses the same theta value for every dimension. This is referred to as “using an isotropic kernel”. If different theta values are used for each dimension, then an anisotropic kernel is used. To enable anisotropic models in spotpython, the number of theta values should be larger than one. We can use surrogate_control=surrogate_control_init(n_theta=2) to enable this behavior (2 is the problem dimension).

surrogate_control = surrogate_control_init(n_theta=2)
spot_2_anisotropic = spot.Spot(fun=fun,
                    fun_control=fun_control,
                    surrogate_control=surrogate_control)
spot_2_anisotropic.run()
spotpython tuning: 1.5904060546935205e-05 [#######---] 73.33% 
spotpython tuning: 1.5904060546935205e-05 [########--] 80.00% 
spotpython tuning: 1.5904060546935205e-05 [#########-] 86.67% 
spotpython tuning: 1.5904060546935205e-05 [#########-] 93.33% 
spotpython tuning: 1.2084513018724136e-05 [##########] 100.00% Done...

<spotpython.spot.spot.Spot at 0x153e5faa0>

The search progress of the optimization with the anisotropic model can be visualized:

spot_2_anisotropic.plot_progress(log_y=True)

spot_2_anisotropic.print_results()
min y: 1.2084513018724136e-05
x0: -0.003294464459178357
x1: -0.0011095120305498227
[['x0', np.float64(-0.003294464459178357)],
 ['x1', np.float64(-0.0011095120305498227)]]
spot_2_anisotropic.surrogate.plot()

9.2.1 Taking a Look at the theta Values

9.2.1.1 theta Values from the spot Model

We can check, whether one or several theta values were used. The theta values from the surrogate can be printed as follows:

spot_2_anisotropic.surrogate.theta
[np.float64(-0.3285891031238254), np.float64(-0.13977864690749586)]
  • Since the surrogate from the isotropic setting was stored as spot_2, we can also take a look at the theta value from this model:
spot_2.surrogate.theta
[np.float64(-0.1652222808345842)]

9.2.1.2 TensorBoard

Now we can start TensorBoard in the background with the following command:

tensorboard --logdir="./runs"

We can access the TensorBoard web server with the following URL:

http://localhost:6006/

The TensorBoard plot illustrates how spotpython can be used as a microscope for the internal mechanisms of the surrogate-based optimization process. Here, one important parameter, the learning rate \(\theta\) of the Kriging surrogate is plotted against the number of optimization steps.

TensorBoard visualization of the spotpython surrogate model.

9.3 Exercises

9.3.1 1. The Branin Function fun_branin

  • Describe the function.
    • The input dimension is 2. The search range is \(-5 \leq x_1 \leq 10\) and \(0 \leq x_2 \leq 15\).
  • Compare the results from spotpython run a) with isotropic and b) anisotropic surrogate models.
  • Modify the termination criterion: instead of the number of evaluations (which is specified via fun_evals), the time should be used as the termination criterion. This can be done as follows (max_time=1 specifies a run time of one minute):
from math import inf
fun_control = fun_control_init(
              fun_evals=inf,
              max_time=1)

9.3.2 2. The Two-dimensional Sin-Cos Function fun_sin_cos

  • Describe the function.
    • The input dimension is 2. The search range is \(-2\pi \leq x_1 \leq 2\pi\) and \(-2\pi \leq x_2 \leq 2\pi\).
  • Compare the results from spotpython run a) with isotropic and b) anisotropic surrogate models.
  • Modify the termination criterion (max_time instead of fun_evals) as described for fun_branin.

9.3.3 3. The Two-dimensional Runge Function fun_runge

  • Describe the function.
    • The input dimension is 2. The search range is \(-5 \leq x_1 \leq 5\) and \(-5 \leq x_2 \leq 5\).
  • Compare the results from spotpython run a) with isotropic and b) anisotropic surrogate models.
  • Modify the termination criterion (max_time instead of fun_evals) as described for fun_branin.

9.3.4 4. The Ten-dimensional Wing-Weight Function fun_wingwt

  • Describe the function.
    • The input dimension is 10. The search ranges are between 0 and 1 (values are mapped internally to their natural bounds).
  • Compare the results from spotpython run a) with isotropic and b) anisotropic surrogate models.
  • Modify the termination criterion (max_time instead of fun_evals) as described for fun_branin.

9.3.5 5. The Two-dimensional Rosenbrock Function fun_rosen

  • Describe the function.
    • The input dimension is 2. The search ranges are between -5 and 10.
  • Compare the results from spotpython run a) with isotropic and b) anisotropic surrogate models.
  • Modify the termination criterion (max_time instead of fun_evals) as described for fun_branin.

9.4 Selected Solutions

9.4.1 Solution to Exercise Section 9.3.5: The Two-dimensional Rosenbrock Function fun_rosen

9.4.1.1 The Two Dimensional fun_rosen: The Isotropic Case

import numpy as np
from spotpython.fun.objectivefunctions import Analytical
from spotpython.utils.init import fun_control_init, surrogate_control_init
from spotpython.spot import spot

The spotpython package provides several classes of objective functions. We will use the fun_rosen in the analytical class [SOURCE].

fun_rosen = Analytical().fun_rosen

Here we will use problem dimension \(k=2\), which can be specified by the lower bound arrays. The size of the lower bound array determines the problem dimension.

The prefix is set to "ROSEN" to distinguish the results from the one-dimensional case. Again, TensorBoard can be used to monitor the progress of the optimization.

fun_control = fun_control_init(
              PREFIX="ROSEN",
              lower = np.array([-5, -5]),
              upper = np.array([10, 10]),
              show_progress=True)
surrogate_control = surrogate_control_init(n_theta=1)
spot_rosen = spot.Spot(fun=fun_rosen,
                  fun_control=fun_control,
                  surrogate_control=surrogate_control)
spot_rosen.run()
spotpython tuning: 52.87634888275474 [#######---] 73.33% 
spotpython tuning: 52.36206921045742 [########--] 80.00% 
spotpython tuning: 52.36206921045742 [#########-] 86.67% 
spotpython tuning: 43.44263277019559 [#########-] 93.33% 
spotpython tuning: 12.275794686387082 [##########] 100.00% Done...

<spotpython.spot.spot.Spot at 0x156a4b9e0>
Note

Now we can start TensorBoard in the background with the following command:

tensorboard --logdir="./runs"

and can access the TensorBoard web server with the following URL:

http://localhost:6006/
9.4.1.1.1 Results
_ = spot_rosen.print_results()
min y: 12.275794686387082
x0: -2.3708433337788763
x1: 5.923091744213865
spot_rosen.plot_progress()

9.4.1.1.2 A Contour Plot

We can select two dimensions, say \(i=0\) and \(j=1\), and generate a contour plot as follows.

min_z = None
max_z = None
spot_rosen.plot_contour(i=0, j=1, min_z=min_z, max_z=max_z)

  • The variable importance cannot be calculated, because only one theta value was used.
9.4.1.1.3 TensorBoard

TBD

9.4.1.2 The Two Dimensional fun_rosen: The Anisotropic Case

import numpy as np
from spotpython.fun.objectivefunctions import Analytical
from spotpython.utils.init import fun_control_init, surrogate_control_init
from spotpython.spot import spot

The spotpython package provides several classes of objective functions. We will use the fun_rosen in the analytical class [SOURCE].

fun_rosen = Analytical().fun_rosen

Here we will use problem dimension \(k=2\), which can be specified by the lower bound arrays. The size of the lower bound array determines the problem dimension.

We can also add interpreable labels to the dimensions, which will be used in the plots.

fun_control = fun_control_init(
              PREFIX="ROSEN",
              lower = np.array([-5, -5]),
              upper = np.array([10, 10]),
              show_progress=True)
surrogate_control = surrogate_control_init(n_theta=2)
spot_rosen = spot.Spot(fun=fun_rosen,
                  fun_control=fun_control,
                  surrogate_control=surrogate_control)
spot_rosen.run()
spotpython tuning: 90.78737194937058 [#######---] 73.33% 
spotpython tuning: 1.0172240416576994 [########--] 80.00% 
spotpython tuning: 1.0172240416576994 [#########-] 86.67% 
spotpython tuning: 1.0172240416576994 [#########-] 93.33% 
spotpython tuning: 1.0172240416576994 [##########] 100.00% Done...

<spotpython.spot.spot.Spot at 0x1555b80b0>
Note

Now we can start TensorBoard in the background with the following command:

tensorboard --logdir="./runs"

and can access the TensorBoard web server with the following URL:

http://localhost:6006/
9.4.1.2.1 Results
_ = spot_rosen.print_results()
min y: 1.0172240416576994
x0: 0.00278369658649557
x1: -0.04772451042254187
spot_rosen.plot_progress()

9.4.1.2.2 A Contour Plot

We can select two dimensions, say \(i=0\) and \(j=1\), and generate a contour plot as follows.

min_z = None
max_z = None
spot_rosen.plot_contour(i=0, j=1, min_z=min_z, max_z=max_z)

  • The variable importance can be calculated as follows:
_ = spot_rosen.print_importance()
x0:  100.00000000000001
x1:  2.227560396746198
spot_rosen.plot_importance()

9.4.1.2.3 TensorBoard

TBD

9.5 Jupyter Notebook

Note