[[13]{.chapter-number}  [Multi-dimensional Functions]{.chapter-title}]{#sec-multi-dim .quarto-section-identifier}

doi:10.48550/arXiv.2307.10262

13 Multi-dimensional Functions

This chapter illustrates how high-dimensional functions can be optimized and analyzed. For reasons of illustration, we will use the three-dimensional Sphere function, which is a simple and well-known function. The problem dimension is \(k=3\), but can be easily adapted to other, higher dimensions.

import numpy as np
from spotpython.fun.objectivefunctions import Analytical
from spotpython.utils.init import fun_control_init, surrogate_control_init, design_control_init
from spotpython.spot import Spot

13.1 The Objective Function: 3-dim Sphere

The spotpython package provides several classes of objective functions. We will use an analytical objective function, i.e., a function that can be described by a (closed) formula: \[ f(x) = \sum_i^k x_i^2. \]

The Sphere function is continuous, convex and unimodal. The plot shows its two-dimensional form. The global minimum is \[ f(x) = 0, \text{at } x = (0,0, \ldots, 0). \]

It is available as fun_sphere in the Analytical class [SOURCE].

fun = Analytical().fun_sphere

Here we will use problem dimension \(k=3\), which can be specified by the lower bound arrays. The size of the lower bound array determines the problem dimension. If we select -1.0 * np.ones(3), a three-dimensional function is created.

In contrast to the one-dimensional case (Section 12.25), where only one theta value was used, we will use three different theta values (one for each dimension). This is done automatically, because the setting isotropic=False is the default in the surrogate_control. As default, spotpython uses separate theta values for each problem dimension. More specifically, if isotropic is set to True, only one theta value is used for all dimensions. If isotropic is set to False, then the \(k\) theta values are used, where \(k\) is the problem dimension. The meaning of “isotropic” is explained in @#sec-iso-aniso-kriging.

The prefix is set to "03" to distinguish the results from the one-dimensional case. Again, TensorBoard can be used to monitor the progress of the optimization.

We can also add interpretable labels to the dimensions, which will be used in the plots. Therefore, we set var_name=["Pressure", "Temp", "Lambda"] instead of the default var_name=None, which would result in the labels x_0, x_1, and x_2.

fun_control = fun_control_init(
              PREFIX="03",
              lower = -1.0*np.ones(3),
              upper = np.ones(3),
              var_name=["Pressure", "Temp", "Lambda"],
              TENSORBOARD_CLEAN=True,
              tensorboard_log=True)
surrogate_control = surrogate_control_init()
spot_3 = Spot(fun=fun,
                  fun_control=fun_control,
                  surrogate_control=surrogate_control)
spot_3.run()

Moving TENSORBOARD_PATH: runs/ to TENSORBOARD_PATH_OLD: runs_OLD/runs_2025_10_19_20_30_00_0
Created spot_tensorboard_path: runs/spot_logs/03_maans08_2025-10-19_20-30-00 for SummaryWriter()
Anisotropic model: n_theta set to 3
Anisotropic model: n_theta set to 3
spotpython tuning: 0.03443412209990359 [#######---] 73.33% 
Anisotropic model: n_theta set to 3
spotpython tuning: 0.03134246099772966 [########--] 80.00% 
Anisotropic model: n_theta set to 3
spotpython tuning: 0.0009648021629613051 [#########-] 86.67% 
Anisotropic model: n_theta set to 3
spotpython tuning: 8.300667639564488e-05 [#########-] 93.33% 
Anisotropic model: n_theta set to 3
spotpython tuning: 3.067126343714693e-05 [##########] 100.00% Done...

Experiment saved to 03_res.pkl

<spotpython.spot.spot.Spot at 0x10872fe00>

Note

Now we can start TensorBoard in the background with the following command:

tensorboard --logdir="./runs"

and can access the TensorBoard web server with the following URL:

http://localhost:6006/

13.1.1 Results

13.1.1.1 Best Objective Function Values

The best objective function value and its corresponding input values are printed as follows:

_ = spot_3.print_results()

min y: 3.067126343714693e-05
Pressure: 0.0033489613527794017
Temp: 0.0008610649183191578
Lambda: 0.004326001444888445

The method plot_progress() plots current and best found solutions versus the number of iterations as shown in Figure 13.1.

spot_3.plot_progress()

Figure 13.1: Progress of the optimization process for the 3-dim Sphere function. The initial design points are shown in black, whereas the points that were found by the search on the surrogate are plotted in red.

13.1.1.2 A Contour Plot

We can select two dimensions, say \(i=0\) and \(j=1\), and generate a contour plot as follows. Note, we have specified identical min_z and max_z values to generate comparable plots.

spot_3.plot_contour(i=0, j=1, min_z=0, max_z=2.25)

In a similar manner, we can plot dimension \(i=0\) and \(j=2\):

spot_3.plot_contour(i=0, j=2, min_z=0, max_z=2.25)

The final combination is \(i=1\) and \(j=2\):

spot_3.plot_contour(i=1, j=2, min_z=0, max_z=2.25)

The three plots look very similar, because the fun_sphere is symmetric.
This can also be seen from the variable importance:

_ = spot_3.print_importance()

Pressure:  95.86267236204144
Temp:  100.0
Lambda:  89.82098231817146

spot_3.plot_importance()

13.1.2 TensorBoard

TensorBoard visualization of the spotpython process. Objective function values plotted against wall time.

The second TensorBoard visualization shows the input values, i.e., \(x_0, \ldots, x_2\), plotted against the wall time. TensorBoard visualization of the spotpython process.

The third TensorBoard plot illustrates how spotpython can be used as a microscope for the internal mechanisms of the surrogate-based optimization process. Here, one important parameter, the learning rate \(\theta\) of the Kriging surrogate is plotted against the number of optimization steps.

TensorBoard visualization of the spotpython surrogate model.

13.1.3 Conclusion

Based on this quick analysis, we can conclude that all three dimensions are equally important (as expected, because the Analytical function is known).

13.2 Exercises

Exercise 13.1 (The Three Dimensional fun_cubed) The spotpython package provides several classes of objective functions.

We will use the fun_cubed in the Analytical class [SOURCE]. The input dimension is 3. The search range is \(-1 \leq x \leq 1\) for all dimensions.

Tasks: * Generate contour plots * Calculate the variable importance. * Discuss the variable importance: * Are all variables equally important? * If not: * Which is the most important variable? * Which is the least important variable?

Exercise 13.2 (The Ten Dimensional fun_wing_wt)

The input dimension is 10. The search range is \(0 \leq x \leq 1\) for all dimensions.
Calculate the variable importance.
Discuss the variable importance:
- Are all variables equally important?
- If not:
  - Which is the most important variable?
  - Which is the least important variable?
- Generate contour plots for the three most important variables. Do they confirm your selection?

Exercise 13.3 (The Three Dimensional fun_runge)

The input dimension is 3. The search range is \(-5 \leq x \leq 5\) for all dimensions.
Generate contour plots
Calculate the variable importance.
Discuss the variable importance:
- Are all variables equally important?
- If not:
  - Which is the most important variable?
  - Which is the least important variable?

Exercise 13.4 (The Three Dimensional fun_linear)

The input dimension is 3. The search range is \(-5 \leq x \leq 5\) for all dimensions.
Generate contour plots
Calculate the variable importance.
Discuss the variable importance:
- Are all variables equally important?
- If not:
  - Which is the most important variable?
  - Which is the least important variable?

Exercise 13.5 (The Two Dimensional Rosenbrock Function fun_rosen)

The input dimension is 2. The search range is \(-5 \leq x \leq 10\) for all dimensions.
See Rosenbrock function and Rosenbrock Function for details.
Generate contour plots
Calculate the variable importance.
Discuss the variable importance:
- Are all variables equally important?
- If not:
  - Which is the most important variable?
  - Which is the least important variable?

13.3 Selected Solutions

Solution 13.1 (Solution to Exercise 13.1: The Three-dimensional Cubed Function fun_cubed). We instanciate the fun_cubed function from the Analytical class.

from spotpython.fun.objectivefunctions import Analytical
fun_cubed = Analytical().fun_cubed

Here we will use problem dimension \(k=3\), which can be specified by the lower bound arrays. The size of the lower bound array determines the problem dimension. If we select -1.0 * np.ones(3), a three-dimensional function is created.
In contrast to the one-dimensional case, where only one theta value was used, we will use three different theta values (one for each dimension).
The prefix is set to "03" to distinguish the results from the one-dimensional case.
We will set the fun_evals=20 to limit the number of function evaluations to 20 for this example.
The size of the initial design is set to 10 by default. It can be changed by setting init_size=10 via design_control_init in the design_control dictionary.
Again, TensorBoard can be used to monitor the progress of the optimization.
We can also add interpretable labels to the dimensions, which will be used in the plots. Therefore, we set var_name=["Pressure", "Temp", "Lambda"] instead of the default var_name=None, which would result in the labels x_0, x_1, and x_2.

Here is the link to the documentation of the fun_control_init function: [DOC]. The documentation of the design_control_init function can be found here: [DOC].

The setup can be done as follows:

fun_control = fun_control_init(
              PREFIX="cubed",
              fun_evals=20,
              lower = -1.0*np.ones(3),
              upper = np.ones(3),
              var_name=["Pressure", "Temp", "Lambda"],
              TENSORBOARD_CLEAN=True,
              tensorboard_log=True
              )

surrogate_control = surrogate_control_init()
design_control = design_control_init(init_size=10)

Moving TENSORBOARD_PATH: runs/ to TENSORBOARD_PATH_OLD: runs_OLD/runs_2025_10_19_20_30_05_0
Created spot_tensorboard_path: runs/spot_logs/cubed_maans08_2025-10-19_20-30-05 for SummaryWriter()

After the setup, we can pass the dictionaries to the Spot class and run the optimization process.

spot_cubed = Spot(fun=fun_cubed,
                  fun_control=fun_control,
                  surrogate_control=surrogate_control)
spot_cubed.run()

Anisotropic model: n_theta set to 3
Anisotropic model: n_theta set to 3
spotpython tuning: -1.4616839557332901 [######----] 55.00% 
Anisotropic model: n_theta set to 3
spotpython tuning: -1.4616839557332901 [######----] 60.00% 
Anisotropic model: n_theta set to 3
spotpython tuning: -1.4616839557332901 [######----] 65.00% 
Anisotropic model: n_theta set to 3
spotpython tuning: -2.200571921617357 [#######---] 70.00% 
Anisotropic model: n_theta set to 3
spotpython tuning: -2.200571921617357 [########--] 75.00% 
Anisotropic model: n_theta set to 3
spotpython tuning: -2.5961345976775108 [########--] 80.00% 
Anisotropic model: n_theta set to 3
spotpython tuning: -3.0 [########--] 85.00% 
Anisotropic model: n_theta set to 3
spotpython tuning: -3.0 [#########-] 90.00% 
Anisotropic model: n_theta set to 3
spotpython tuning: -3.0 [##########] 95.00% 
Anisotropic model: n_theta set to 3
spotpython tuning: -3.0 [##########] 100.00% Done...

Experiment saved to cubed_res.pkl

<spotpython.spot.spot.Spot at 0x157ed8910>

Results

_ = spot_cubed.print_results()

min y: -3.0
Pressure: -1.0
Temp: -1.0
Lambda: -1.0

spot_cubed.plot_progress()

Contour Plots

We can select two dimensions, say \(i=0\) and \(j=1\), and generate a contour plot as follows.

We can specify identical min_z and max_z values to generate comparable plots. The default values are min_z=None and max_z=None, which will be replaced by the minimum and maximum values of the objective function.

min_z = -3
max_z = 1
spot_cubed.plot_contour(i=0, j=1, min_z=min_z, max_z=max_z)

In a similar manner, we can plot dimension \(i=0\) and \(j=2\):

spot_cubed.plot_contour(i=0, j=2, min_z=min_z, max_z=max_z)

The final combination is \(i=1\) and \(j=2\):

spot_cubed.plot_contour(i=1, j=2, min_z=min_z, max_z=max_z)

The variable importance can be printed and visualized as follows:

_ = spot_cubed.print_importance()

Pressure:  25.140455019814226
Temp:  100.0
Lambda:  79.65872854488313

spot_cubed.plot_importance()

Solution 13.2 (Solution to Exercise 13.5: The Two-dimensional Rosenbrock Function fun_rosen).

import numpy as np
from spotpython.fun.objectivefunctions import Analytical
from spotpython.utils.init import fun_control_init, surrogate_control_init
from spotpython.spot import Spot

The Objective Function: 2-dim fun_rosen

The spotpython package provides several classes of objective functions. We will use the fun_rosen in the Analytical class [SOURCE].

fun_rosen = Analytical().fun_rosen

Here we will use problem dimension \(k=2\), which can be specified by the lower bound arrays.
The size of the lower bound array determines the problem dimension. If we select -5.0 * np.ones(2), a two-dimensional function is created.
In contrast to the one-dimensional case, where only one theta value is used, we will use \(k\) different theta values (one for each dimension).
The prefix is set to "ROSEN".
Again, TensorBoard can be used to monitor the progress of the optimization.

fun_control = fun_control_init(
              PREFIX="ROSEN",
              lower = -5.0*np.ones(2),
              upper = 10*np.ones(2),
              fun_evals=25)
surrogate_control = surrogate_control_init()
spot_rosen = Spot(fun=fun_rosen,
                  fun_control=fun_control,
                  surrogate_control=surrogate_control)
spot_rosen.run()

Anisotropic model: n_theta set to 2
Anisotropic model: n_theta set to 2
spotpython tuning: 90.80194106348245 [####------] 44.00% 
Anisotropic model: n_theta set to 2
spotpython tuning: 1.0168856465748572 [#####-----] 48.00% 
Anisotropic model: n_theta set to 2
spotpython tuning: 1.0168856465748572 [#####-----] 52.00% 
Anisotropic model: n_theta set to 2
spotpython tuning: 1.0168856465748572 [######----] 56.00% 
Anisotropic model: n_theta set to 2
spotpython tuning: 1.0168856465748572 [######----] 60.00% 
Anisotropic model: n_theta set to 2
spotpython tuning: 1.0168856465748572 [######----] 64.00% 
Anisotropic model: n_theta set to 2
spotpython tuning: 1.0168856465748572 [#######---] 68.00% 
Anisotropic model: n_theta set to 2
spotpython tuning: 1.0168856465748572 [#######---] 72.00% 
Anisotropic model: n_theta set to 2
spotpython tuning: 1.0168856465748572 [########--] 76.00% 
Anisotropic model: n_theta set to 2
spotpython tuning: 0.7198700265837477 [########--] 80.00% 
Anisotropic model: n_theta set to 2
spotpython tuning: 0.7198700265837477 [########--] 84.00% 
Anisotropic model: n_theta set to 2
spotpython tuning: 0.7198700265837477 [#########-] 88.00% 
Anisotropic model: n_theta set to 2
spotpython tuning: 0.7198700265837477 [#########-] 92.00% 
Anisotropic model: n_theta set to 2
spotpython tuning: 0.7198700265837477 [##########] 96.00% 
Anisotropic model: n_theta set to 2
spotpython tuning: 0.7198700265837477 [##########] 100.00% Done...

Experiment saved to ROSEN_res.pkl

<spotpython.spot.spot.Spot at 0x1745c2490>

Now we can start TensorBoard in the background with the following command:

tensorboard --logdir="./runs"

and can access the TensorBoard web server with the following URL:

http://localhost:6006/

Results

_ = spot_rosen.print_results()

min y: 0.7198700265837477
x0: 0.19638653749225776
x1: 0.12463475609508881

spot_rosen.plot_progress(log_y=True)

A Contour Plot: We can select two dimensions, say \(i=0\) and \(j=1\), and generate a contour plot as follows.
Note: For higher dimensions, it might be useful to have identical min_z and max_z values to generate comparable plots. The default values are min_z=None and max_z=None, which will be replaced by the minimum and maximum values of the objective function.

min_z = None
max_z = None
spot_rosen.plot_contour(i=0, j=1, min_z=min_z, max_z=max_z)

The variable importance can be calculated as follows:

_ = spot_rosen.print_importance()

x0:  100.0
x1:  2.8679319810703214

spot_rosen.plot_importance()

13.4 Jupyter Notebook

Note

The Jupyter-Notebook of this lecture is available on GitHub in the Hyperparameter-Tuning-Cookbook Repository