import numpy as np
from math import inf
from spotpython.fun.objectivefunctions import Analytical
from spotpython.spot import Spot
import matplotlib.pyplot as plt
from spotpython.utils.init import fun_control_init, get_spot_tensorboard_path
from spotpython.utils.init import fun_control_init, design_control_init, surrogate_control_init
= "14" PREFIX
14 Optimal Computational Budget Allocation in spotpython
This chapter demonstrates how noisy functions can be handled spotpython
:
- First, Optimal Computational Budget Allocation (OCBA) is introduced in Section 14.2.
- Then, the nugget effect is explained in Section 14.3.
Citation
If this document has been useful to you and you wish to cite it in a scientific publication, please refer to the following paper, which can be found on arXiv: https://arxiv.org/abs/2307.10262.
@ARTICLE{bart23iArXiv,
author = {{Bartz-Beielstein}, Thomas},
title = "{Hyperparameter Tuning Cookbook:
A guide for scikit-learn, PyTorch, river, and spotpython}",
journal = {arXiv e-prints},
keywords = {Computer Science - Machine Learning,
Computer Science - Artificial Intelligence, 90C26, I.2.6, G.1.6},
year = 2023,
month = jul,
eid = {arXiv:2307.10262},
doi = {10.48550/arXiv.2307.10262},
archivePrefix = {arXiv},
eprint = {2307.10262},
primaryClass = {cs.LG}
}
14.1 Example: spotpython
, OCBA, and the Noisy Sphere Function
14.1.1 The Objective Function: Noisy Sphere
The spotpython
package provides several classes of objective functions. We will use an analytical objective function with noise, i.e., a function that can be described by a (closed) formula: \[f(x) = x^2 + \epsilon\]
Since sigma
is set to 0.1
, noise is added to the function:
= Analytical().fun_sphere
fun = fun_control_init(
fun_control =PREFIX,
PREFIX=0.1) sigma
A plot (Figure 14.1) illustrates the noise:
= np.linspace(-1,1,100).reshape(-1,1)
x = fun(x, fun_control=fun_control)
y
plt.figure()"k")
plt.plot(x,y, plt.show()

spotpython
spotpython
has several options to cope with noisy functions:
fun_repeats
is set to a value larger than 1, e.g., 2, which means every function evaluation during the search on the surrogate is repeated twice. The mean of the two evaluations is used as the function value.init size
(of thedesign_control
dictionary) is set to a value larger than 1 (here: 2).ocba_delta
is set to a value larger than 1 (here: 2). This means that the OCBA algorithm is used to allocate the computational budget optimally. An example is given in Section 14.2.- Using a nugget term in the surrogate model. This is done by setting
method="regression"
in thesurrogate_control
dictionary. An example is given in Section 14.3.
14.2 Using Optimal Computational Budget Allocation (OCBA)
The Optimal Computational Budget Allocation (OCBA) algorithm is a powerful tool for efficiently distributing computational resources (Chen 2010). It is specifically designed to maximize the Probability of Correct Selection (PCS) while minimizing computational costs. By strategically allocating more simulation effort to design alternatives that are either more challenging to evaluate or more likely to yield optimal results, OCBA ensures an efficient use of resources. This approach enables researchers and decision-makers to achieve accurate outcomes more quickly and with fewer computational demands, making it an invaluable method for simulation optimization.
The OCBA algorithm is implemented in spotpython
and can be used by setting ocba_delta
to a value larger than 0
. The source code is available in the spotpython
package, see [DOC]. See also Bartz-Beielstein et al. (2011).
Example 14.1 To reproduce the example from p.49 in Chen (2010), the following spotpython
code can be used:
import numpy as np
from spotpython.budget.ocba import get_ocba
= np.array([1,2,3,4,5])
mean_y = np.array([1,1,9,9,4])
var_y 50) get_ocba(mean_y, var_y,
array([11, 9, 19, 9, 2])
14.2.1 The Noisy Sphere
We will demonstrate the OCBA algorithm on the noisy sphere function defined in Section 14.1.1. The OCBA algorithm is used to allocate the computational budget optimally. This means that the function evaluations are repeated several times, and the best function value is used for the next iteration.
- The
show_models
parameter in thefun_control
dictionary is set toTrue
. This means that the surrogate model is shown during the search. - To keep the visualization simple, only the ground truth and the surrogate model are shown. The surrogate model is shown in blue, and the ground truth is shown in orange. The noisy function was shown in Figure 14.1.
= Spot(fun=fun,
spot_1_noisy =fun_control_init(
fun_control= np.array([-1]),
lower = np.array([1]),
upper = 20,
fun_evals = 1,
fun_repeats = True,
noise =0.0,
tolerance_x= 2,
ocba_delta =True),
show_models=design_control_init(init_size=5, repeats=2),
design_control=surrogate_control_init(method="regression")) surrogate_control
spot_1_noisy.run()
spotpython tuning: 0.005320352324811128 [######----] 55.00%
spotpython tuning: 0.000565644894785201 [######----] 60.00%
spotpython tuning: 1.591451674748005e-05 [######----] 65.00%
spotpython tuning: 6.91090859790612e-07 [#######---] 70.00%
spotpython tuning: 1.8899294735019857e-07 [########--] 75.00%
spotpython tuning: 1.2178603905219397e-07 [########--] 80.00%
spotpython tuning: 1.0212580354992866e-07 [########--] 85.00%
spotpython tuning: 9.357713991612833e-08 [#########-] 90.00%
spotpython tuning: 9.357713991612833e-08 [##########] 100.00% Done...
Experiment saved to 000_res.pkl
14.2.2 Print the Results
spot_1_noisy.print_results()
min y: 9.357713991612833e-08
min mean y: 9.357713991612833e-08
x0: 0.0003059038082733334
[['x0', np.float64(0.0003059038082733334)]]
=True) spot_1_noisy.plot_progress(log_y
14.3 Noise and Surrogates: The Nugget Effect
In the previous example, we have seen that the fun_repeats
parameter can be used to repeat function evaluations. This is useful when the function is noisy. However, it is not always possible to repeat function evaluations, e.g., when the function is expensive to evaluate. In this case, we can use a surrogate model with a nugget term. The nugget term is a small value that is added to the diagonal of the covariance matrix. This allows the surrogate model to fit the data better, even if the data is noisy. The nugget term is added, if method="regression"
is set in the surrogate_control
dictionary.
14.3.1 The Noisy Sphere
14.3.1.1 The Data
We prepare some data first:
import numpy as np
import spotpython
from spotpython.fun.objectivefunctions import Analytical
from spotpython.spot import Spot
from spotpython.design.spacefilling import SpaceFilling
from spotpython.surrogate.kriging import Kriging
import matplotlib.pyplot as plt
= SpaceFilling(1)
gen = np.random.RandomState(1)
rng = np.array([-10])
lower = np.array([10])
upper = Analytical().fun_sphere
fun = fun_control_init(
fun_control =2,
sigma=125)
seed= gen.scipy_lhd(10, lower=lower, upper = upper)
X = fun(X, fun_control=fun_control)
y = X.reshape(-1,1)
X_train = y y_train
A surrogate without nugget is fitted to these data:
= Kriging(name='kriging',
S =123,
seed=50,
log_level=1,
n_theta="interpolation")
method
S.fit(X_train, y_train)
= np.linspace(start=-13, stop=13, num=1000).reshape(-1, 1)
X_axis = S.predict(X_axis, return_val="all")
mean_prediction, std_prediction, ei
="Observations")
plt.scatter(X_train, y_train, label="mue")
plt.plot(X_axis, mean_prediction, label
plt.legend()"$x$")
plt.xlabel("$f(x)$")
plt.ylabel(= plt.title("Sphere: Gaussian process regression on noisy dataset") _
In comparison to the surrogate without nugget, we fit a surrogate with nugget to the data:
= Kriging(name='kriging',
S_nug =123,
seed=50,
log_level=1,
n_theta="regression")
method
S_nug.fit(X_train, y_train)= np.linspace(start=-13, stop=13, num=1000).reshape(-1, 1)
X_axis = S_nug.predict(X_axis, return_val="all")
mean_prediction, std_prediction, ei ="Observations")
plt.scatter(X_train, y_train, label="mue")
plt.plot(X_axis, mean_prediction, label
plt.legend()"$x$")
plt.xlabel("$f(x)$")
plt.ylabel(= plt.title("Sphere: Gaussian process regression with nugget on noisy dataset") _
The value of the nugget term can be extracted from the model as follows:
S.Lambda
10**S_nug.Lambda
np.float64(0.0002592770433217388)
We see:
- the first model
S
has no nugget, - whereas the second model has a nugget value (
Lambda
) larger than zero.
14.4 Exercises
14.4.1 Noisy fun_cubed
Analyse the effect of noise on the fun_cubed
function with the following settings:
= Analytical().fun_cubed
fun = fun_control_init(
fun_control =10,
sigma=123)
seed= np.array([-10])
lower = np.array([10]) upper
14.4.2 fun_runge
Analyse the effect of noise on the fun_runge
function with the following settings:
= np.array([-10])
lower = np.array([10])
upper = Analytical().fun_runge
fun = fun_control_init(
fun_control =0.25,
sigma=123) seed
14.4.3 fun_forrester
Analyse the effect of noise on the fun_forrester
function with the following settings:
= np.array([0])
lower = np.array([1])
upper = Analytical().fun_forrester
fun = {"sigma": 5,
fun_control "seed": 123}
14.4.4 fun_xsin
Analyse the effect of noise on the fun_xsin
function with the following settings:
= np.array([-1.])
lower = np.array([1.])
upper = Analytical().fun_xsin
fun = fun_control_init(
fun_control =0.5,
sigma=123) seed
14.5 Jupyter Notebook
- The Jupyter-Notebook of this chapter is available on GitHub in the Hyperparameter-Tuning-Cookbook Repository