import numpy as np
from math import inf
from spotpython.fun.objectivefunctions import analytical
from spotpython.spot import spot
import matplotlib.pyplot as plt
from spotpython.utils.init import fun_control_init, get_spot_tensorboard_path
from spotpython.utils.init import fun_control_init, design_control_init, surrogate_control_init
= "08" PREFIX
13 Handling Noise
This chapter demonstrates how noisy functions can be handled by Spot
and how noise can be simulated, i.e., added to the objective function.
13.1 Example: Spot
and the Noisy Sphere Function
13.1.1 The Objective Function: Noisy Sphere
The spotpython
package provides several classes of objective functions, which return a one-dimensional output \(y=f(x)\) for a given input \(x\) (independent variable). Several objective functions allow one- or multidimensional input, some also combinations of real-valued and categorial input values.
An objective function is considered as “analytical” if it can be described by a closed mathematical formula, e.g., \[ f(x, y) = x^2 + y^2. \]
To simulate measurement errors, adding artificial noise to the function value \(y\) is a common practice, e.g.,:
\[ f(x, y) = x^2 + y^2 + \epsilon. \]
Usually, noise is assumed to be normally distributed with mean \(\mu=0\) and standard deviation \(\sigma\). spotpython uses numpy’s scale
parameter, which specifies the standard deviation (spread or “width”) of the distribution is used. This must be a non-negative value, see https://numpy.org/doc/stable/reference/random/generated/numpy.random.normal.html.
The default setting does not use any noise.
from spotpython.fun.objectivefunctions import analytical
= analytical().fun_sphere
fun = np.linspace(-1,1,100).reshape(-1,1)
x = fun(x)
y
plt.figure()"k")
plt.plot(x,y, plt.show()
Noise can be added to the sphere function as follows:
from spotpython.fun.objectivefunctions import analytical
= analytical(seed=123, sigma=0.02).fun_sphere
fun = np.linspace(-1,1,100).reshape(-1,1)
x = fun(x)
y
plt.figure()"k")
plt.plot(x,y, plt.show()
13.1.2 Reproducibility: Noise Generation and Seed Handling
spotpython provides two mechanisms for generating random noise:
- The seed is initialized once, i.e., when the objective function is instantiated. This can be done using the following call:
fun = analytical(sigma=0.02, seed=123).fun_sphere
. - The seed is set every time the objective function is called. This can be done using the following call:
y = fun(x, sigma=0.02, seed=123)
.
These two different ways lead to different results as explained in the following tables:
Since sigma
is set to 0.02
, noise is added to the function:
from spotpython.fun.objectivefunctions import analytical
= analytical(sigma=0.02, seed=123).fun_sphere
fun = np.array([1]).reshape(-1,1)
x for i in range(3):
print(f"{i}: {fun(x)}")
0: [0.98021757]
1: [0.99264427]
2: [1.02575851]
The seed is set once. Every call to fun()
results in a different value. The whole experiment can be repeated, the initial seed is used to generate the same sequence as shown below:
Since sigma
is set to 0.02
, noise is added to the function:
from spotpython.fun.objectivefunctions import analytical
= analytical(sigma=0.02, seed=123).fun_sphere
fun = np.array([1]).reshape(-1,1)
x for i in range(3):
print(f"{i}: {fun(x)}")
0: [0.98021757]
1: [0.99264427]
2: [1.02575851]
If spotpython
is used as a hyperparameter tuner, it is important that only one realization of the noise function is optimized. This behaviour can be accomplished by passing the same seed via the dictionary fun_control
to every call of the objective function fun
as shown below:
Since sigma
is set to 0.02
, noise is added to the function:
from spotpython.fun.objectivefunctions import analytical
= analytical().fun_sphere
fun = fun_control_init(
fun_control =PREFIX,
PREFIX=0.02)
sigma= fun(x, fun_control=fun_control)
y = np.array([1]).reshape(-1,1)
x for i in range(3):
print(f"{i}: {fun(x)}")
0: [0.98021757]
1: [0.98021757]
2: [0.98021757]
13.2 spotpython’s Noise Handling Approaches
The following setting will be used for the next steps:
= analytical().fun_sphere
fun = fun_control_init(
fun_control =PREFIX,
PREFIX=0.02,
sigma )
spotpython
is adopted as follows to cope with noisy functions:
fun_repeats
is set to a value larger than 1 (here: 2)noise
is set totrue
. Therefore, a nugget (Lambda
) term is added to the correlation matrixinit size
(of thedesign_control
dictionary) is set to a value larger than 1 (here: 3)
= spot.Spot(fun=fun,
spot_1_noisy =fun_control_init(
fun_control= np.array([-1]),
lower = np.array([1]),
upper = 20,
fun_evals = 2,
fun_repeats = True,
noise =True),
show_models=design_control_init(init_size=3, repeats=2),
design_control=surrogate_control_init(noise=True)) surrogate_control
spot_1_noisy.run()
spotpython tuning: 0.034752873669989026 [####------] 40.00%
spotpython tuning: 0.03230817945928789 [#####-----] 50.00%
spotpython tuning: 0.015578418800855254 [######----] 60.00%
spotpython tuning: 0.0009550994714026289 [#######---] 70.00%
spotpython tuning: 5.561590542963861e-05 [########--] 80.00%
spotpython tuning: 7.181090066713707e-07 [#########-] 90.00%
spotpython tuning: 4.5254143126895086e-07 [##########] 100.00% Done...
13.3 Print the Results
spot_1_noisy.print_results()
min y: 4.5254143126895086e-07
min mean y: 4.5254143126895086e-07
x0: 0.0006727119972684825
[['x0', 0.0006727119972684825]]
=False,
spot_1_noisy.plot_progress(log_y="./figures/" + PREFIX + "_progress.png") filename
13.4 Noise and Surrogates: The Nugget Effect
13.4.1 The Noisy Sphere
13.4.1.1 The Data
- We prepare some data first:
import numpy as np
import spotpython
from spotpython.fun.objectivefunctions import analytical
from spotpython.spot import spot
from spotpython.design.spacefilling import spacefilling
from spotpython.build.kriging import Kriging
import matplotlib.pyplot as plt
= spacefilling(1)
gen = np.random.RandomState(1)
rng = np.array([-10])
lower = np.array([10])
upper = analytical().fun_sphere
fun = fun_control_init(
fun_control =PREFIX,
PREFIX=4)
sigma= gen.scipy_lhd(10, lower=lower, upper = upper)
X = fun(X, fun_control=fun_control)
y = X.reshape(-1,1)
X_train = y y_train
- A surrogate without nugget is fitted to these data:
= Kriging(name='kriging',
S =1,
n_theta=False)
noise
S.fit(X_train, y_train)
= np.linspace(start=-13, stop=13, num=1000).reshape(-1, 1)
X_axis = S.predict(X_axis, return_val="all")
mean_prediction, std_prediction, ei
="Observations")
plt.scatter(X_train, y_train, label="mue")
plt.plot(X_axis, mean_prediction, label
plt.legend()"$x$")
plt.xlabel("$f(x)$")
plt.ylabel(= plt.title("Sphere: Gaussian process regression on noisy dataset") _
- In comparison to the surrogate without nugget, we fit a surrogate with nugget to the data:
= Kriging(name='kriging',
S_nug =1,
n_theta=True)
noise
S_nug.fit(X_train, y_train)= np.linspace(start=-13, stop=13, num=1000).reshape(-1, 1)
X_axis = S_nug.predict(X_axis, return_val="all")
mean_prediction, std_prediction, ei ="Observations")
plt.scatter(X_train, y_train, label="mue")
plt.plot(X_axis, mean_prediction, label
plt.legend()"$x$")
plt.xlabel("$f(x)$")
plt.ylabel(= plt.title("Sphere: Gaussian process regression with nugget on noisy dataset") _
- The value of the nugget term can be extracted from the model as follows:
S.Lambda
S_nug.Lambda
0.0005592051705322895
- We see:
- the first model
S
has no nugget, - whereas the second model has a nugget value (
Lambda
) larger than zero.
- the first model
13.5 Exercises
13.5.1 Noisy fun_cubed
- Analyse the effect of noise on the
fun_cubed
function with the following settings:
= analytical().fun_cubed
fun = fun_control_init(
fun_control =10)
sigma= np.array([-10])
lower = np.array([10]) upper
13.5.2 fun_runge
- Analyse the effect of noise on the
fun_runge
function with the following settings:
= np.array([-10])
lower = np.array([10])
upper = analytical().fun_runge
fun = fun_control_init(
fun_control =0.25) sigma
13.5.3 fun_forrester
- Analyse the effect of noise on the
fun_forrester
function with the following settings:
= np.array([0])
lower = np.array([1])
upper = analytical().fun_forrester
fun = fun_control_init(
fun_control =5) sigma
13.5.4 fun_xsin
- Analyse the effect of noise on the
fun_xsin
function with the following settings:
= np.array([-1.])
lower = np.array([1.])
upper = analytical().fun_xsin
fun = fun_control_init(
fun_control =0.5) sigma