import numpy as np
from math import inf
from spotPython.fun.objectivefunctions import analytical
from spotPython.spot import spot
import matplotlib.pyplot as plt
from spotPython.utils.init import fun_control_init, get_spot_tensorboard_path
from spotPython.utils.init import fun_control_init, design_control_init, surrogate_control_init
= "08" PREFIX
13 Handling Noise
This chapter demonstrates how noisy functions can be handled by Spot
and how noise can be simulated, i.e., added to the objective function.
13.1 Example: Spot
and the Noisy Sphere Function
13.1.1 The Objective Function: Noisy Sphere
The spotPython
package provides several classes of objective functions, which return a one-dimensional output \(y=f(x)\) for a given input \(x\) (independent variable). Several objective functions allow one- or multidimensional input, some also combinations of real-valued and categorial input values.
An objective function is considered as “analytical” if it can be described by a closed mathematical formula, e.g., \[ f(x, y) = x^2 + y^2. \]
To simulate measurement errors, adding artificial noise to the function value \(y\) is a common practice, e.g.,:
\[ f(x, y) = x^2 + y^2 + \epsilon. \]
Usually, noise is assumed to be normally distributed with mean \(\mu=0\) and standard deviation \(\sigma\). spotPython uses numpy’s scale
parameter, which specifies the standard deviation (spread or “width”) of the distribution is used. This must be a non-negative value, see https://numpy.org/doc/stable/reference/random/generated/numpy.random.normal.html.
The default setting does not use any noise.
from spotPython.fun.objectivefunctions import analytical
= analytical().fun_sphere
fun = np.linspace(-1,1,100).reshape(-1,1)
x = fun(x)
y
plt.figure()"k")
plt.plot(x,y, plt.show()
Noise can be added to the sphere function as follows:
from spotPython.fun.objectivefunctions import analytical
= analytical(seed=123, sigma=0.02).fun_sphere
fun = np.linspace(-1,1,100).reshape(-1,1)
x = fun(x)
y
plt.figure()"k")
plt.plot(x,y, plt.show()
13.1.2 Reproducibility: Noise Generation and Seed Handling
spotPython provides two mechanisms for generating random noise:
- The seed is initialized once, i.e., when the objective function is instantiated. This can be done using the following call:
fun = analytical(sigma=0.02, seed=123).fun_sphere
. - The seed is set every time the objective function is called. This can be done using the following call:
y = fun(x, sigma=0.02, seed=123)
.
These two different ways lead to different results as explained in the following tables:
Since sigma
is set to 0.02
, noise is added to the function:
from spotPython.fun.objectivefunctions import analytical
= analytical(sigma=0.02, seed=123).fun_sphere
fun = np.array([1]).reshape(-1,1)
x for i in range(3):
print(f"{i}: {fun(x)}")
0: [0.98021757]
1: [0.99264427]
2: [1.02575851]
The seed is set once. Every call to fun()
results in a different value. The whole experiment can be repeated, the initial seed is used to generate the same sequence as shown below:
Since sigma
is set to 0.02
, noise is added to the function:
from spotPython.fun.objectivefunctions import analytical
= analytical(sigma=0.02, seed=123).fun_sphere
fun = np.array([1]).reshape(-1,1)
x for i in range(3):
print(f"{i}: {fun(x)}")
0: [0.98021757]
1: [0.99264427]
2: [1.02575851]
If spotPython
is used as a hyperparameter tuner, it is important that only one realization of the noise function is optimized. This behaviour can be accomplished by passing the same seed via the dictionary fun_control
to every call of the objective function fun
as shown below:
Since sigma
is set to 0.02
, noise is added to the function:
from spotPython.fun.objectivefunctions import analytical
= analytical().fun_sphere
fun = fun_control_init(
fun_control =PREFIX,
PREFIX=0.02)
sigma= fun(x, fun_control=fun_control)
y = np.array([1]).reshape(-1,1)
x for i in range(3):
print(f"{i}: {fun(x)}")
Created spot_tensorboard_path: runs/spot_logs/08_maans14_2024-04-22_00-27-19 for SummaryWriter()
0: [0.98021757]
1: [0.98021757]
2: [0.98021757]
13.2 spotPython’s Noise Handling Approaches
The following setting will be used for the next steps:
= analytical().fun_sphere
fun = fun_control_init(
fun_control =PREFIX,
PREFIX=0.02,
sigma )
Created spot_tensorboard_path: runs/spot_logs/08_maans14_2024-04-22_00-27-19 for SummaryWriter()
spotPython
is adopted as follows to cope with noisy functions:
fun_repeats
is set to a value larger than 1 (here: 2)noise
is set totrue
. Therefore, a nugget (Lambda
) term is added to the correlation matrixinit size
(of thedesign_control
dictionary) is set to a value larger than 1 (here: 3)
= spot.Spot(fun=fun,
spot_1_noisy =fun_control_init(
fun_control= np.array([-1]),
lower = np.array([1]),
upper = 20,
fun_evals = 2,
fun_repeats = True,
noise =True),
show_models=design_control_init(init_size=3, repeats=2),
design_control=surrogate_control_init(noise=True)) surrogate_control
spot_1_noisy.run()
spotPython tuning: 0.034754930918721325 [####------] 40.00%
spotPython tuning: 0.03339769765455772 [#####-----] 50.00%
spotPython tuning: 0.015786142156557437 [######----] 60.00%
spotPython tuning: 0.0005791214064992311 [#######---] 70.00%
spotPython tuning: 3.5506618676925576e-05 [########--] 80.00%
spotPython tuning: 5.325186406243954e-07 [#########-] 90.00%
spotPython tuning: 4.335518265610199e-07 [##########] 100.00% Done...
{'CHECKPOINT_PATH': 'runs/saved_models/',
'DATASET_PATH': 'data/',
'PREFIX': None,
'RESULTS_PATH': 'results/',
'TENSORBOARD_PATH': 'runs/',
'_L_in': None,
'_L_out': None,
'_torchmetric': None,
'accelerator': 'auto',
'converters': None,
'core_model': None,
'core_model_name': None,
'counter': 20,
'data': None,
'data_dir': './data',
'data_module': None,
'data_set': None,
'data_set_name': None,
'db_dict_name': None,
'design': None,
'device': None,
'devices': 1,
'enable_progress_bar': False,
'eval': None,
'fun_evals': 20,
'fun_repeats': 2,
'horizon': None,
'infill_criterion': 'y',
'k_folds': 3,
'log_graph': False,
'log_level': 50,
'loss_function': None,
'lower': array([-1]),
'max_surrogate_points': 30,
'max_time': 1,
'metric_params': {},
'metric_river': None,
'metric_sklearn': None,
'metric_sklearn_name': None,
'metric_torch': None,
'model_dict': {},
'n_points': 1,
'n_samples': None,
'n_total': None,
'noise': True,
'num_workers': 0,
'ocba_delta': 0,
'oml_grace_period': None,
'optimizer': None,
'path': None,
'prep_model': None,
'prep_model_name': None,
'progress_file': None,
'save_model': False,
'scenario': None,
'seed': 123,
'show_batch_interval': 1000000,
'show_models': True,
'show_progress': True,
'shuffle': None,
'sigma': 0.0,
'spot_tensorboard_path': None,
'spot_writer': None,
'target_column': None,
'target_type': None,
'task': None,
'test': None,
'test_seed': 1234,
'test_size': 0.4,
'tolerance_x': 0,
'train': None,
'upper': array([1]),
'var_name': None,
'var_type': ['num'],
'verbosity': 0,
'weight_coeff': 0.0,
'weights': 1.0,
'weights_entry': None}
13.3 Print the Results
spot_1_noisy.print_results()
min y: 4.335518265610199e-07
min mean y: 4.335518265610199e-07
x0: 0.0006584465252099216
[['x0', 0.0006584465252099216]]
=False,
spot_1_noisy.plot_progress(log_y="./figures/" + PREFIX + "_progress.png") filename
13.4 Noise and Surrogates: The Nugget Effect
13.4.1 The Noisy Sphere
13.4.1.1 The Data
- We prepare some data first:
import numpy as np
import spotPython
from spotPython.fun.objectivefunctions import analytical
from spotPython.spot import spot
from spotPython.design.spacefilling import spacefilling
from spotPython.build.kriging import Kriging
import matplotlib.pyplot as plt
= spacefilling(1)
gen = np.random.RandomState(1)
rng = np.array([-10])
lower = np.array([10])
upper = analytical().fun_sphere
fun = fun_control_init(
fun_control =PREFIX,
PREFIX=4)
sigma= gen.scipy_lhd(10, lower=lower, upper = upper)
X = fun(X, fun_control=fun_control)
y = X.reshape(-1,1)
X_train = y y_train
Created spot_tensorboard_path: runs/spot_logs/08_maans14_2024-04-22_00-27-24 for SummaryWriter()
- A surrogate without nugget is fitted to these data:
= Kriging(name='kriging',
S =1,
n_theta=False)
noise
S.fit(X_train, y_train)
= np.linspace(start=-13, stop=13, num=1000).reshape(-1, 1)
X_axis = S.predict(X_axis, return_val="all")
mean_prediction, std_prediction, ei
="Observations")
plt.scatter(X_train, y_train, label="mue")
plt.plot(X_axis, mean_prediction, label
plt.legend()"$x$")
plt.xlabel("$f(x)$")
plt.ylabel(= plt.title("Sphere: Gaussian process regression on noisy dataset") _
- In comparison to the surrogate without nugget, we fit a surrogate with nugget to the data:
= Kriging(name='kriging',
S_nug =1,
n_theta=True)
noise
S_nug.fit(X_train, y_train)= np.linspace(start=-13, stop=13, num=1000).reshape(-1, 1)
X_axis = S_nug.predict(X_axis, return_val="all")
mean_prediction, std_prediction, ei ="Observations")
plt.scatter(X_train, y_train, label="mue")
plt.plot(X_axis, mean_prediction, label
plt.legend()"$x$")
plt.xlabel("$f(x)$")
plt.ylabel(= plt.title("Sphere: Gaussian process regression with nugget on noisy dataset") _
- The value of the nugget term can be extracted from the model as follows:
S.Lambda
S_nug.Lambda
0.00055921881757264
- We see:
- the first model
S
has no nugget, - whereas the second model has a nugget value (
Lambda
) larger than zero.
- the first model
13.5 Exercises
13.5.1 Noisy fun_cubed
- Analyse the effect of noise on the
fun_cubed
function with the following settings:
= analytical().fun_cubed
fun = fun_control_init(
fun_control =10)
sigma= np.array([-10])
lower = np.array([10]) upper
13.5.2 fun_runge
- Analyse the effect of noise on the
fun_runge
function with the following settings:
= np.array([-10])
lower = np.array([10])
upper = analytical().fun_runge
fun = fun_control_init(
fun_control =0.25) sigma
13.5.3 fun_forrester
- Analyse the effect of noise on the
fun_forrester
function with the following settings:
= np.array([0])
lower = np.array([1])
upper = analytical().fun_forrester
fun = fun_control_init(
fun_control =5) sigma
13.5.4 fun_xsin
- Analyse the effect of noise on the
fun_xsin
function with the following settings:
= np.array([-1.])
lower = np.array([1.])
upper = analytical().fun_xsin
fun = fun_control_init(
fun_control =0.5) sigma