13  Handling Noise

This chapter demonstrates how noisy functions can be handled by Spot and how noise can be simulated, i.e., added to the objective function.

13.1 Example: Spot and the Noisy Sphere Function

import numpy as np
from math import inf
from spotPython.fun.objectivefunctions import analytical
from spotPython.spot import spot
import matplotlib.pyplot as plt
from spotPython.utils.init import fun_control_init, get_spot_tensorboard_path
from spotPython.utils.init import fun_control_init, design_control_init, surrogate_control_init

PREFIX = "08"

13.1.1 The Objective Function: Noisy Sphere

The spotPython package provides several classes of objective functions, which return a one-dimensional output \(y=f(x)\) for a given input \(x\) (independent variable). Several objective functions allow one- or multidimensional input, some also combinations of real-valued and categorial input values.

An objective function is considered as “analytical” if it can be described by a closed mathematical formula, e.g., \[ f(x, y) = x^2 + y^2. \]

To simulate measurement errors, adding artificial noise to the function value \(y\) is a common practice, e.g.,:

\[ f(x, y) = x^2 + y^2 + \epsilon. \]

Usually, noise is assumed to be normally distributed with mean \(\mu=0\) and standard deviation \(\sigma\). spotPython uses numpy’s scale parameter, which specifies the standard deviation (spread or “width”) of the distribution is used. This must be a non-negative value, see https://numpy.org/doc/stable/reference/random/generated/numpy.random.normal.html.

Example: The sphere function without noise

The default setting does not use any noise.

from spotPython.fun.objectivefunctions import analytical
fun = analytical().fun_sphere
x = np.linspace(-1,1,100).reshape(-1,1)
y = fun(x)
plt.figure()
plt.plot(x,y, "k")
plt.show()

Example: The sphere function with noise

Noise can be added to the sphere function as follows:

from spotPython.fun.objectivefunctions import analytical
fun = analytical(seed=123, sigma=0.02).fun_sphere
x = np.linspace(-1,1,100).reshape(-1,1)
y = fun(x)
plt.figure()
plt.plot(x,y, "k")
plt.show()

13.1.2 Reproducibility: Noise Generation and Seed Handling

spotPython provides two mechanisms for generating random noise:

  1. The seed is initialized once, i.e., when the objective function is instantiated. This can be done using the following call: fun = analytical(sigma=0.02, seed=123).fun_sphere.
  2. The seed is set every time the objective function is called. This can be done using the following call: y = fun(x, sigma=0.02, seed=123).

These two different ways lead to different results as explained in the following tables:

Example: Noise added to the sphere function

Since sigma is set to 0.02, noise is added to the function:

from spotPython.fun.objectivefunctions import analytical
fun = analytical(sigma=0.02, seed=123).fun_sphere
x = np.array([1]).reshape(-1,1)
for i in range(3):
    print(f"{i}: {fun(x)}")
0: [0.98021757]
1: [0.99264427]
2: [1.02575851]

The seed is set once. Every call to fun() results in a different value. The whole experiment can be repeated, the initial seed is used to generate the same sequence as shown below:

Example: Noise added to the sphere function

Since sigma is set to 0.02, noise is added to the function:

from spotPython.fun.objectivefunctions import analytical
fun = analytical(sigma=0.02, seed=123).fun_sphere
x = np.array([1]).reshape(-1,1)
for i in range(3):
    print(f"{i}: {fun(x)}")
0: [0.98021757]
1: [0.99264427]
2: [1.02575851]

If spotPython is used as a hyperparameter tuner, it is important that only one realization of the noise function is optimized. This behaviour can be accomplished by passing the same seed via the dictionary fun_control to every call of the objective function fun as shown below:

Example: The same noise added to the sphere function

Since sigma is set to 0.02, noise is added to the function:

from spotPython.fun.objectivefunctions import analytical
fun = analytical().fun_sphere
fun_control = fun_control_init(
    PREFIX=PREFIX,
    sigma=0.02)
y = fun(x, fun_control=fun_control)
x = np.array([1]).reshape(-1,1)
for i in range(3):
    print(f"{i}: {fun(x)}")
Created spot_tensorboard_path: runs/spot_logs/08_maans14_2024-04-22_00-27-19 for SummaryWriter()
0: [0.98021757]
1: [0.98021757]
2: [0.98021757]

13.2 spotPython’s Noise Handling Approaches

The following setting will be used for the next steps:

fun = analytical().fun_sphere
fun_control = fun_control_init(
    PREFIX=PREFIX,
    sigma=0.02,
)
Created spot_tensorboard_path: runs/spot_logs/08_maans14_2024-04-22_00-27-19 for SummaryWriter()

spotPython is adopted as follows to cope with noisy functions:

  1. fun_repeats is set to a value larger than 1 (here: 2)
  2. noise is set to true. Therefore, a nugget (Lambda) term is added to the correlation matrix
  3. init size (of the design_control dictionary) is set to a value larger than 1 (here: 3)
spot_1_noisy = spot.Spot(fun=fun,
                   fun_control=fun_control_init(
                                    lower = np.array([-1]),
                                    upper = np.array([1]),
                                    fun_evals = 20,
                                    fun_repeats = 2,
                                    noise = True,
                                    show_models=True),
                   design_control=design_control_init(init_size=3, repeats=2),
                   surrogate_control=surrogate_control_init(noise=True))
spot_1_noisy.run()

spotPython tuning: 0.034754930918721325 [####------] 40.00% 
spotPython tuning: 0.03339769765455772 [#####-----] 50.00% 
spotPython tuning: 0.015786142156557437 [######----] 60.00% 
spotPython tuning: 0.0005791214064992311 [#######---] 70.00% 
spotPython tuning: 3.5506618676925576e-05 [########--] 80.00% 
spotPython tuning: 5.325186406243954e-07 [#########-] 90.00% 
spotPython tuning: 4.335518265610199e-07 [##########] 100.00% Done...

{'CHECKPOINT_PATH': 'runs/saved_models/',
 'DATASET_PATH': 'data/',
 'PREFIX': None,
 'RESULTS_PATH': 'results/',
 'TENSORBOARD_PATH': 'runs/',
 '_L_in': None,
 '_L_out': None,
 '_torchmetric': None,
 'accelerator': 'auto',
 'converters': None,
 'core_model': None,
 'core_model_name': None,
 'counter': 20,
 'data': None,
 'data_dir': './data',
 'data_module': None,
 'data_set': None,
 'data_set_name': None,
 'db_dict_name': None,
 'design': None,
 'device': None,
 'devices': 1,
 'enable_progress_bar': False,
 'eval': None,
 'fun_evals': 20,
 'fun_repeats': 2,
 'horizon': None,
 'infill_criterion': 'y',
 'k_folds': 3,
 'log_graph': False,
 'log_level': 50,
 'loss_function': None,
 'lower': array([-1]),
 'max_surrogate_points': 30,
 'max_time': 1,
 'metric_params': {},
 'metric_river': None,
 'metric_sklearn': None,
 'metric_sklearn_name': None,
 'metric_torch': None,
 'model_dict': {},
 'n_points': 1,
 'n_samples': None,
 'n_total': None,
 'noise': True,
 'num_workers': 0,
 'ocba_delta': 0,
 'oml_grace_period': None,
 'optimizer': None,
 'path': None,
 'prep_model': None,
 'prep_model_name': None,
 'progress_file': None,
 'save_model': False,
 'scenario': None,
 'seed': 123,
 'show_batch_interval': 1000000,
 'show_models': True,
 'show_progress': True,
 'shuffle': None,
 'sigma': 0.0,
 'spot_tensorboard_path': None,
 'spot_writer': None,
 'target_column': None,
 'target_type': None,
 'task': None,
 'test': None,
 'test_seed': 1234,
 'test_size': 0.4,
 'tolerance_x': 0,
 'train': None,
 'upper': array([1]),
 'var_name': None,
 'var_type': ['num'],
 'verbosity': 0,
 'weight_coeff': 0.0,
 'weights': 1.0,
 'weights_entry': None}

13.4 Noise and Surrogates: The Nugget Effect

13.4.1 The Noisy Sphere

13.4.1.1 The Data

  • We prepare some data first:
import numpy as np
import spotPython
from spotPython.fun.objectivefunctions import analytical
from spotPython.spot import spot
from spotPython.design.spacefilling import spacefilling
from spotPython.build.kriging import Kriging
import matplotlib.pyplot as plt

gen = spacefilling(1)
rng = np.random.RandomState(1)
lower = np.array([-10])
upper = np.array([10])
fun = analytical().fun_sphere
fun_control = fun_control_init(
    PREFIX=PREFIX,
    sigma=4)
X = gen.scipy_lhd(10, lower=lower, upper = upper)
y = fun(X, fun_control=fun_control)
X_train = X.reshape(-1,1)
y_train = y
Created spot_tensorboard_path: runs/spot_logs/08_maans14_2024-04-22_00-27-24 for SummaryWriter()
  • A surrogate without nugget is fitted to these data:
S = Kriging(name='kriging',
            n_theta=1,
            noise=False)
S.fit(X_train, y_train)

X_axis = np.linspace(start=-13, stop=13, num=1000).reshape(-1, 1)
mean_prediction, std_prediction, ei = S.predict(X_axis, return_val="all")

plt.scatter(X_train, y_train, label="Observations")
plt.plot(X_axis, mean_prediction, label="mue")
plt.legend()
plt.xlabel("$x$")
plt.ylabel("$f(x)$")
_ = plt.title("Sphere: Gaussian process regression on noisy dataset")

  • In comparison to the surrogate without nugget, we fit a surrogate with nugget to the data:
S_nug = Kriging(name='kriging',
            n_theta=1,
            noise=True)
S_nug.fit(X_train, y_train)
X_axis = np.linspace(start=-13, stop=13, num=1000).reshape(-1, 1)
mean_prediction, std_prediction, ei = S_nug.predict(X_axis, return_val="all")
plt.scatter(X_train, y_train, label="Observations")
plt.plot(X_axis, mean_prediction, label="mue")
plt.legend()
plt.xlabel("$x$")
plt.ylabel("$f(x)$")
_ = plt.title("Sphere: Gaussian process regression with nugget on noisy dataset")

  • The value of the nugget term can be extracted from the model as follows:
S.Lambda
S_nug.Lambda
0.00055921881757264
  • We see:
    • the first model S has no nugget,
    • whereas the second model has a nugget value (Lambda) larger than zero.

13.5 Exercises

13.5.1 Noisy fun_cubed

  • Analyse the effect of noise on the fun_cubed function with the following settings:
fun = analytical().fun_cubed
fun_control = fun_control_init(
    sigma=10)
lower = np.array([-10])
upper = np.array([10])

13.5.2 fun_runge

  • Analyse the effect of noise on the fun_runge function with the following settings:
lower = np.array([-10])
upper = np.array([10])
fun = analytical().fun_runge
fun_control = fun_control_init(
    sigma=0.25)

13.5.3 fun_forrester

  • Analyse the effect of noise on the fun_forrester function with the following settings:
lower = np.array([0])
upper = np.array([1])
fun = analytical().fun_forrester
fun_control = fun_control_init(
    sigma=5)

13.5.4 fun_xsin

  • Analyse the effect of noise on the fun_xsin function with the following settings:
lower = np.array([-1.])
upper = np.array([1.])
fun = analytical().fun_xsin
fun_control = fun_control_init(    
    sigma=0.5)