import numpy as np
from math import inf
from spotPython.fun.objectivefunctions import analytical
from spotPython.spot import spot
from spotPython.utils.init import fun_control_init, surrogate_control_init
="015" PREFIX
15 Kriging with Varying Correlation-p
This chapter illustrates the difference between Kriging models with varying p. The difference is illustrated with the help of the spotPython
package.
15.1 Example: Spot
Surrogate and the 2-dim Sphere Function
15.1.1 The Objective Function: 2-dim Sphere
- The
spotPython
package provides several classes of objective functions. - We will use an analytical objective function, i.e., a function that can be described by a (closed) formula: \[f(x, y) = x^2 + y^2\]
- The size of the
lower
bound vector determines the problem dimension. - Here we will use
np.array([-1, -1])
, i.e., a two-dim function.
= analytical().fun_sphere
fun = fun_control_init(PREFIX=PREFIX,
fun_control = np.array([-1, -1]),
lower = np.array([1, 1])) upper
Created spot_tensorboard_path: runs/spot_logs/015_maans14_2024-04-22_00-28-06 for SummaryWriter()
- Although the default
spot
surrogate model is an isotropic Kriging model, we will explicitly set thetheta
parameter to a value of1
for both dimensions. This is done to illustrate the difference between isotropic and anisotropic Kriging models.
=surrogate_control_init(n_p=1,
surrogate_control=2.0,) p_val
= spot.Spot(fun=fun,
spot_2 =fun_control,
fun_control=surrogate_control)
surrogate_control
spot_2.run()
spotPython tuning: 1.801603872454505e-05 [#######---] 73.33%
spotPython tuning: 1.801603872454505e-05 [########--] 80.00%
spotPython tuning: 1.801603872454505e-05 [#########-] 86.67%
spotPython tuning: 1.801603872454505e-05 [#########-] 93.33%
spotPython tuning: 1.801603872454505e-05 [##########] 100.00% Done...
{'CHECKPOINT_PATH': 'runs/saved_models/',
'DATASET_PATH': 'data/',
'PREFIX': '015',
'RESULTS_PATH': 'results/',
'TENSORBOARD_PATH': 'runs/',
'_L_in': None,
'_L_out': None,
'_torchmetric': None,
'accelerator': 'auto',
'converters': None,
'core_model': None,
'core_model_name': None,
'counter': 15,
'data': None,
'data_dir': './data',
'data_module': None,
'data_set': None,
'data_set_name': None,
'db_dict_name': None,
'design': None,
'device': None,
'devices': 1,
'enable_progress_bar': False,
'eval': None,
'fun_evals': 15,
'fun_repeats': 1,
'horizon': None,
'infill_criterion': 'y',
'k_folds': 3,
'log_graph': False,
'log_level': 50,
'loss_function': None,
'lower': array([-1, -1]),
'max_surrogate_points': 30,
'max_time': 1,
'metric_params': {},
'metric_river': None,
'metric_sklearn': None,
'metric_sklearn_name': None,
'metric_torch': None,
'model_dict': {},
'n_points': 1,
'n_samples': None,
'n_total': None,
'noise': False,
'num_workers': 0,
'ocba_delta': 0,
'oml_grace_period': None,
'optimizer': None,
'path': None,
'prep_model': None,
'prep_model_name': None,
'progress_file': None,
'save_model': False,
'scenario': None,
'seed': 123,
'show_batch_interval': 1000000,
'show_models': False,
'show_progress': True,
'shuffle': None,
'sigma': 0.0,
'spot_tensorboard_path': 'runs/spot_logs/015_maans14_2024-04-22_00-28-06',
'spot_writer': <torch.utils.tensorboard.writer.SummaryWriter object at 0x13fd13c90>,
'target_column': None,
'target_type': None,
'task': None,
'test': None,
'test_seed': 1234,
'test_size': 0.4,
'tolerance_x': 0,
'train': None,
'upper': array([1, 1]),
'var_name': None,
'var_type': ['num'],
'verbosity': 0,
'weight_coeff': 0.0,
'weights': 1.0,
'weights_entry': None}
<spotPython.spot.spot.Spot at 0x3b06c5250>
15.1.2 Results
spot_2.print_results()
min y: 1.801603872454505e-05
x0: 0.0019077911677074135
x1: 0.003791618596979743
[['x0', 0.0019077911677074135], ['x1', 0.003791618596979743]]
=True) spot_2.plot_progress(log_y
spot_2.surrogate.plot()
15.2 Example With Modified p
- We can use set
p
to a value other than2
to obtain a different Kriging model.
= surrogate_control_init(n_p=1,
surrogate_control =1.0)
p_val= spot.Spot(fun=fun,
spot_2_p1=fun_control,
fun_control=surrogate_control)
surrogate_control spot_2_p1.run()
spotPython tuning: 1.801603872454505e-05 [#######---] 73.33%
spotPython tuning: 1.801603872454505e-05 [########--] 80.00%
spotPython tuning: 1.801603872454505e-05 [#########-] 86.67%
spotPython tuning: 1.801603872454505e-05 [#########-] 93.33%
spotPython tuning: 1.801603872454505e-05 [##########] 100.00% Done...
{'CHECKPOINT_PATH': 'runs/saved_models/',
'DATASET_PATH': 'data/',
'PREFIX': '015',
'RESULTS_PATH': 'results/',
'TENSORBOARD_PATH': 'runs/',
'_L_in': None,
'_L_out': None,
'_torchmetric': None,
'accelerator': 'auto',
'converters': None,
'core_model': None,
'core_model_name': None,
'counter': 15,
'data': None,
'data_dir': './data',
'data_module': None,
'data_set': None,
'data_set_name': None,
'db_dict_name': None,
'design': None,
'device': None,
'devices': 1,
'enable_progress_bar': False,
'eval': None,
'fun_evals': 15,
'fun_repeats': 1,
'horizon': None,
'infill_criterion': 'y',
'k_folds': 3,
'log_graph': False,
'log_level': 50,
'loss_function': None,
'lower': array([-1, -1]),
'max_surrogate_points': 30,
'max_time': 1,
'metric_params': {},
'metric_river': None,
'metric_sklearn': None,
'metric_sklearn_name': None,
'metric_torch': None,
'model_dict': {},
'n_points': 1,
'n_samples': None,
'n_total': None,
'noise': False,
'num_workers': 0,
'ocba_delta': 0,
'oml_grace_period': None,
'optimizer': None,
'path': None,
'prep_model': None,
'prep_model_name': None,
'progress_file': None,
'save_model': False,
'scenario': None,
'seed': 123,
'show_batch_interval': 1000000,
'show_models': False,
'show_progress': True,
'shuffle': None,
'sigma': 0.0,
'spot_tensorboard_path': 'runs/spot_logs/015_maans14_2024-04-22_00-28-06',
'spot_writer': <torch.utils.tensorboard.writer.SummaryWriter object at 0x13fd13c90>,
'target_column': None,
'target_type': None,
'task': None,
'test': None,
'test_seed': 1234,
'test_size': 0.4,
'tolerance_x': 0,
'train': None,
'upper': array([1, 1]),
'var_name': None,
'var_type': ['num'],
'verbosity': 0,
'weight_coeff': 0.0,
'weights': 1.0,
'weights_entry': None}
<spotPython.spot.spot.Spot at 0x3b0e7eed0>
- The search progress of the optimization with the anisotropic model can be visualized:
=True) spot_2_p1.plot_progress(log_y
spot_2_p1.print_results()
min y: 1.801603872454505e-05
x0: 0.0019077911677074135
x1: 0.003791618596979743
[['x0', 0.0019077911677074135], ['x1', 0.003791618596979743]]
spot_2_p1.surrogate.plot()
15.2.1 Taking a Look at the p
Values
15.2.1.1 p
Values from the spot
Model
- We can check, which
p
values thespot
model has used: - The
p
values from the surrogate can be printed as follows:
spot_2_p1.surrogate.p
array([1.])
- Since the surrogate from the isotropic setting was stored as
spot_2
, we can also take a look at thetheta
value from this model:
spot_2.surrogate.p
array([2.])
15.3 Optimization of the p
Values
= surrogate_control_init(n_p=1,
surrogate_control =True)
optim_p= spot.Spot(fun=fun,
spot_2_pm=fun_control,
fun_control=surrogate_control)
surrogate_control spot_2_pm.run()
spotPython tuning: 1.893023485380876e-05 [#######---] 73.33%
spotPython tuning: 1.893023485380876e-05 [########--] 80.00%
spotPython tuning: 1.893023485380876e-05 [#########-] 86.67%
spotPython tuning: 1.893023485380876e-05 [#########-] 93.33%
spotPython tuning: 1.893023485380876e-05 [##########] 100.00% Done...
{'CHECKPOINT_PATH': 'runs/saved_models/',
'DATASET_PATH': 'data/',
'PREFIX': '015',
'RESULTS_PATH': 'results/',
'TENSORBOARD_PATH': 'runs/',
'_L_in': None,
'_L_out': None,
'_torchmetric': None,
'accelerator': 'auto',
'converters': None,
'core_model': None,
'core_model_name': None,
'counter': 15,
'data': None,
'data_dir': './data',
'data_module': None,
'data_set': None,
'data_set_name': None,
'db_dict_name': None,
'design': None,
'device': None,
'devices': 1,
'enable_progress_bar': False,
'eval': None,
'fun_evals': 15,
'fun_repeats': 1,
'horizon': None,
'infill_criterion': 'y',
'k_folds': 3,
'log_graph': False,
'log_level': 50,
'loss_function': None,
'lower': array([-1, -1]),
'max_surrogate_points': 30,
'max_time': 1,
'metric_params': {},
'metric_river': None,
'metric_sklearn': None,
'metric_sklearn_name': None,
'metric_torch': None,
'model_dict': {},
'n_points': 1,
'n_samples': None,
'n_total': None,
'noise': False,
'num_workers': 0,
'ocba_delta': 0,
'oml_grace_period': None,
'optimizer': None,
'path': None,
'prep_model': None,
'prep_model_name': None,
'progress_file': None,
'save_model': False,
'scenario': None,
'seed': 123,
'show_batch_interval': 1000000,
'show_models': False,
'show_progress': True,
'shuffle': None,
'sigma': 0.0,
'spot_tensorboard_path': 'runs/spot_logs/015_maans14_2024-04-22_00-28-06',
'spot_writer': <torch.utils.tensorboard.writer.SummaryWriter object at 0x13fd13c90>,
'target_column': None,
'target_type': None,
'task': None,
'test': None,
'test_seed': 1234,
'test_size': 0.4,
'tolerance_x': 0,
'train': None,
'upper': array([1, 1]),
'var_name': None,
'var_type': ['num'],
'verbosity': 0,
'weight_coeff': 0.0,
'weights': 1.0,
'weights_entry': None}
<spotPython.spot.spot.Spot at 0x3b12d2950>
=True) spot_2_pm.plot_progress(log_y
spot_2_pm.print_results()
min y: 1.893023485380876e-05
x0: 0.0017549984724977892
x1: 0.003981232876300906
[['x0', 0.0017549984724977892], ['x1', 0.003981232876300906]]
spot_2_pm.surrogate.plot()
spot_2_pm.surrogate.p
array([1.77398298])
15.4 Optimization of Multiple p
Values
= surrogate_control_init(n_p=2,
surrogate_control =True)
optim_p= spot.Spot(fun=fun,
spot_2_pmo=fun_control,
fun_control=surrogate_control)
surrogate_control spot_2_pmo.run()
spotPython tuning: 2.162397189403005e-05 [#######---] 73.33%
spotPython tuning: 2.162397189403005e-05 [########--] 80.00%
spotPython tuning: 2.162397189403005e-05 [#########-] 86.67%
spotPython tuning: 2.162397189403005e-05 [#########-] 93.33%
spotPython tuning: 2.162397189403005e-05 [##########] 100.00% Done...
{'CHECKPOINT_PATH': 'runs/saved_models/',
'DATASET_PATH': 'data/',
'PREFIX': '015',
'RESULTS_PATH': 'results/',
'TENSORBOARD_PATH': 'runs/',
'_L_in': None,
'_L_out': None,
'_torchmetric': None,
'accelerator': 'auto',
'converters': None,
'core_model': None,
'core_model_name': None,
'counter': 15,
'data': None,
'data_dir': './data',
'data_module': None,
'data_set': None,
'data_set_name': None,
'db_dict_name': None,
'design': None,
'device': None,
'devices': 1,
'enable_progress_bar': False,
'eval': None,
'fun_evals': 15,
'fun_repeats': 1,
'horizon': None,
'infill_criterion': 'y',
'k_folds': 3,
'log_graph': False,
'log_level': 50,
'loss_function': None,
'lower': array([-1, -1]),
'max_surrogate_points': 30,
'max_time': 1,
'metric_params': {},
'metric_river': None,
'metric_sklearn': None,
'metric_sklearn_name': None,
'metric_torch': None,
'model_dict': {},
'n_points': 1,
'n_samples': None,
'n_total': None,
'noise': False,
'num_workers': 0,
'ocba_delta': 0,
'oml_grace_period': None,
'optimizer': None,
'path': None,
'prep_model': None,
'prep_model_name': None,
'progress_file': None,
'save_model': False,
'scenario': None,
'seed': 123,
'show_batch_interval': 1000000,
'show_models': False,
'show_progress': True,
'shuffle': None,
'sigma': 0.0,
'spot_tensorboard_path': 'runs/spot_logs/015_maans14_2024-04-22_00-28-06',
'spot_writer': <torch.utils.tensorboard.writer.SummaryWriter object at 0x13fd13c90>,
'target_column': None,
'target_type': None,
'task': None,
'test': None,
'test_seed': 1234,
'test_size': 0.4,
'tolerance_x': 0,
'train': None,
'upper': array([1, 1]),
'var_name': None,
'var_type': ['num'],
'verbosity': 0,
'weight_coeff': 0.0,
'weights': 1.0,
'weights_entry': None}
<spotPython.spot.spot.Spot at 0x3b0f64490>
=True) spot_2_pmo.plot_progress(log_y
spot_2_pmo.print_results()
min y: 2.162397189403005e-05
x0: 0.0018245082309241386
x1: 0.00427728203527896
[['x0', 0.0018245082309241386], ['x1', 0.00427728203527896]]
spot_2_pmo.surrogate.plot()
spot_2_pmo.surrogate.p
array([1.09037777, 1.76346322])
15.5 Exercises
15.5.1 fun_branin
- Describe the function.
- The input dimension is
2
. The search range is \(-5 \leq x_1 \leq 10\) and \(0 \leq x_2 \leq 15\).
- The input dimension is
- Compare the results from
spotPython
runs with different options forp
. - Modify the termination criterion: instead of the number of evaluations (which is specified via
fun_evals
), the time should be used as the termination criterion. This can be done as follows (max_time=1
specifies a run time of one minute):
=inf,
fun_evals=1, max_time
15.5.2 fun_sin_cos
- Describe the function.
- The input dimension is
2
. The search range is \(-2\pi \leq x_1 \leq 2\pi\) and \(-2\pi \leq x_2 \leq 2\pi\).
- The input dimension is
- Compare the results from
spotPython
run a) with isotropic and b) anisotropic surrogate models. - Modify the termination criterion (
max_time
instead offun_evals
) as described forfun_branin
.
15.5.3 fun_runge
- Describe the function.
- The input dimension is
2
. The search range is \(-5 \leq x_1 \leq 5\) and \(-5 \leq x_2 \leq 5\).
- The input dimension is
- Compare the results from
spotPython
runs with different options forp
. - Modify the termination criterion (
max_time
instead offun_evals
) as described forfun_branin
.
15.5.4 fun_wingwt
- Describe the function.
- The input dimension is
10
. The search ranges are between 0 and 1 (values are mapped internally to their natural bounds).
- The input dimension is
- Compare the results from
spotPython
runs with different options forp
. - Modify the termination criterion (
max_time
instead offun_evals
) as described forfun_branin
.
15.6 Jupyter Notebook
Note
- The Jupyter-Notebook of this lecture is available on GitHub in the Hyperparameter-Tuning-Cookbook Repository