[[33]{.chapter-number}  [HPT: sklearn SVR on Regression Data]{.chapter-title}]{#sec-hpt-sklearn-svR .quarto-section-identifier}

doi:10.48550/arXiv.2307.10262

33 HPT: sklearn SVR on Regression Data

This chapter is a tutorial for the Hyperparameter Tuning (HPT) of a sklearn SVR model on a regression dataset.

33.1 Step 1: Setup

Before we consider the detailed experimental setup, we select the parameters that affect run time, initial design size and the device that is used.

Caution: Run time and initial design size should be increased for real experiments

MAX_TIME is set to one minute for demonstration purposes. For real experiments, this should be increased to at least 1 hour.
INIT_SIZE is set to 5 for demonstration purposes. For real experiments, this should be increased to at least 10.

MAX_TIME = 1
INIT_SIZE = 20
PREFIX = "18"

33.2 Step 2: Initialization of the Empty `fun_control` Dictionary

spotpython supports the visualization of the hyperparameter tuning process with TensorBoard. The following example shows how to use TensorBoard with spotpython. The fun_control dictionary is the central data structure that is used to control the optimization process. It is initialized as follows:

from spotpython.utils.init import fun_control_init
from spotpython.hyperparameters.values import set_control_key_value
from spotpython.utils.eda import print_res_table
fun_control = fun_control_init(
    PREFIX=PREFIX,
    TENSORBOARD_CLEAN=True,
    max_time=MAX_TIME,
    fun_evals=inf,
    tolerance_x = np.sqrt(np.spacing(1)))

Moving TENSORBOARD_PATH: runs/ to TENSORBOARD_PATH_OLD: runs_OLD/runs_2025_10_19_20_45_58_0

Tip: TensorBoard

Since the spot_tensorboard_path argument is not None, which is the default, spotpython will log the optimization process in the TensorBoard folder.
The TENSORBOARD_CLEAN argument is set to True to archive the TensorBoard folder if it already exists. This is useful if you want to start a hyperparameter tuning process from scratch. If you want to continue a hyperparameter tuning process, set TENSORBOARD_CLEAN to False. Then the TensorBoard folder will not be archived and the old and new TensorBoard files will shown in the TensorBoard dashboard.

33.3 Step 3: SKlearn Load Data (Classification)

Randomly generate classification data. Here, we use similar data as in Comparison of kernel ridge regression and SVR.

import numpy as np

rng = np.random.RandomState(42)

X = 5 * rng.rand(10, 1)
y = np.sin(1/X).ravel()*np.cos(X).ravel()

# Add noise to targets
y[::5] += 3 * (0.5 - rng.rand(X.shape[0] // 5))

X_plot = np.linspace(0, 5, 100000)[:, None]

import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split

n_features = 1
target_column = "y"
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=42
)
train = pd.DataFrame(np.hstack((X_train, y_train.reshape(-1, 1))))
test = pd.DataFrame(np.hstack((X_test, y_test.reshape(-1, 1))))
train.columns = [f"x{i}" for i in range(1, n_features+1)] + [target_column]
test.columns = [f"x{i}" for i in range(1, n_features+1)] + [target_column]
train.head()

	x1	y
0	1.872701	1.286910
1	4.330881	-0.085207
2	3.659970	-0.234389
3	3.540363	-0.256848
4	0.780093	0.681389

n_samples = len(train)
# add the dataset to the fun_control
fun_control.update({"data": None, # dataset,
               "train": train,
               "test": test,
               "n_samples": n_samples,
               "target_column": target_column})

33.4 Step 4: Specification of the Preprocessing Model

Data preprocesssing can be very simple, e.g., you can ignore it. Then you would choose the prep_model “None”:

prep_model = None
fun_control.update({"prep_model": prep_model})

A default approach for numerical data is the StandardScaler (mean 0, variance 1). This can be selected as follows:

from sklearn.preprocessing import StandardScaler
prep_model = StandardScaler
fun_control.update({"prep_model": prep_model})

Even more complicated pre-processing steps are possible, e.g., the follwing pipeline:

categorical_columns = []
one_hot_encoder = OneHotEncoder(handle_unknown="ignore", sparse_output=False)
prep_model = ColumnTransformer(
         transformers=[
             ("categorical", one_hot_encoder, categorical_columns),
         ],
         remainder=StandardScaler,
     )

33.5 Step 5: Select Model (`algorithm`) and `core_model_hyper_dict`

The selection of the algorithm (ML model) that should be tuned is done by specifying the its name from the sklearn implementation. For example, the SVC support vector machine classifier is selected as follows:

from spotpython.hyperparameters.values import add_core_model_to_fun_control
from spotpython.hyperdict.sklearn_hyper_dict import SklearnHyperDict
from sklearn.svm import SVR
add_core_model_to_fun_control(core_model=SVR,
                              fun_control=fun_control,
                              hyper_dict=SklearnHyperDict,
                              filename=None)

Now fun_control has the information from the JSON file. The corresponding entries for the core_model class are shown below.

fun_control['core_model_hyper_dict']

{'C': {'type': 'float',
  'default': 1.0,
  'transform': 'None',
  'lower': 0.1,
  'upper': 10.0},
 'kernel': {'levels': ['linear', 'poly', 'rbf', 'sigmoid'],
  'type': 'factor',
  'default': 'rbf',
  'transform': 'None',
  'core_model_parameter_type': 'str',
  'lower': 0,
  'upper': 3},
 'degree': {'type': 'int',
  'default': 3,
  'transform': 'None',
  'lower': 3,
  'upper': 3},
 'gamma': {'levels': ['scale', 'auto'],
  'type': 'factor',
  'default': 'scale',
  'transform': 'None',
  'core_model_parameter_type': 'str',
  'lower': 0,
  'upper': 1},
 'coef0': {'type': 'float',
  'default': 0.0,
  'transform': 'None',
  'lower': 0.0,
  'upper': 0.0},
 'epsilon': {'type': 'float',
  'default': 0.1,
  'transform': 'None',
  'lower': 0.01,
  'upper': 1.0},
 'shrinking': {'levels': [0, 1],
  'type': 'factor',
  'default': 0,
  'transform': 'None',
  'core_model_parameter_type': 'bool',
  'lower': 0,
  'upper': 1},
 'tol': {'type': 'float',
  'default': 0.001,
  'transform': 'None',
  'lower': 0.0001,
  'upper': 0.01},
 'cache_size': {'type': 'float',
  'default': 200,
  'transform': 'None',
  'lower': 100,
  'upper': 400}}

sklearn Model Selection

The following sklearn models are supported by default:

RidgeCV
RandomForestClassifier
SVC
SVR
LogisticRegression
KNeighborsClassifier
GradientBoostingClassifier
GradientBoostingRegressor
ElasticNet

They can be imported as follows:

from sklearn.linear_model import RidgeCV
from sklearn.ensemble import RandomForestClassifier
from sklearn.svm import SVC
from sklearn.svm import SVR
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.linear_model import ElasticNet

33.6 Step 6: Modify `hyper_dict` Hyperparameters for the Selected Algorithm aka `core_model`

spotpython provides functions for modifying the hyperparameters, their bounds and factors as well as for activating and de-activating hyperparameters without re-compilation of the Python source code. These functions were described in Section 12.19.1.

33.6.1 Modify hyperparameter of type numeric and integer (boolean)

Numeric and boolean values can be modified using the modify_hyper_parameter_bounds method.

sklearn Model Hyperparameters

The hyperparameters of the sklearn SVC model are described in the sklearn documentation.

For example, to change the tol hyperparameter of the SVC model to the interval [1e-5, 1e-3], the following code can be used:

from spotpython.hyperparameters.values import modify_hyper_parameter_bounds
modify_hyper_parameter_bounds(fun_control, "tol", bounds=[1e-5, 1e-3])
modify_hyper_parameter_bounds(fun_control, "epsilon", bounds=[0.1, 1.0])
# modify_hyper_parameter_bounds(fun_control, "degree", bounds=[2, 5])
fun_control["core_model_hyper_dict"]["tol"]

{'type': 'float',
 'default': 0.001,
 'transform': 'None',
 'lower': 1e-05,
 'upper': 0.001}

33.6.2 Modify hyperparameter of type factor

Factors can be modified with the modify_hyper_parameter_levels function. For example, to exclude the sigmoid kernel from the tuning, the kernel hyperparameter of the SVR model can be modified as follows:

from spotpython.hyperparameters.values import modify_hyper_parameter_levels
# modify_hyper_parameter_levels(fun_control, "kernel", ["poly", "rbf"])
modify_hyper_parameter_levels(fun_control, "kernel", ["rbf"])
fun_control["core_model_hyper_dict"]["kernel"]

{'levels': ['rbf'],
 'type': 'factor',
 'default': 'rbf',
 'transform': 'None',
 'core_model_parameter_type': 'str',
 'lower': 0,
 'upper': 0}

33.6.3 Optimizers

Optimizers are described in Section 15.2.

33.7 Step 7: Selection of the Objective (Loss) Function

There are two metrics:

metric_river is used for the river based evaluation via eval_oml_iter_progressive.
metric_sklearn is used for the sklearn based evaluation.

from sklearn.metrics import mean_absolute_error, accuracy_score, roc_curve, roc_auc_score, log_loss, mean_squared_error
fun_control.update({
               "metric_sklearn": mean_squared_error,
               "weights": 1.0,
               })

metric_sklearn: Minimization and Maximization

Because the metric_sklearn is used for the sklearn based evaluation, it is important to know whether the metric should be minimized or maximized.
The weights parameter is used to indicate whether the metric should be minimized or maximized.
If weights is set to -1.0, the metric is maximized.
If weights is set to 1.0, the metric is minimized, e.g., weights = 1.0 for mean_absolute_error, or weights = -1.0 for roc_auc_score.

33.7.1 Predict Classes or Class Probabilities

If the key "predict_proba" is set to True, the class probabilities are predicted. False is the default, i.e., the classes are predicted.

fun_control.update({
               "predict_proba": False,
               })

33.8 Step 8: Calling the SPOT Function

33.8.1 The Objective Function

The objective function is selected next. It implements an interface from sklearn’s training, validation, and testing methods to spotpython.

from spotpython.fun.hypersklearn import HyperSklearn
fun = HyperSklearn().fun_sklearn

The following code snippet shows how to get the default hyperparameters as an array, so that they can be passed to the Spot function.

from spotpython.hyperparameters.values import get_default_hyperparameters_as_array
X_start = get_default_hyperparameters_as_array(fun_control)

33.8.2 Run the `Spot` Optimizer

The class Spot [SOURCE] is the hyperparameter tuning workhorse. It is initialized with the following parameters:

fun: the objective function
fun_control: the dictionary with the control parameters for the objective function
design: the experimental design
design_control: the dictionary with the control parameters for the experimental design
surrogate: the surrogate model
surrogate_control: the dictionary with the control parameters for the surrogate model
optimizer: the optimizer
optimizer_control: the dictionary with the control parameters for the optimizer

Note: Total run time

The total run time may exceed the specified max_time, because the initial design (here: init_size = INIT_SIZE as specified above) is always evaluated, even if this takes longer than max_time.

from spotpython.utils.init import design_control_init, surrogate_control_init
design_control = design_control_init()
set_control_key_value(control_dict=design_control,
                        key="init_size",
                        value=INIT_SIZE,
                        replace=True)

surrogate_control = surrogate_control_init(method="regression")
from spotpython.spot import Spot
spot_tuner = Spot(fun=fun,
                   fun_control=fun_control,
                   design_control=design_control,
                   surrogate_control=surrogate_control)
spot_tuner.run(X_start=X_start)

Anisotropic model: n_theta set to 6
Anisotropic model: n_theta set to 6
spotpython tuning: 0.11831336203046443 [----------] 0.81% 
Anisotropic model: n_theta set to 6
spotpython tuning: 0.11831336203046443 [----------] 1.30% 
Anisotropic model: n_theta set to 6
spotpython tuning: 0.11831336203046443 [----------] 2.06% 
Anisotropic model: n_theta set to 6
spotpython tuning: 0.11831336203046443 [----------] 2.95% 
Anisotropic model: n_theta set to 6
spotpython tuning: 0.11831336203046443 [----------] 3.47% 
Anisotropic model: n_theta set to 6
spotpython tuning: 0.11831336203046443 [----------] 3.94% 
Anisotropic model: n_theta set to 6
spotpython tuning: 0.11831336203046443 [----------] 4.49% 
Anisotropic model: n_theta set to 6
spotpython tuning: 0.09468362738999862 [#---------] 5.07% 
Anisotropic model: n_theta set to 6
spotpython tuning: 0.08423427702103133 [#---------] 5.82% 
Anisotropic model: n_theta set to 6
spotpython tuning: 0.08423427702103133 [#---------] 6.61% 
Anisotropic model: n_theta set to 6
spotpython tuning: 0.08423427702103133 [#---------] 8.28% 
Anisotropic model: n_theta set to 6
spotpython tuning: 0.08423427702103133 [#---------] 9.03% 
Anisotropic model: n_theta set to 6
spotpython tuning: 0.08423427702103133 [#---------] 9.67% 
Anisotropic model: n_theta set to 6
spotpython tuning: 0.08423427702103133 [#---------] 11.25% 
Anisotropic model: n_theta set to 6
spotpython tuning: 0.08423427702103133 [#---------] 12.80% 
Anisotropic model: n_theta set to 6
spotpython tuning: 0.08423427702103133 [#---------] 14.37% 
Anisotropic model: n_theta set to 6
spotpython tuning: 0.08423427702103133 [##--------] 16.06% 
Anisotropic model: n_theta set to 6
spotpython tuning: 0.08423427702103133 [##--------] 17.45% 
Anisotropic model: n_theta set to 6
spotpython tuning: 0.08423427702103133 [##--------] 18.96% 
Anisotropic model: n_theta set to 6
spotpython tuning: 0.08423427702103133 [##--------] 20.48% 
Anisotropic model: n_theta set to 6
spotpython tuning: 0.08423427702103133 [##--------] 21.73% 
Anisotropic model: n_theta set to 6
spotpython tuning: 0.08423427702103133 [##--------] 23.40% 
Anisotropic model: n_theta set to 6
spotpython tuning: 0.08423427702103133 [##--------] 24.93% 
Anisotropic model: n_theta set to 6
spotpython tuning: 0.08423427702103133 [###-------] 25.55% 
Anisotropic model: n_theta set to 6
spotpython tuning: 0.08423427702103133 [###-------] 27.00% 
Anisotropic model: n_theta set to 6
spotpython tuning: 0.08423427702103133 [###-------] 28.65% 
Anisotropic model: n_theta set to 6
spotpython tuning: 0.08423427702103133 [###-------] 30.44% 
Anisotropic model: n_theta set to 6
spotpython tuning: 0.08423427702103133 [###-------] 31.57% 
Anisotropic model: n_theta set to 6
spotpython tuning: 0.08423427702103133 [###-------] 32.95% 
Anisotropic model: n_theta set to 6
spotpython tuning: 0.08423427702103133 [###-------] 34.12% 
Anisotropic model: n_theta set to 6
spotpython tuning: 0.08423427702103133 [###-------] 34.87% 
Anisotropic model: n_theta set to 6
spotpython tuning: 0.08423427702103133 [####------] 36.16% 
Anisotropic model: n_theta set to 6
spotpython tuning: 0.08423427702103133 [####------] 37.33% 
Anisotropic model: n_theta set to 6
spotpython tuning: 0.08423427702103133 [####------] 38.66% 
Anisotropic model: n_theta set to 6
spotpython tuning: 0.08423427702103133 [####------] 39.90% 
Anisotropic model: n_theta set to 6
spotpython tuning: 0.08423427702103133 [####------] 41.48% 
Anisotropic model: n_theta set to 6
spotpython tuning: 0.08423427702103133 [####------] 42.97% 
Anisotropic model: n_theta set to 6
spotpython tuning: 0.08423427702103133 [####------] 44.59% 
Anisotropic model: n_theta set to 6
spotpython tuning: 0.08423427702103133 [#####-----] 46.29% 
Anisotropic model: n_theta set to 6
spotpython tuning: 0.08423427702103133 [#####-----] 47.73% 
Anisotropic model: n_theta set to 6
spotpython tuning: 0.08423427702103133 [#####-----] 49.37% 
Anisotropic model: n_theta set to 6
spotpython tuning: 0.08423427702103133 [#####-----] 50.86% 
Anisotropic model: n_theta set to 6
spotpython tuning: 0.08423427702103133 [#####-----] 52.29% 
Anisotropic model: n_theta set to 6
spotpython tuning: 0.08423427702103133 [#####-----] 53.69% 
Anisotropic model: n_theta set to 6
spotpython tuning: 0.08423427702103133 [######----] 55.27% 
Anisotropic model: n_theta set to 6
spotpython tuning: 0.08423427702103133 [######----] 56.62% 
Anisotropic model: n_theta set to 6
spotpython tuning: 0.08423427702103133 [######----] 58.01% 
Anisotropic model: n_theta set to 6
spotpython tuning: 0.08423427702103133 [######----] 59.45% 
Anisotropic model: n_theta set to 6
spotpython tuning: 0.08423427702103133 [######----] 60.91% 
Anisotropic model: n_theta set to 6
spotpython tuning: 0.08423427702103133 [######----] 62.21% 
Anisotropic model: n_theta set to 6
spotpython tuning: 0.08423427702103133 [######----] 63.68% 
Anisotropic model: n_theta set to 6
spotpython tuning: 0.08423427702103133 [######----] 64.93% 
Anisotropic model: n_theta set to 6
spotpython tuning: 0.08423427702103133 [#######---] 66.34% 
Anisotropic model: n_theta set to 6
spotpython tuning: 0.08423427702103133 [#######---] 67.97% 
Anisotropic model: n_theta set to 6
spotpython tuning: 0.08423427702103133 [#######---] 69.76% 
Anisotropic model: n_theta set to 6
spotpython tuning: 0.08423427702103133 [#######---] 71.19% 
Anisotropic model: n_theta set to 6
spotpython tuning: 0.08423427702103133 [#######---] 72.63% 
Anisotropic model: n_theta set to 6
spotpython tuning: 0.08423427702103133 [#######---] 73.92% 
Anisotropic model: n_theta set to 6
spotpython tuning: 0.08423427702103133 [########--] 75.58% 
Anisotropic model: n_theta set to 6
spotpython tuning: 0.08423427702103133 [########--] 76.99% 
Anisotropic model: n_theta set to 6
spotpython tuning: 0.08423427702103133 [########--] 78.48% 
Anisotropic model: n_theta set to 6
spotpython tuning: 0.08423427702103133 [########--] 79.62% 
Anisotropic model: n_theta set to 6
spotpython tuning: 0.08423427702103133 [########--] 80.61% 
Anisotropic model: n_theta set to 6
spotpython tuning: 0.08423427702103133 [########--] 81.84% 
Anisotropic model: n_theta set to 6
spotpython tuning: 0.08423427702103133 [########--] 83.43% 
Anisotropic model: n_theta set to 6
spotpython tuning: 0.08423427702103133 [#########-] 85.12% 
Anisotropic model: n_theta set to 6
spotpython tuning: 0.08423427702103133 [#########-] 86.24% 
Anisotropic model: n_theta set to 6
spotpython tuning: 0.08423427702103133 [#########-] 87.39% 
Anisotropic model: n_theta set to 6
spotpython tuning: 0.08423427702103133 [#########-] 88.91% 
Anisotropic model: n_theta set to 6
spotpython tuning: 0.08423427702103133 [#########-] 90.40% 
Anisotropic model: n_theta set to 6
spotpython tuning: 0.08423427702103133 [#########-] 91.77% 
Anisotropic model: n_theta set to 6
spotpython tuning: 0.08423427702103133 [#########-] 93.31% 
Anisotropic model: n_theta set to 6
spotpython tuning: 0.08423427702103133 [#########-] 94.61% 
Anisotropic model: n_theta set to 6
spotpython tuning: 0.08423427702103133 [##########] 96.02% 
Anisotropic model: n_theta set to 6
spotpython tuning: 0.08423427702103133 [##########] 97.31% 
Anisotropic model: n_theta set to 6
spotpython tuning: 0.08423427702103133 [##########] 98.73% 
Anisotropic model: n_theta set to 6
spotpython tuning: 0.08423427702103133 [##########] 100.00% Done...

Experiment saved to 18_res.pkl

<spotpython.spot.spot.Spot at 0x15d2c27b0>

33.8.3 TensorBoard

Now we can start TensorBoard in the background with the following command, where ./runs is the default directory for the TensorBoard log files:

tensorboard --logdir="./runs"

from spotpython.utils.init import get_tensorboard_path
get_tensorboard_path(fun_control)

'runs/'

After the hyperparameter tuning run is finished, the progress of the hyperparameter tuning can be visualized. The black points represent the performace values (score or metric) of hyperparameter configurations from the initial design, whereas the red points represents the hyperparameter configurations found by the surrogate model based optimization.

spot_tuner.plot_progress(log_y=True)

Results can also be printed in tabular form.

print_res_table(spot_tuner)

| name       | type   | default   |   lower |   upper | tuned                 | transform   |   importance | stars   |
|------------|--------|-----------|---------|---------|-----------------------|-------------|--------------|---------|
| C          | float  | 1.0       |     0.1 |    10.0 | 8.878190272661849     | None        |         0.21 | .       |
| kernel     | factor | rbf       |     0.0 |     0.0 | rbf                   | None        |         0.00 |         |
| degree     | int    | 3         |     3.0 |     3.0 | 3.0                   | None        |         0.00 |         |
| gamma      | factor | scale     |     0.0 |     1.0 | auto                  | None        |         0.02 |         |
| coef0      | float  | 0.0       |     0.0 |     0.0 | 0.0                   | None        |         0.00 |         |
| epsilon    | float  | 0.1       |     0.1 |     1.0 | 0.1                   | None        |       100.00 | ***     |
| shrinking  | factor | 0         |     0.0 |     1.0 | 0                     | None        |         0.02 |         |
| tol        | float  | 0.001     |   1e-05 |   0.001 | 0.0009278346625092568 | None        |         0.02 |         |
| cache_size | float  | 200.0     |   100.0 |   400.0 | 186.25855830293085    | None        |         0.02 |         |

A histogram can be used to visualize the most important hyperparameters.

spot_tuner.plot_importance(threshold=0.0025)

33.9 Get Default Hyperparameters

The default hyperparameters, which will be used for a comparion with the tuned hyperparameters, can be obtained with the following commands:

from spotpython.hyperparameters.values import get_one_core_model_from_X
from spotpython.hyperparameters.values import get_default_hyperparameters_as_array
X_start = get_default_hyperparameters_as_array(fun_control)
model_default = get_one_core_model_from_X(X_start, fun_control, default=True)
model_default

SVR(cache_size=200.0, shrinking=False)

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

33.10 Get SPOT Results

In a similar way, we can obtain the hyperparameters found by spotpython.

from spotpython.hyperparameters.values import get_one_core_model_from_X
X_tuned = spot_tuner.to_all_dim(spot_tuner.min_X.reshape(1,-1))
model_spot = get_one_core_model_from_X(X_tuned, fun_control)

33.10.1 Plot: Compare Predictions

model_default.fit(X_train, y_train)
y_default = model_default.predict(X_plot)

model_spot.fit(X_train, y_train)
y_spot = model_spot.predict(X_plot)

import matplotlib.pyplot as plt
plt.scatter(X[:100], y[:100], c="orange", label="data", zorder=1, edgecolors=(0, 0, 0))
plt.plot(
    X_plot,
    y_default,
    c="red",
    label="Default SVR")

plt.plot(
    X_plot, y_spot, c="blue", label="SPOT SVR")

plt.xlabel("data")
plt.ylabel("target")
plt.title("SVR")
_ = plt.legend()

33.10.2 Detailed Hyperparameter Plots

spot_tuner.plot_important_hyperparameter_contour(filename=None)

C:  0.21254707556577718
gamma:  0.018998262435359903
epsilon:  100.0
shrinking:  0.018998262435359903
tol:  0.019322387866414794
cache_size:  0.018998262435359903

33.10.3 Parallel Coordinates Plot

spot_tuner.parallel_plot()