import os
from math import inf
import warnings
warnings.filterwarnings("ignore")43 Early Stopping Explained: HPT with spotpython and PyTorch Lightning for the Diabetes Data Set
In this section, we will show how early stopping can be integrated into the PyTorch Lightning training workflow for a regression task. We will use the setting described in Chapter 42, i.e., the Diabetes data set, which is provided by spotpython, and the HyperLight class to define the objective function.
43.1 The Basic Setting
After importing the necessary libraries, the fun_control dictionary is set up via the fun_control_init function. The fun_control dictionary contains the same parameters as in Chapter 42, i.e., it contains the following parameters:
PREFIX: a unique identifier for the experimentfun_evals: the number of function evaluationsmax_time: the maximum run time in minutesdata_set: the data set. Here we use theDiabetesdata set that is provided byspotpython.core_model_name: the class name of the neural network model. This neural network model is provided byspotpython.hyperdict: the hyperparameter dictionary. This dictionary is used to define the hyperparameters of the neural network model. It is also provided byspotpython._L_in: the number of input features. Since theDiabetesdata set has 10 features,_L_inis set to 10._L_out: the number of output features. Since we want to predict a single value,_L_outis set to 1.
In addition, the fun_control dictionary contains the following parameters that are specific to the early-stopping mechanism:
divergence_threshold: Stop training as soon as the monitored quantity becomes worse than this threshold.check_finite: When set True, stops training when the monitor becomes NaN or infinite.stopping_threshold: Stop training immediately once the monitored quantity reaches this threshold.
divergence_threshold
- We will set the
divergence_thresholdto 25,000, because good values are in a range around 15,000. This means that the training will be stopped if the monitored quantity becomes worse than this threshold.
Furthermore, the patience parameter can be used as a hyperparameter to control the early stopping mechanism. It defines how many validation checks with no improvement are allowed before training is stopped.
patience
It must be noted that the patience parameter counts the number of validation checks with no improvement, and not the number of training epochs. Therefore, with parameters check_val_every_n_epoch=10 and patience=3, the trainer will perform at least 40 training epochs before being stopped.
The HyperLight class is used to define the objective function fun. It connects the PyTorch and the spotpython methods and is provided by spotpython.
from spotpython.data.diabetes import Diabetes
from spotpython.hyperdict.light_hyper_dict import LightHyperDict
from spotpython.fun.hyperlight import HyperLight
from spotpython.utils.init import (fun_control_init, surrogate_control_init, design_control_init)
from spotpython.utils.eda import print_exp_table, print_res_table
from spotpython.spot import Spot
from spotpython.utils.file import get_experiment_filename
PREFIX="601-early_stopping"
data_set = Diabetes()
fun_control = fun_control_init(
PREFIX=PREFIX,
fun_evals=inf,
TENSORBOARD_CLEAN=True,
tensorboard_log=True,
max_time=1,
data_set = data_set,
core_model_name="light.regression.NNLinearRegressor",
hyperdict=LightHyperDict,
divergence_threshold=25_000,
_L_in=10,
_L_out=1)
fun = HyperLight().funMoving TENSORBOARD_PATH: runs/ to TENSORBOARD_PATH_OLD: runs_OLD/runs_2025_07_25_21_54_18_0
Created spot_tensorboard_path: runs/spot_logs/601-early_stopping_p040025_2025-07-25_21-54-18 for SummaryWriter()
module_name: light
submodule_name: regression
model_name: NNLinearRegressorThe method set_hyperparameter allows the user to modify default hyperparameter settings. Here we modify some hyperparameters to keep the model small and to decrease the tuning time.
from spotpython.hyperparameters.values import set_hyperparameter
set_hyperparameter(fun_control, "optimizer", [ "Adadelta", "Adam", "Adamax"])
set_hyperparameter(fun_control, "l1", [3,4])
set_hyperparameter(fun_control, "epochs", [3,7])
set_hyperparameter(fun_control, "batch_size", [4,11])
set_hyperparameter(fun_control, "dropout_prob", [0.0, 0.025])
set_hyperparameter(fun_control, "patience", [2,3])
design_control = design_control_init(init_size=10)
print_exp_table(fun_control)| name | type | default | lower | upper | transform |
|----------------|--------|-----------|---------|---------|-----------------------|
| l1 | int | 3 | 3 | 4 | transform_power_2_int |
| epochs | int | 4 | 3 | 7 | transform_power_2_int |
| batch_size | int | 4 | 4 | 11 | transform_power_2_int |
| act_fn | factor | ReLU | 0 | 5 | None |
| optimizer | factor | SGD | 0 | 2 | None |
| dropout_prob | float | 0.01 | 0 | 0.025 | None |
| lr_mult | float | 1.0 | 0.1 | 10 | None |
| patience | int | 2 | 2 | 3 | transform_power_2_int |
| batch_norm | factor | 0 | 0 | 1 | None |
| initialization | factor | Default | 0 | 4 | None |Finally, a Spot object is created. Calling the method run() starts the hyperparameter tuning process.
S = Spot(fun=fun,fun_control=fun_control, design_control=design_control)
S.run()train_model result: {'val_loss': 23075.09765625, 'hp_metric': 23075.09765625}train_model result: {'val_loss': 3466.6259765625, 'hp_metric': 3466.6259765625}train_model result: {'val_loss': 44729220.0, 'hp_metric': 44729220.0}train_model result: {'val_loss': 24005.150390625, 'hp_metric': 24005.150390625}train_model result: {'val_loss': 22915.138671875, 'hp_metric': 22915.138671875}train_model result: {'val_loss': 4215.99072265625, 'hp_metric': 4215.99072265625}train_model result: {'val_loss': 20590.515625, 'hp_metric': 20590.515625}train_model result: {'val_loss': 4035.48291015625, 'hp_metric': 4035.48291015625}train_model result: {'val_loss': 20699.109375, 'hp_metric': 20699.109375}
train_model result: {'val_loss': 23858.537109375, 'hp_metric': 23858.537109375}train_model result: {'val_loss': 26152.673828125, 'hp_metric': 26152.673828125}
spotpython tuning: 3466.6259765625 [----------] 1.08% train_model result: {'val_loss': 23378.26171875, 'hp_metric': 23378.26171875}
spotpython tuning: 3466.6259765625 [----------] 4.24% train_model result: {'val_loss': 22719.28125, 'hp_metric': 22719.28125}
spotpython tuning: 3466.6259765625 [#---------] 5.28% train_model result: {'val_loss': 13361.490234375, 'hp_metric': 13361.490234375}
spotpython tuning: 3466.6259765625 [#---------] 10.94% train_model result: {'val_loss': 23906.50390625, 'hp_metric': 23906.50390625}
spotpython tuning: 3466.6259765625 [#---------] 12.64% train_model result: {'val_loss': 22906.419921875, 'hp_metric': 22906.419921875}
spotpython tuning: 3466.6259765625 [#---------] 14.56% train_model result: {'val_loss': 23614.091796875, 'hp_metric': 23614.091796875}
spotpython tuning: 3466.6259765625 [##--------] 16.43% train_model result: {'val_loss': 23973.421875, 'hp_metric': 23973.421875}
spotpython tuning: 3466.6259765625 [##--------] 18.30% train_model result: {'val_loss': 5899.0517578125, 'hp_metric': 5899.0517578125}
spotpython tuning: 3466.6259765625 [##--------] 21.66% train_model result: {'val_loss': 23639.759765625, 'hp_metric': 23639.759765625}
spotpython tuning: 3466.6259765625 [##--------] 24.96% train_model result: {'val_loss': 22265.041015625, 'hp_metric': 22265.041015625}
spotpython tuning: 3466.6259765625 [###-------] 28.17% train_model result: {'val_loss': 21614.546875, 'hp_metric': 21614.546875}
spotpython tuning: 3466.6259765625 [###-------] 30.42% train_model result: {'val_loss': 23080.599609375, 'hp_metric': 23080.599609375}
spotpython tuning: 3466.6259765625 [###-------] 33.01% train_model result: {'val_loss': 3888.931640625, 'hp_metric': 3888.931640625}
spotpython tuning: 3466.6259765625 [####------] 35.44% train_model result: {'val_loss': 24372146.0, 'hp_metric': 24372146.0}
spotpython tuning: 3466.6259765625 [####------] 37.38% train_model result: {'val_loss': 23751.765625, 'hp_metric': 23751.765625}
spotpython tuning: 3466.6259765625 [####------] 41.22% train_model result: {'val_loss': 23913.1015625, 'hp_metric': 23913.1015625}
spotpython tuning: 3466.6259765625 [#####-----] 48.99% train_model result: {'val_loss': 23230.490234375, 'hp_metric': 23230.490234375}
spotpython tuning: 3466.6259765625 [#####-----] 53.95% train_model result: {'val_loss': 23997.240234375, 'hp_metric': 23997.240234375}
spotpython tuning: 3466.6259765625 [######----] 58.38% train_model result: {'val_loss': 149062656.0, 'hp_metric': 149062656.0}
spotpython tuning: 3466.6259765625 [######----] 61.82% train_model result: {'val_loss': 23505.140625, 'hp_metric': 23505.140625}
spotpython tuning: 3466.6259765625 [#######---] 66.09% train_model result: {'val_loss': 6262.00634765625, 'hp_metric': 6262.00634765625}
spotpython tuning: 3466.6259765625 [#######---] 69.31% train_model result: {'val_loss': 21239.228515625, 'hp_metric': 21239.228515625}
spotpython tuning: 3466.6259765625 [#######---] 72.29% train_model result: {'val_loss': 16612.53515625, 'hp_metric': 16612.53515625}
spotpython tuning: 3466.6259765625 [########--] 75.84% train_model result: {'val_loss': 23238.45703125, 'hp_metric': 23238.45703125}
spotpython tuning: 3466.6259765625 [########--] 78.98% train_model result: {'val_loss': 23977.640625, 'hp_metric': 23977.640625}
spotpython tuning: 3466.6259765625 [########--] 81.49% train_model result: {'val_loss': 11396.7705078125, 'hp_metric': 11396.7705078125}
spotpython tuning: 3466.6259765625 [########--] 82.95% train_model result: {'val_loss': 5497.52001953125, 'hp_metric': 5497.52001953125}
spotpython tuning: 3466.6259765625 [#########-] 86.56% train_model result: {'val_loss': 24249.099609375, 'hp_metric': 24249.099609375}
spotpython tuning: 3466.6259765625 [#########-] 90.47% train_model result: {'val_loss': 20551.02734375, 'hp_metric': 20551.02734375}
spotpython tuning: 3466.6259765625 [#########-] 94.41% train_model result: {'val_loss': 23480.009765625, 'hp_metric': 23480.009765625}
spotpython tuning: 3466.6259765625 [##########] 100.00% Done...
Experiment saved to 601-early_stopping_res.pkl<spotpython.spot.spot.Spot at 0x154ba51c0>43.2 Looking at the Results
43.2.1 Tuning Progress
After the hyperparameter tuning run is finished, the progress of the hyperparameter tuning can be visualized with spotpython’s method plot_progress. The black points represent the performace values (score or metric) of hyperparameter configurations from the initial design, whereas the red points represents the hyperparameter configurations found by the surrogate model based optimization.
S.plot_progress(log_y=True)
43.2.2 Tuned Hyperparameters and Their Importance
Results can be printed in tabular form.
print_res_table(S)| name | type | default | lower | upper | tuned | transform | importance | stars |
|----------------|--------|-----------|---------|---------|--------------------|-----------------------|--------------|---------|
| l1 | int | 3 | 3.0 | 4.0 | 3.0 | transform_power_2_int | 0.00 | |
| epochs | int | 4 | 3.0 | 7.0 | 5.0 | transform_power_2_int | 0.00 | |
| batch_size | int | 4 | 4.0 | 11.0 | 6.0 | transform_power_2_int | 0.00 | |
| act_fn | factor | ReLU | 0.0 | 5.0 | ReLU | None | 0.00 | |
| optimizer | factor | SGD | 0.0 | 2.0 | Adadelta | None | 0.00 | |
| dropout_prob | float | 0.01 | 0.0 | 0.025 | 0.0184251494885258 | None | 0.01 | |
| lr_mult | float | 1.0 | 0.1 | 10.0 | 3.1418668140600845 | None | 0.43 | . |
| patience | int | 2 | 2.0 | 3.0 | 3.0 | transform_power_2_int | 0.00 | |
| batch_norm | factor | 0 | 0.0 | 1.0 | 0 | None | 0.00 | |
| initialization | factor | Default | 0.0 | 4.0 | kaiming_normal | None | 100.00 | *** |A histogram can be used to visualize the most important hyperparameters.
S.plot_importance(threshold=1.0)
S.plot_important_hyperparameter_contour(max_imp=3)l1: 0.0019663296830085253
epochs: 0.0019663296830085253
batch_size: 0.0019663296830085253
act_fn: 0.0019663296830085253
optimizer: 0.0019663296830085253
dropout_prob: 0.00725586899953753
lr_mult: 0.43276843571002355
patience: 0.0019663296830085253
batch_norm: 0.0019663296830085253
initialization: 100.0
43.2.3 Get the Tuned Architecture
import pprint
from spotpython.hyperparameters.values import get_tuned_architecture
config = get_tuned_architecture(S)
pprint.pprint(config){'act_fn': ReLU(),
'batch_norm': False,
'batch_size': 64,
'dropout_prob': 0.0184251494885258,
'epochs': 32,
'initialization': 'kaiming_normal',
'l1': 8,
'lr_mult': 3.1418668140600845,
'optimizer': 'Adadelta',
'patience': 8}43.2.4 Test on the full data set
fun_control.update({"TENSORBOARD_CLEAN": True})
fun_control.update({"tensorboard_log": True})from spotpython.light.testmodel import test_model
from spotpython.utils.init import get_feature_names
test_model(config, fun_control)
get_feature_names(fun_control)┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Test metric ┃ DataLoader 0 ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ hp_metric │ 3514.14794921875 │ │ val_loss │ 3514.14794921875 │ └───────────────────────────┴───────────────────────────┘
test_model result: {'val_loss': 3514.14794921875, 'hp_metric': 3514.14794921875}['age',
'sex',
'bmi',
'bp',
's1_tc',
's2_ldl',
's3_hdl',
's4_tch',
's5_ltg',
's6_glu']43.3 Cross Validation With Lightning
- The
KFoldclass fromsklearn.model_selectionis used to generate the folds for cross-validation. - These mechanism is used to generate the folds for the final evaluation of the model.
- The
CrossValidationDataModuleclass [SOURCE] is used to generate the folds for the hyperparameter tuning process. - It is called from the
cv_modelfunction [SOURCE].
config{'l1': 8,
'epochs': 32,
'batch_size': 64,
'act_fn': ReLU(),
'optimizer': 'Adadelta',
'dropout_prob': 0.0184251494885258,
'lr_mult': 3.1418668140600845,
'patience': 8,
'batch_norm': False,
'initialization': 'kaiming_normal'}from spotpython.light.cvmodel import cv_model
fun_control.update({"k_folds": 2})
fun_control.update({"test_size": 0.6})
cv_model(config, fun_control)k: 0train_model result: {'val_loss': 3196.562255859375, 'hp_metric': 3196.562255859375}
k: 1
train_model result: {'val_loss': 3129.9423828125, 'hp_metric': 3129.9423828125}3163.252319335937543.4 Extending the Basic Setup
This basic setup can be adapted to user-specific needs in many ways. For example, the user can specify a custom data set, a custom model, or a custom loss function. The following sections provide more details on how to customize the hyperparameter tuning process. Before we proceed, we will provide an overview of the basic settings of the hyperparameter tuning process and explain the parameters used so far.
43.4.1 General Experiment Setup
To keep track of the different experiments, we use a PREFIX for the experiment name. The PREFIX is used to create a unique experiment name. The PREFIX is also used to create a unique TensorBoard folder, which is used to store the TensorBoard log files.
spotpython allows the specification of two different types of stopping criteria: first, the number of function evaluations (fun_evals), and second, the maximum run time in seconds (max_time). Here, we will set the number of function evaluations to infinity and the maximum run time to one minute.
max_time is set to one minute for demonstration purposes. For real experiments, this value should be increased. Note, the total run time may exceed the specified max_time, because the initial design is always evaluated, even if this takes longer than max_time.
43.4.2 Data Setup
Here, we have provided the Diabetes data set class, which is a subclass of torch.utils.data.Dataset. Data preprocessing is handled by Lightning and PyTorch. It is described in the LIGHTNINGDATAMODULE documentation.
The data splitting, i.e., the generation of training, validation, and testing data, is handled by Lightning.
43.4.3 Objective Function fun
The objective function fun from the class HyperLight [SOURCE] is selected next. It implements an interface from PyTorch’s training, validation, and testing methods to spotpython.
43.4.4 Core-Model Setup
By using core_model_name = "light.regression.NNLinearRegressor", the spotpython model class NetLightRegression [SOURCE] from the light.regression module is selected.
43.4.5 Hyperdict Setup
For a given core_model_name, the corresponding hyperparameters are automatically loaded from the associated dictionary, which is stored as a JSON file. The JSON file contains hyperparameter type information, names, and bounds. For spotpython models, the hyperparameters are stored in the LightHyperDict, see [SOURCE] Alternatively, you can load a local hyper_dict. The hyperdict uses the default hyperparameter settings. These can be modified as described in Section 12.15.1.
43.4.6 Other Settings
There are several additional parameters that can be specified, e.g., since we did not specify a loss function, mean_squared_error is used, which is the default loss function. These will be explained in more detail in the following sections.
43.5 Tensorboard
The textual output shown in the console (or code cell) can be visualized with Tensorboard, if the argument tensorboard_log to fun_control_init() is set to True. The Tensorboard log files are stored in the runs folder. To start Tensorboard, run the following command in the terminal:
tensorboard --logdir="runs/"Further information can be found in the PyTorch Lightning documentation for Tensorboard.
43.6 Loading the Saved Experiment and Getting the Hyperparameters of the Tuned Model
To get the tuned hyperparameters as a dictionary, the get_tuned_architecture function can be used.
from spotpython.utils.file import load_result
spot_tuner = load_result(PREFIX=PREFIX)
config = get_tuned_architecture(spot_tuner)
configLoaded experiment from 601-early_stopping_res.pkl{'l1': 8,
'epochs': 32,
'batch_size': 64,
'act_fn': ReLU(),
'optimizer': 'Adadelta',
'dropout_prob': 0.0184251494885258,
'lr_mult': 3.1418668140600845,
'patience': 8,
'batch_norm': False,
'initialization': 'kaiming_normal'}43.7 Using the spotgui
The spotgui [github] provides a convenient way to interact with the hyperparameter tuning process. To obtain the settings from Section 42.1, the spotgui can be started as shown in Figure 43.1.
43.8 Summary
This section presented an introduction to the basic setup of hyperparameter tuning with spotpython and PyTorch Lightning.