39  Hyperparameter Tuning with spotpython and PyTorch Lightning for the Diabetes Data Set Using a ResNet Model

In this section, we will show how spotpython can be integrated into the PyTorch Lightning training workflow for a regression task. It demonstrates how easy it is to use spotpython to tune hyperparameters for a PyTorch Lightning model.

After importing the necessary libraries, the fun_control dictionary is set up via the fun_control_init function. The fun_control dictionary contains

The HyperLight class is used to define the objective function fun. It connects the PyTorch and the spotpython methods and is provided by spotpython.

from spotpython.data.diabetes import Diabetes
from spotpython.hyperdict.light_hyper_dict import LightHyperDict
from spotpython.fun.hyperlight import HyperLight
from spotpython.utils.init import (fun_control_init, surrogate_control_init, design_control_init)
from spotpython.utils.eda import gen_design_table
from spotpython.spot import spot
from spotpython.utils.file import get_experiment_filename

PREFIX="605"

data_set = Diabetes()

fun_control = fun_control_init(
    PREFIX=PREFIX,
    fun_evals=inf,
    max_time=1,
    data_set = data_set,
    core_model_name="light.regression.NNResNetRegressor",
    hyperdict=LightHyperDict,
    _L_in=10,
    _L_out=1)

fun = HyperLight().fun
module_name: light
submodule_name: regression
model_name: NNResNetRegressor

The method set_hyperparameter allows the user to modify default hyperparameter settings. Here we modify some hyperparameters to keep the model small and to decrease the tuning time.

from spotpython.hyperparameters.values import set_hyperparameter
set_hyperparameter(fun_control, "optimizer", [ "Adadelta", "Adam", "Adamax"])
set_hyperparameter(fun_control, "l1", [3,4])
set_hyperparameter(fun_control, "epochs", [3,7])
set_hyperparameter(fun_control, "batch_size", [4,11])
set_hyperparameter(fun_control, "dropout_prob", [0.0, 0.025])
set_hyperparameter(fun_control, "patience", [2,3])
set_hyperparameter(fun_control, "lr_mult", [0.1, 20.0])

design_control = design_control_init(init_size=10)

print(gen_design_table(fun_control))
| name           | type   | default   |   lower |   upper | transform             |
|----------------|--------|-----------|---------|---------|-----------------------|
| l1             | int    | 3         |     3   |   4     | transform_power_2_int |
| epochs         | int    | 4         |     3   |   7     | transform_power_2_int |
| batch_size     | int    | 4         |     4   |  11     | transform_power_2_int |
| act_fn         | factor | ReLU      |     0   |   5     | None                  |
| optimizer      | factor | SGD       |     0   |   2     | None                  |
| dropout_prob   | float  | 0.01      |     0   |   0.025 | None                  |
| lr_mult        | float  | 1.0       |     0.1 |  20     | None                  |
| patience       | int    | 2         |     2   |   3     | transform_power_2_int |
| initialization | factor | Default   |     0   |   4     | None                  |

Finally, a Spot object is created. Calling the method run() starts the hyperparameter tuning process.

spot_tuner = spot.Spot(fun=fun,fun_control=fun_control, design_control=design_control)
res = spot_tuner.run()

In fun(): config:
{'act_fn': Tanh(),
 'batch_size': 128,
 'dropout_prob': np.float64(0.0004697637333605659),
 'epochs': 64,
 'initialization': 'kaiming_normal',
 'l1': 8,
 'lr_mult': np.float64(12.192543453926158),
 'optimizer': 'Adamax',
 'patience': 4}
train_model result: {'val_loss': 23404.1953125, 'hp_metric': 23404.1953125}

In fun(): config:
{'act_fn': LeakyReLU(),
 'batch_size': 128,
 'dropout_prob': np.float64(0.010646332369413225),
 'epochs': 8,
 'initialization': 'kaiming_normal',
 'l1': 16,
 'lr_mult': np.float64(4.816418992866534),
 'optimizer': 'Adam',
 'patience': 8}
train_model result: {'val_loss': 22284.63671875, 'hp_metric': 22284.63671875}

In fun(): config:
{'act_fn': ELU(),
 'batch_size': 512,
 'dropout_prob': np.float64(0.018543188398461703),
 'epochs': 64,
 'initialization': 'xavier_uniform',
 'l1': 16,
 'lr_mult': np.float64(1.7231674050173253),
 'optimizer': 'Adam',
 'patience': 4}
train_model result: {'val_loss': 23765.34375, 'hp_metric': 23765.34375}

In fun(): config:
{'act_fn': Sigmoid(),
 'batch_size': 16,
 'dropout_prob': np.float64(0.01445464281687503),
 'epochs': 8,
 'initialization': 'Default',
 'l1': 16,
 'lr_mult': np.float64(18.276406412830898),
 'optimizer': 'Adam',
 'patience': 4}
train_model result: {'val_loss': 23489.228515625, 'hp_metric': 23489.228515625}

In fun(): config:
{'act_fn': ReLU(),
 'batch_size': 1024,
 'dropout_prob': np.float64(0.006041903974622355),
 'epochs': 128,
 'initialization': 'xavier_uniform',
 'l1': 16,
 'lr_mult': np.float64(11.56656626473706),
 'optimizer': 'Adam',
 'patience': 8}
train_model result: {'val_loss': 22797.18359375, 'hp_metric': 22797.18359375}

In fun(): config:
{'act_fn': ReLU(),
 'batch_size': 2048,
 'dropout_prob': np.float64(0.017180774192768807),
 'epochs': 128,
 'initialization': 'kaiming_uniform',
 'l1': 8,
 'lr_mult': np.float64(17.497450615230047),
 'optimizer': 'Adadelta',
 'patience': 8}
train_model result: {'val_loss': 6766.41552734375, 'hp_metric': 6766.41552734375}

In fun(): config:
{'act_fn': Tanh(),
 'batch_size': 256,
 'dropout_prob': np.float64(0.020472444721704793),
 'epochs': 16,
 'initialization': 'xavier_normal',
 'l1': 8,
 'lr_mult': np.float64(9.056021366896996),
 'optimizer': 'Adam',
 'patience': 4}
train_model result: {'val_loss': 24115.458984375, 'hp_metric': 24115.458984375}

In fun(): config:
{'act_fn': Swish(),
 'batch_size': 512,
 'dropout_prob': np.float64(0.022634716447473148),
 'epochs': 16,
 'initialization': 'kaiming_uniform',
 'l1': 8,
 'lr_mult': np.float64(3.2137011004706815),
 'optimizer': 'Adadelta',
 'patience': 4}
train_model result: {'val_loss': 23931.376953125, 'hp_metric': 23931.376953125}

In fun(): config:
{'act_fn': LeakyReLU(),
 'batch_size': 32,
 'dropout_prob': np.float64(0.0031403022317008592),
 'epochs': 32,
 'initialization': 'xavier_uniform',
 'l1': 8,
 'lr_mult': np.float64(15.164637662335846),
 'optimizer': 'Adam',
 'patience': 8}
train_model result: {'val_loss': 21796.736328125, 'hp_metric': 21796.736328125}

In fun(): config:
{'act_fn': ELU(),
 'batch_size': 64,
 'dropout_prob': np.float64(0.007623810344374791),
 'epochs': 32,
 'initialization': 'kaiming_normal',
 'l1': 16,
 'lr_mult': np.float64(7.425439588318577),
 'optimizer': 'Adamax',
 'patience': 8}
train_model result: {'val_loss': 23608.984375, 'hp_metric': 23608.984375}

In fun(): config:
{'act_fn': ReLU(),
 'batch_size': 2048,
 'dropout_prob': np.float64(0.01552995414446641),
 'epochs': 128,
 'initialization': 'kaiming_uniform',
 'l1': 8,
 'lr_mult': np.float64(20.0),
 'optimizer': 'Adadelta',
 'patience': 8}
train_model result: {'val_loss': 4165.15234375, 'hp_metric': 4165.15234375}
spotpython tuning: 4165.15234375 [#---------] 5.62% 

In fun(): config:
{'act_fn': ELU(),
 'batch_size': 2048,
 'dropout_prob': np.float64(0.025),
 'epochs': 128,
 'initialization': 'kaiming_uniform',
 'l1': 8,
 'lr_mult': np.float64(20.0),
 'optimizer': 'Adadelta',
 'patience': 8}
train_model result: {'val_loss': 21898.88671875, 'hp_metric': 21898.88671875}
spotpython tuning: 4165.15234375 [##--------] 24.87% 

In fun(): config:
{'act_fn': ReLU(),
 'batch_size': 2048,
 'dropout_prob': np.float64(0.013321149691097397),
 'epochs': 128,
 'initialization': 'kaiming_uniform',
 'l1': 8,
 'lr_mult': np.float64(20.0),
 'optimizer': 'Adadelta',
 'patience': 8}
train_model result: {'val_loss': 23360.689453125, 'hp_metric': 23360.689453125}
spotpython tuning: 4165.15234375 [####------] 44.76% 

In fun(): config:
{'act_fn': ReLU(),
 'batch_size': 16,
 'dropout_prob': np.float64(0.025),
 'epochs': 128,
 'initialization': 'kaiming_normal',
 'l1': 8,
 'lr_mult': np.float64(20.0),
 'optimizer': 'Adamax',
 'patience': 8}
train_model result: {'val_loss': 20871.68359375, 'hp_metric': 20871.68359375}
spotpython tuning: 4165.15234375 [##########] 100.00% Done...

39.1 Looking at the Results

39.1.1 Tuning Progress

After the hyperparameter tuning run is finished, the progress of the hyperparameter tuning can be visualized with spotpython’s method plot_progress. The black points represent the performace values (score or metric) of hyperparameter configurations from the initial design, whereas the red points represents the hyperparameter configurations found by the surrogate model based optimization.

spot_tuner.plot_progress()

39.1.2 Tuned Hyperparameters and Their Importance

Results can be printed in tabular form.

from spotpython.utils.eda import gen_design_table
print(gen_design_table(fun_control=fun_control, spot=spot_tuner))
| name           | type   | default   |   lower |   upper | tuned               | transform             |   importance | stars   |
|----------------|--------|-----------|---------|---------|---------------------|-----------------------|--------------|---------|
| l1             | int    | 3         |     3.0 |     4.0 | 3.0                 | transform_power_2_int |         0.01 |         |
| epochs         | int    | 4         |     3.0 |     7.0 | 7.0                 | transform_power_2_int |         0.00 |         |
| batch_size     | int    | 4         |     4.0 |    11.0 | 11.0                | transform_power_2_int |         0.03 |         |
| act_fn         | factor | ReLU      |     0.0 |     5.0 | ReLU                | None                  |         0.08 |         |
| optimizer      | factor | SGD       |     0.0 |     2.0 | Adadelta            | None                  |         0.00 |         |
| dropout_prob   | float  | 0.01      |     0.0 |   0.025 | 0.01552995414446641 | None                  |       100.00 | ***     |
| lr_mult        | float  | 1.0       |     0.1 |    20.0 | 20.0                | None                  |         0.00 |         |
| patience       | int    | 2         |     2.0 |     3.0 | 3.0                 | transform_power_2_int |         0.00 |         |
| initialization | factor | Default   |     0.0 |     4.0 | kaiming_uniform     | None                  |         0.00 |         |

A histogram can be used to visualize the most important hyperparameters.

spot_tuner.plot_importance(threshold=1.0)

spot_tuner.plot_important_hyperparameter_contour(max_imp=3)
l1:  0.01062745364702381
epochs:  0.001
batch_size:  0.02978691665200304
act_fn:  0.07734913702242881
optimizer:  0.001
dropout_prob:  100.0
lr_mult:  0.001
patience:  0.001
initialization:  0.001

39.1.3 Get the Tuned Architecture

import pprint
from spotpython.hyperparameters.values import get_tuned_architecture
config = get_tuned_architecture(spot_tuner, fun_control)
pprint.pprint(config)
{'act_fn': ReLU(),
 'batch_size': 2048,
 'dropout_prob': np.float64(0.01552995414446641),
 'epochs': 128,
 'initialization': 'kaiming_uniform',
 'l1': 8,
 'lr_mult': np.float64(20.0),
 'optimizer': 'Adadelta',
 'patience': 8}

39.1.4 Test on the full data set

# set the value of the key "TENSORBOARD_CLEAN" to True in the fun_control dictionary and use the update() method to update the fun_control dictionary
import os
# if the directory "./runs" exists, delete it
if os.path.exists("./runs"):
    os.system("rm -r ./runs")
fun_control.update({"tensorboard_log": True})
from spotpython.light.testmodel import test_model
from spotpython.utils.init import get_feature_names

test_model(config, fun_control)
get_feature_names(fun_control)
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃        Test metric               DataLoader 0        ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│         hp_metric               31501.638671875      │
│         val_loss                31501.638671875      │
└───────────────────────────┴───────────────────────────┘
test_model result: {'val_loss': 31501.638671875, 'hp_metric': 31501.638671875}
['age',
 'sex',
 'bmi',
 'bp',
 's1_tc',
 's2_ldl',
 's3_hdl',
 's4_tch',
 's5_ltg',
 's6_glu']

39.1.5 Cross Validation With Lightning

  • The KFold class from sklearn.model_selection is used to generate the folds for cross-validation.
  • These mechanism is used to generate the folds for the final evaluation of the model.
  • The CrossValidationDataModule class [SOURCE] is used to generate the folds for the hyperparameter tuning process.
  • It is called from the cv_model function [SOURCE].
config
{'l1': 8,
 'epochs': 128,
 'batch_size': 2048,
 'act_fn': ReLU(),
 'optimizer': 'Adadelta',
 'dropout_prob': np.float64(0.01552995414446641),
 'lr_mult': np.float64(20.0),
 'patience': 8,
 'initialization': 'kaiming_uniform'}
from spotpython.light.cvmodel import cv_model
fun_control.update({"k_folds": 2})
fun_control.update({"test_size": 0.6})
cv_model(config, fun_control)
k: 0
train_model result: {'val_loss': 30915.755859375, 'hp_metric': 30915.755859375}
k: 1
train_model result: {'val_loss': 5199.56640625, 'hp_metric': 5199.56640625}
18057.6611328125

39.2 Summary

This section presented an introduction to the basic setup of hyperparameter tuning with spotpython and PyTorch Lightning using a ResNet model for the Diabetes data set.