53  Hyperparameter Tuning of a Transformer Network with PyTorch Lightning

53.1 Basic Setup

This section provides an overview of the hyperparameter tuning process using spotpython and PyTorch Lightning. It uses the Diabetes data set (see Section E.1) for a regression task.

In this section, we will show how spotpython can be integrated into the PyTorch Lightning training workflow for a regression task. It demonstrates how easy it is to use spotpython to tune hyperparameters for a PyTorch Lightning model.

After importing the necessary libraries, the fun_control dictionary is set up via the fun_control_init function. The fun_control dictionary contains

  • PREFIX: a unique identifier for the experiment
  • fun_evals: the number of function evaluations
  • max_time: the maximum run time in minutes
  • data_set: the data set. Here we use the Diabetes data set that is provided by spotpython.
  • core_model_name: the class name of the neural network model. This neural network model is provided by spotpython.
  • hyperdict: the hyperparameter dictionary. This dictionary is used to define the hyperparameters of the neural network model. It is also provided by spotpython.
  • _L_in: the number of input features. Since the Diabetes data set has 10 features, _L_in is set to 10.
  • _L_out: the number of output features. Since we want to predict a single value, _L_out is set to 1.

The method set_hyperparameter allows the user to modify default hyperparameter settings. Here we set the initialization method to ["Default"]. No other initializations are used in this experiment. The HyperLight class is used to define the objective function fun. It connects the PyTorch and the spotpython methods and is provided by spotpython. Finally, a Spot object is created.

Note, the divergence_threshold is set to 25,000, which is based on some pre-experiments with the Diabetes data set.

from spotpython.data.diabetes import Diabetes
from spotpython.hyperdict.light_hyper_dict import LightHyperDict
from spotpython.fun.hyperlight import HyperLight
from spotpython.utils.init import (fun_control_init, surrogate_control_init, design_control_init)
from spotpython.utils.eda import print_exp_table
from spotpython.hyperparameters.values import set_hyperparameter
from spotpython.spot import Spot
from spotpython.utils.file import get_experiment_filename
from spotpython.utils.scaler import TorchStandardScaler

fun_control = fun_control_init(
    PREFIX="603",
    TENSORBOARD_CLEAN=True,
    tensorboard_log=True,
    fun_evals=inf,
    max_time=1,
    data_set = Diabetes(),
    scaler=TorchStandardScaler(),
    core_model_name="light.regression.NNTransformerRegressor",
    hyperdict=LightHyperDict,
    divergence_threshold=25_000,
    _L_in=10,
    _L_out=1)

set_hyperparameter(fun_control, "optimizer", [
                "Adadelta",
                "Adagrad",
                "Adam",
                "AdamW",
                "Adamax",
            ])
set_hyperparameter(fun_control, "epochs", [5, 7])
set_hyperparameter(fun_control, "nhead", [1, 2])
set_hyperparameter(fun_control, "dim_feedforward_mult", [1, 1])

design_control = design_control_init(init_size=5)
surrogate_control = surrogate_control_init(
    method="regression",
    min_Lambda=1e-3,
    max_Lambda=10,
)

fun = HyperLight().fun

spot_tuner = Spot(fun=fun,fun_control=fun_control, design_control=design_control, surrogate_control=surrogate_control)
Moving TENSORBOARD_PATH: runs/ to TENSORBOARD_PATH_OLD: runs_OLD/runs_2025_10_19_21_20_27_0
Created spot_tensorboard_path: runs/spot_logs/603_maans08_2025-10-19_21-20-27 for SummaryWriter()
module_name: light
submodule_name: regression
model_name: NNTransformerRegressor

We can take a look at the design table to see the initial design.

print_exp_table(fun_control)
| name                 | type   | default        |   lower |   upper | transform             |
|----------------------|--------|----------------|---------|---------|-----------------------|
| d_model_mult         | int    | 4              |    1    |     5   | transform_power_2_int |
| nhead                | int    | 3              |    1    |     2   | transform_power_2_int |
| num_encoder_layers   | int    | 1              |    1    |     4   | transform_power_2_int |
| dim_feedforward_mult | int    | 1              |    1    |     1   | transform_power_2_int |
| epochs               | int    | 7              |    5    |     7   | transform_power_2_int |
| batch_size           | int    | 5              |    5    |     8   | transform_power_2_int |
| optimizer            | factor | Adam           |    0    |     4   | None                  |
| dropout              | float  | 0.1            |    0.01 |     0.1 | None                  |
| lr_mult              | float  | 0.1            |    0.01 |     0.3 | None                  |
| patience             | int    | 5              |    4    |     7   | transform_power_2_int |
| initialization       | factor | xavier_uniform |    0    |     3   | None                  |

Calling the method run() starts the hyperparameter tuning process on the local machine.

res = spot_tuner.run()
d_model: 8, dim_feedforward: 16
┏━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃   ┃ Name                ┃ Type               ┃ Params ┃ Mode  ┃  FLOPs ┃      In sizes ┃     Out sizes ┃
┡━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ 0 │ input_proj          │ Linear             │     88 │ train │ 20.5 K │     [128, 10] │      [128, 8] │
│ 1 │ positional_encoding │ PositionalEncoding │      0 │ train │      0 │      [128, 8] │ [128, 128, 8] │
│ 2 │ transformer_encoder │ TransformerEncoder │  4.8 K │ train │  134 M │ [128, 128, 8] │ [128, 128, 8] │
│ 3 │ fc_out              │ Linear             │      9 │ train │  2.0 K │      [128, 8] │      [128, 1] │
└───┴─────────────────────┴────────────────────┴────────┴───────┴────────┴───────────────┴───────────────┘

Trainable params: 4.9 K                                                                                            
Non-trainable params: 0                                                                                            
Total params: 4.9 K                                                                                                
Total estimated model params size (MB): 0                                                                          
Modules in train mode: 85                                                                                          
Modules in eval mode: 0                                                                                            
Total FLOPs: 134 M                                                                                                 

train_model result: {'val_loss': 23955.66015625, 'hp_metric': 23955.66015625}
d_model: 128, dim_feedforward: 256
┏━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃   ┃ Name                ┃ Type               ┃ Params ┃ Mode  ┃  FLOPs ┃      In sizes ┃     Out sizes ┃
┡━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ 0 │ input_proj          │ Linear             │  1.4 K │ train │ 81.9 K │      [32, 10] │     [32, 128] │
│ 1 │ positional_encoding │ PositionalEncoding │      0 │ train │      0 │     [32, 128] │ [32, 32, 128] │
│ 2 │ transformer_encoder │ TransformerEncoder │  529 K │ train │  1.1 B │ [32, 32, 128] │ [32, 32, 128] │
│ 3 │ fc_out              │ Linear             │    129 │ train │  8.2 K │     [32, 128] │       [32, 1] │
└───┴─────────────────────┴────────────────────┴────────┴───────┴────────┴───────────────┴───────────────┘

Trainable params: 531 K                                                                                            
Non-trainable params: 0                                                                                            
Total params: 531 K                                                                                                
Total estimated model params size (MB): 2                                                                          
Modules in train mode: 45                                                                                          
Modules in eval mode: 0                                                                                            
Total FLOPs: 1.1 B                                                                                                 

train_model result: {'val_loss': 20858.498046875, 'hp_metric': 20858.498046875}
d_model: 32, dim_feedforward: 64
┏━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┓
┃   ┃ Name                ┃ Type               ┃ Params ┃ Mode  ┃  FLOPs ┃       In sizes ┃      Out sizes ┃
┡━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━┩
│ 0 │ input_proj          │ Linear             │    352 │ train │ 81.9 K │      [128, 10] │      [128, 32] │
│ 1 │ positional_encoding │ PositionalEncoding │      0 │ train │      0 │      [128, 32] │ [128, 128, 32] │
│ 2 │ transformer_encoder │ TransformerEncoder │ 68.4 K │ train │  2.1 B │ [128, 128, 32] │ [128, 128, 32] │
│ 3 │ fc_out              │ Linear             │     33 │ train │  8.2 K │      [128, 32] │       [128, 1] │
└───┴─────────────────────┴────────────────────┴────────┴───────┴────────┴────────────────┴────────────────┘

Trainable params: 68.7 K                                                                                           
Non-trainable params: 0                                                                                            
Total params: 68.7 K                                                                                               
Total estimated model params size (MB): 0                                                                          
Modules in train mode: 85                                                                                          
Modules in eval mode: 0                                                                                            
Total FLOPs: 2.1 B                                                                                                 

train_model result: {'val_loss': 23231.283203125, 'hp_metric': 23231.283203125}
d_model: 16, dim_feedforward: 32
┏━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
┃   ┃ Name                ┃ Type               ┃ Params ┃ Mode  ┃  FLOPs ┃     In sizes ┃    Out sizes ┃
┡━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
│ 0 │ input_proj          │ Linear             │    176 │ train │ 20.5 K │     [64, 10] │     [64, 16] │
│ 1 │ positional_encoding │ PositionalEncoding │      0 │ train │      0 │     [64, 16] │ [64, 64, 16] │
│ 2 │ transformer_encoder │ TransformerEncoder │ 35.6 K │ train │  268 M │ [64, 64, 16] │ [64, 64, 16] │
│ 3 │ fc_out              │ Linear             │     17 │ train │  2.0 K │     [64, 16] │      [64, 1] │
└───┴─────────────────────┴────────────────────┴────────┴───────┴────────┴──────────────┴──────────────┘

Trainable params: 35.8 K                                                                                           
Non-trainable params: 0                                                                                            
Total params: 35.8 K                                                                                               
Total estimated model params size (MB): 0                                                                          
Modules in train mode: 165                                                                                         
Modules in eval mode: 0                                                                                            
Total FLOPs: 268 M                                                                                                 

train_model result: {'val_loss': 23903.546875, 'hp_metric': 23903.546875}
d_model: 8, dim_feedforward: 16
┏━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃   ┃ Name                ┃ Type               ┃ Params ┃ Mode  ┃  FLOPs ┃      In sizes ┃     Out sizes ┃
┡━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ 0 │ input_proj          │ Linear             │     88 │ train │ 41.0 K │     [256, 10] │      [256, 8] │
│ 1 │ positional_encoding │ PositionalEncoding │      0 │ train │      0 │      [256, 8] │ [256, 256, 8] │
│ 2 │ transformer_encoder │ TransformerEncoder │  1.2 K │ train │  134 M │ [256, 256, 8] │ [256, 256, 8] │
│ 3 │ fc_out              │ Linear             │      9 │ train │  4.1 K │      [256, 8] │      [256, 1] │
└───┴─────────────────────┴────────────────────┴────────┴───────┴────────┴───────────────┴───────────────┘

Trainable params: 1.3 K                                                                                            
Non-trainable params: 0                                                                                            
Total params: 1.3 K                                                                                                
Total estimated model params size (MB): 0                                                                          
Modules in train mode: 25                                                                                          
Modules in eval mode: 0                                                                                            
Total FLOPs: 134 M                                                                                                 

train_model result: {'val_loss': 23954.796875, 'hp_metric': 23954.796875}
Anisotropic model: n_theta set to 10
d_model: 128, dim_feedforward: 256
┏━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃   ┃ Name                ┃ Type               ┃ Params ┃ Mode  ┃  FLOPs ┃      In sizes ┃     Out sizes ┃
┡━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ 0 │ input_proj          │ Linear             │  1.4 K │ train │ 81.9 K │      [32, 10] │     [32, 128] │
│ 1 │ positional_encoding │ PositionalEncoding │      0 │ train │      0 │     [32, 128] │ [32, 32, 128] │
│ 2 │ transformer_encoder │ TransformerEncoder │  529 K │ train │  1.1 B │ [32, 32, 128] │ [32, 32, 128] │
│ 3 │ fc_out              │ Linear             │    129 │ train │  8.2 K │     [32, 128] │       [32, 1] │
└───┴─────────────────────┴────────────────────┴────────┴───────┴────────┴───────────────┴───────────────┘

Trainable params: 531 K                                                                                            
Non-trainable params: 0                                                                                            
Total params: 531 K                                                                                                
Total estimated model params size (MB): 2                                                                          
Modules in train mode: 45                                                                                          
Modules in eval mode: 0                                                                                            
Total FLOPs: 1.1 B                                                                                                 

train_model result: {'val_loss': 20565.634765625, 'hp_metric': 20565.634765625}
Anisotropic model: n_theta set to 10
spotpython tuning: 20565.634765625 [##--------] 18.98% 
d_model: 128, dim_feedforward: 256
┏━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃   ┃ Name                ┃ Type               ┃ Params ┃ Mode  ┃  FLOPs ┃      In sizes ┃     Out sizes ┃
┡━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ 0 │ input_proj          │ Linear             │  1.4 K │ train │ 81.9 K │      [32, 10] │     [32, 128] │
│ 1 │ positional_encoding │ PositionalEncoding │      0 │ train │      0 │     [32, 128] │ [32, 32, 128] │
│ 2 │ transformer_encoder │ TransformerEncoder │  529 K │ train │  1.1 B │ [32, 32, 128] │ [32, 32, 128] │
│ 3 │ fc_out              │ Linear             │    129 │ train │  8.2 K │     [32, 128] │       [32, 1] │
└───┴─────────────────────┴────────────────────┴────────┴───────┴────────┴───────────────┴───────────────┘

Trainable params: 531 K                                                                                            
Non-trainable params: 0                                                                                            
Total params: 531 K                                                                                                
Total estimated model params size (MB): 2                                                                          
Modules in train mode: 45                                                                                          
Modules in eval mode: 0                                                                                            
Total FLOPs: 1.1 B                                                                                                 

train_model result: {'val_loss': 20591.740234375, 'hp_metric': 20591.740234375}
Anisotropic model: n_theta set to 10
spotpython tuning: 20565.634765625 [####------] 37.89% 
d_model: 128, dim_feedforward: 256
┏━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃   ┃ Name                ┃ Type               ┃ Params ┃ Mode  ┃  FLOPs ┃      In sizes ┃     Out sizes ┃
┡━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ 0 │ input_proj          │ Linear             │  1.4 K │ train │ 81.9 K │      [32, 10] │     [32, 128] │
│ 1 │ positional_encoding │ PositionalEncoding │      0 │ train │      0 │     [32, 128] │ [32, 32, 128] │
│ 2 │ transformer_encoder │ TransformerEncoder │  529 K │ train │  1.1 B │ [32, 32, 128] │ [32, 32, 128] │
│ 3 │ fc_out              │ Linear             │    129 │ train │  8.2 K │     [32, 128] │       [32, 1] │
└───┴─────────────────────┴────────────────────┴────────┴───────┴────────┴───────────────┴───────────────┘

Trainable params: 531 K                                                                                            
Non-trainable params: 0                                                                                            
Total params: 531 K                                                                                                
Total estimated model params size (MB): 2                                                                          
Modules in train mode: 45                                                                                          
Modules in eval mode: 0                                                                                            
Total FLOPs: 1.1 B                                                                                                 

train_model result: {'val_loss': 21099.330078125, 'hp_metric': 21099.330078125}
Anisotropic model: n_theta set to 10
spotpython tuning: 20565.634765625 [######----] 57.06% 
d_model: 128, dim_feedforward: 256
┏━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃   ┃ Name                ┃ Type               ┃ Params ┃ Mode  ┃  FLOPs ┃      In sizes ┃     Out sizes ┃
┡━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ 0 │ input_proj          │ Linear             │  1.4 K │ train │ 81.9 K │      [32, 10] │     [32, 128] │
│ 1 │ positional_encoding │ PositionalEncoding │      0 │ train │      0 │     [32, 128] │ [32, 32, 128] │
│ 2 │ transformer_encoder │ TransformerEncoder │  529 K │ train │  1.1 B │ [32, 32, 128] │ [32, 32, 128] │
│ 3 │ fc_out              │ Linear             │    129 │ train │  8.2 K │     [32, 128] │       [32, 1] │
└───┴─────────────────────┴────────────────────┴────────┴───────┴────────┴───────────────┴───────────────┘

Trainable params: 531 K                                                                                            
Non-trainable params: 0                                                                                            
Total params: 531 K                                                                                                
Total estimated model params size (MB): 2                                                                          
Modules in train mode: 45                                                                                          
Modules in eval mode: 0                                                                                            
Total FLOPs: 1.1 B                                                                                                 

train_model result: {'val_loss': 20446.396484375, 'hp_metric': 20446.396484375}
Anisotropic model: n_theta set to 10
spotpython tuning: 20446.396484375 [########--] 76.40% 
d_model: 128, dim_feedforward: 256
┏━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃   ┃ Name                ┃ Type               ┃ Params ┃ Mode  ┃  FLOPs ┃      In sizes ┃     Out sizes ┃
┡━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ 0 │ input_proj          │ Linear             │  1.4 K │ train │ 81.9 K │      [32, 10] │     [32, 128] │
│ 1 │ positional_encoding │ PositionalEncoding │      0 │ train │      0 │     [32, 128] │ [32, 32, 128] │
│ 2 │ transformer_encoder │ TransformerEncoder │  529 K │ train │  1.1 B │ [32, 32, 128] │ [32, 32, 128] │
│ 3 │ fc_out              │ Linear             │    129 │ train │  8.2 K │     [32, 128] │       [32, 1] │
└───┴─────────────────────┴────────────────────┴────────┴───────┴────────┴───────────────┴───────────────┘

Trainable params: 531 K                                                                                            
Non-trainable params: 0                                                                                            
Total params: 531 K                                                                                                
Total estimated model params size (MB): 2                                                                          
Modules in train mode: 45                                                                                          
Modules in eval mode: 0                                                                                            
Total FLOPs: 1.1 B                                                                                                 

train_model result: {'val_loss': 20555.681640625, 'hp_metric': 20555.681640625}
Anisotropic model: n_theta set to 10
spotpython tuning: 20446.396484375 [##########] 95.85% 
d_model: 128, dim_feedforward: 256
┏━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃   ┃ Name                ┃ Type               ┃ Params ┃ Mode  ┃  FLOPs ┃      In sizes ┃     Out sizes ┃
┡━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ 0 │ input_proj          │ Linear             │  1.4 K │ train │ 81.9 K │      [32, 10] │     [32, 128] │
│ 1 │ positional_encoding │ PositionalEncoding │      0 │ train │      0 │     [32, 128] │ [32, 32, 128] │
│ 2 │ transformer_encoder │ TransformerEncoder │  529 K │ train │  1.1 B │ [32, 32, 128] │ [32, 32, 128] │
│ 3 │ fc_out              │ Linear             │    129 │ train │  8.2 K │     [32, 128] │       [32, 1] │
└───┴─────────────────────┴────────────────────┴────────┴───────┴────────┴───────────────┴───────────────┘

Trainable params: 531 K                                                                                            
Non-trainable params: 0                                                                                            
Total params: 531 K                                                                                                
Total estimated model params size (MB): 2                                                                          
Modules in train mode: 45                                                                                          
Modules in eval mode: 0                                                                                            
Total FLOPs: 1.1 B                                                                                                 

train_model result: {'val_loss': 20497.22265625, 'hp_metric': 20497.22265625}
Anisotropic model: n_theta set to 10
spotpython tuning: 20446.396484375 [##########] 100.00% Done...

Experiment saved to 603_res.pkl

Note that we have enabled Tensorboard-Logging, so we can visualize the results with Tensorboard. Execute the following command in the terminal to start Tensorboard.

tensorboard --logdir="runs/"

53.2 Looking at the Results

53.2.1 Tuning Progress

After the hyperparameter tuning run is finished, the progress of the hyperparameter tuning can be visualized with spotpython’s method plot_progress. The black points represent the performace values (score or metric) of hyperparameter configurations from the initial design, whereas the red points represents the hyperparameter configurations found by the surrogate model based optimization.

spot_tuner.plot_progress(log_y=True, filename=None)

53.2.2 Tuned Hyperparameters and Their Importance

Results can be printed in tabular form.

from spotpython.utils.eda import print_res_table
print_res_table(spot_tuner)
| name                 | type   | default        |   lower |   upper | tuned                | transform             |   importance | stars   |
|----------------------|--------|----------------|---------|---------|----------------------|-----------------------|--------------|---------|
| d_model_mult         | int    | 4              |     1.0 |     5.0 | 5.0                  | transform_power_2_int |       100.00 | ***     |
| nhead                | int    | 3              |     1.0 |     2.0 | 2.0                  | transform_power_2_int |         0.17 | .       |
| num_encoder_layers   | int    | 1              |     1.0 |     4.0 | 2.0                  | transform_power_2_int |         0.17 | .       |
| dim_feedforward_mult | int    | 1              |     1.0 |     1.0 | 1.0                  | transform_power_2_int |         0.00 |         |
| epochs               | int    | 7              |     5.0 |     7.0 | 7.0                  | transform_power_2_int |         0.17 | .       |
| batch_size           | int    | 5              |     5.0 |     8.0 | 5.0                  | transform_power_2_int |         0.17 | .       |
| optimizer            | factor | Adam           |     0.0 |     4.0 | Adagrad              | None                  |         0.17 | .       |
| dropout              | float  | 0.1            |    0.01 |     0.1 | 0.010000000000000004 | None                  |         0.17 | .       |
| lr_mult              | float  | 0.1            |    0.01 |     0.3 | 0.19522409527793771  | None                  |         0.17 | .       |
| patience             | int    | 5              |     4.0 |     7.0 | 4.0                  | transform_power_2_int |         0.17 | .       |
| initialization       | factor | xavier_uniform |     0.0 |     3.0 | xavier_normal        | None                  |         0.17 | .       |

53.3 Hyperparameter Considerations

  1. d_model (or d_embedding):

    • This is the dimension of the embedding space or the number of expected features in the input.
    • All input features are projected into this dimensional space before entering the transformer encoder.
    • This dimension must be divisible by nhead since each head in the multi-head attention mechanism will process a subset of d_model/nhead features.

  2. nhead:

    • This is the number of attention heads in the multi-head attention mechanism.
    • It allows the transformer to jointly attend to information from different representation subspaces.
    • It’s important that d_model % nhead == 0 to ensure the dimensions are evenly split among the heads.

  3. num_encoder_layers:

    • This specifies the number of transformer encoder layers stacked together.
    • Each layer contains a multi-head attention mechanism followed by position-wise feedforward layers.

  4. dim_feedforward:

    • This is the dimension of the feedforward network model within the transformer encoder layer.
    • Typically, this dimension is larger than d_model (e.g., 2048 for a Transformer model with d_model=512).

53.3.1 Important: Constraints and Interconnections:

  • d_model and nhead:
    • As mentioned, d_model must be divisible by nhead. This is critical because each attention head operates simultaneously on a part of the embedding, so d_model/nhead should be an integer.
  • num_encoder_layers and dim_feedforward**:
    • These parameters are more flexible and can be chosen independently of d_model and nhead.
    • However, the choice of dim_feedforward does influence the computational cost and model capacity, as larger dimensions allow learning more complex representations.
  • One hyperparameter does not strictly need to be a multiple of others except for ensuring d_model % nhead == 0.

53.3.2 Practical Considerations:

  1. Setting d_model:

    • Common choices for d_model are powers of 2 (e.g., 256, 512, 1024).
    • Ensure that it matches the size of the input data after the linear projection layer.

  2. Setting nhead:

    • Typically, values are 1, 2, 4, 8, etc., depending on the d_model value.
    • Each head works on a subset of features, so d_model / nhead should be large enough to be meaningful.

  3. Setting num_encoder_layers:

    • Practical values range from 1 to 12 or more depending on the depth desired.
    • Deeper models can capture more complex patterns but are also more computationally intensive.

  4. Setting dim_feedforward:

    • Often set to a multiple of d_model, such as 2048 when d_model is 512.
    • Ensures sufficient capacity in the intermediate layers for complex feature transformations.
NoteNote: d_model Calculation

Since d_model % nhead == 0 is a critical constraint to ensure that the multi-head attention mechanism can operate effectively, spotpython computes the value of d_model based on the nhead value provided by the user. This ensures that the hyperparameter configuration is valid. So, the final value of d_model is a multiple of nhead. spotpython uses the hyperparameter d_model_mult to determine the multiple of nhead to use for d_model, i.e., d_model = nhead * d_model_mult.

NoteNote: dim_feedforward Calculation

Since this dimension is typically larger than d_model (e.g., 2048 for a Transformer model with d_model=512), spotpython uses the hyperparameter dim_feedforward_mult to determine the multiple of d_model to use for dim_feedforward, i.e., dim_feedforward = d_model * dim_feedforward_mult.

53.4 Summary

This section presented an introduction to the basic setup of hyperparameter tuning of a transformer with spotpython and PyTorch Lightning.