32  Hyperparameter Tuning with PyTorch Lightning and User Data Sets

In this section, we will show how user specfied data can be used for the PyTorch Lightning hyperparameter tuning workflow with spotpython.

32.1 Loading a User Specified Data Set

Using a user-specified data set is straightforward.

The user simply needs to provide a data set and loads is as a spotpython CVSDataset() class by specifying the path, filename, and target column.

Consider the following example, where the user has a data set stored in the userData directory. The data set is stored in a file named data.csv. The target column is named target. To show the data, it is loaded as a pandas data frame and the first 5 rows are displayed. This step is not necessary for the hyperparameter tuning process, but it is useful for understanding the data.

# load the csv data set as a pandas dataframe and dislay the first 5 rows
import pandas as pd
data = pd.read_csv("./userData/data.csv")
print(data.head())

        age       sex       bmi        bp        s1        s2        s3  \
0  0.038076  0.050680  0.061696  0.021872 -0.044223 -0.034821 -0.043401   
1 -0.001882 -0.044642 -0.051474 -0.026328 -0.008449 -0.019163  0.074412   
2  0.085299  0.050680  0.044451 -0.005670 -0.045599 -0.034194 -0.032356   
3 -0.089063 -0.044642 -0.011595 -0.036656  0.012191  0.024991 -0.036038   
4  0.005383 -0.044642 -0.036385  0.021872  0.003935  0.015596  0.008142   

         s4        s5        s6  target  
0 -0.002592  0.019907 -0.017646   151.0  
1 -0.039493 -0.068332 -0.092204    75.0  
2 -0.002592  0.002861 -0.025930   141.0  
3  0.034309  0.022688 -0.009362   206.0  
4 -0.002592 -0.031988 -0.046641   135.0  

Next, the data set is loaded as a spotpython CSVDataset() class. This step is necessary for the hyperparameter tuning process.

from spotpython.data.csvdataset import CSVDataset
import torch
data_set = CSVDataset(directory="./userData/",
                     filename="data.csv",
                     target_column="target",
                     feature_type=torch.float32,
                     target_type=torch.float32,
                     rmNA=True)
print(len(data_set))

442

The following step is not necessary for the hyperparameter tuning process, but it is useful for understanding the data. The data set is loaded as a DataLoader from torch.utils.data to check the data.

# Set batch size for DataLoader
batch_size = 5
# Create DataLoader
from torch.utils.data import DataLoader
dataloader = DataLoader(data_set, batch_size=batch_size, shuffle=False)

# Iterate over the data in the DataLoader
for batch in dataloader:
    inputs, targets = batch
    print(f"Batch Size: {inputs.size(0)}")
    print(f"Inputs Shape: {inputs.shape}")
    print(f"Targets Shape: {targets.shape}")
    print("---------------")
    print(f"Inputs: {inputs}")
    print(f"Targets: {targets}")
    break

Batch Size: 5
Inputs Shape: torch.Size([5, 10])
Targets Shape: torch.Size([5])
---------------
Inputs: tensor([[ 0.0381,  0.0507,  0.0617,  0.0219, -0.0442, -0.0348, -0.0434, -0.0026,
          0.0199, -0.0176],
        [-0.0019, -0.0446, -0.0515, -0.0263, -0.0084, -0.0192,  0.0744, -0.0395,
         -0.0683, -0.0922],
        [ 0.0853,  0.0507,  0.0445, -0.0057, -0.0456, -0.0342, -0.0324, -0.0026,
          0.0029, -0.0259],
        [-0.0891, -0.0446, -0.0116, -0.0367,  0.0122,  0.0250, -0.0360,  0.0343,
          0.0227, -0.0094],
        [ 0.0054, -0.0446, -0.0364,  0.0219,  0.0039,  0.0156,  0.0081, -0.0026,
         -0.0320, -0.0466]])
Targets: tensor([151.,  75., 141., 206., 135.])

Similar to the setting from Section 31.1, the hyperparameter tuning setup is defined. Instead of using the Diabetes data set, the user data set is used. The data_set parameter is set to the user data set. The fun_control dictionary is set up via the fun_control_init function.

Note, that we have modified the fun_evals parameter to 12 and the init_size to 7 to reduce the computational time for this example.

from spotpython.hyperdict.light_hyper_dict import LightHyperDict
from spotpython.fun.hyperlight import HyperLight
from spotpython.utils.init import (fun_control_init, surrogate_control_init, design_control_init)
from spotpython.utils.eda import gen_design_table
from spotpython.hyperparameters.values import set_hyperparameter
from spotpython.spot import spot

fun_control = fun_control_init(
    PREFIX="601",
    fun_evals=12,
    max_time=1,
    data_set = data_set,
    core_model_name="light.regression.NNLinearRegressor",
    hyperdict=LightHyperDict,
    _L_in=10,
    _L_out=1)

design_control = design_control_init(init_size=7)

set_hyperparameter(fun_control, "initialization", ["Default"])

fun = HyperLight().fun

spot_tuner = spot.Spot(fun=fun,fun_control=fun_control, design_control=design_control)

module_name: light
submodule_name: regression
model_name: NNLinearRegressor

res = spot_tuner.run()
print(gen_design_table(fun_control=fun_control, spot=spot_tuner))
spot_tuner.plot_important_hyperparameter_contour(max_imp=3)

In fun(): config:
{'act_fn': Tanh(),
 'batch_norm': False,
 'batch_size': 4,
 'dropout_prob': np.float64(0.1852823390480081),
 'epochs': 512,
 'initialization': 'Default',
 'l1': 8,
 'lr_mult': np.float64(3.0369835027903065),
 'optimizer': 'SparseAdam',
 'patience': 32}

In fun(): config:
{'act_fn': LeakyReLU(),
 'batch_norm': True,
 'batch_size': 8,
 'dropout_prob': np.float64(0.009233319563046084),
 'epochs': 64,
 'initialization': 'Default',
 'l1': 128,
 'lr_mult': np.float64(2.037655996366023),
 'optimizer': 'ASGD',
 'patience': 32}

train_model result: {'val_loss': nan, 'hp_metric': nan}

In fun(): config:
{'act_fn': Swish(),
 'batch_norm': True,
 'batch_size': 4,
 'dropout_prob': np.float64(0.08633126283516715),
 'epochs': 128,
 'initialization': 'Default',
 'l1': 16,
 'lr_mult': np.float64(5.496436274922579),
 'optimizer': 'Adagrad',
 'patience': 8}

train_model result: {'val_loss': 11776.03515625, 'hp_metric': 11776.03515625}

In fun(): config:
{'act_fn': Sigmoid(),
 'batch_norm': True,
 'batch_size': 8,
 'dropout_prob': np.float64(0.06363775452678615),
 'epochs': 256,
 'initialization': 'Default',
 'l1': 64,
 'lr_mult': np.float64(7.360762633670198),
 'optimizer': 'RAdam',
 'patience': 8}

train_model result: {'val_loss': 21722.390625, 'hp_metric': 21722.390625}

In fun(): config:
{'act_fn': ELU(),
 'batch_norm': False,
 'batch_size': 8,
 'dropout_prob': np.float64(0.12202719963746221),
 'epochs': 32,
 'initialization': 'Default',
 'l1': 32,
 'lr_mult': np.float64(6.834960949095255),
 'optimizer': 'SGD',
 'patience': 32}

train_model result: {'val_loss': nan, 'hp_metric': nan}

In fun(): config:
{'act_fn': LeakyReLU(),
 'batch_norm': False,
 'batch_size': 2,
 'dropout_prob': np.float64(0.24543963132526866),
 'epochs': 128,
 'initialization': 'Default',
 'l1': 256,
 'lr_mult': np.float64(1.1500187430565307),
 'optimizer': 'NAdam',
 'patience': 16}

train_model result: {'val_loss': 8.58584665870927e+27, 'hp_metric': 8.58584665870927e+27}

In fun(): config:
{'act_fn': ReLU(),
 'batch_norm': True,
 'batch_size': 16,
 'dropout_prob': np.float64(0.14960635316721133),
 'epochs': 16,
 'initialization': 'Default',
 'l1': 32,
 'lr_mult': np.float64(9.293583024571447),
 'optimizer': 'AdamW',
 'patience': 8}

train_model result: {'val_loss': 23611.259765625, 'hp_metric': 23611.259765625}

In fun(): config:
{'act_fn': ReLU(),
 'batch_norm': True,
 'batch_size': 8,
 'dropout_prob': np.float64(0.11171640685799808),
 'epochs': 64,
 'initialization': 'Default',
 'l1': 32,
 'lr_mult': np.float64(7.505542772999748),
 'optimizer': 'AdamW',
 'patience': 8}
train_model result: {'val_loss': 10570.32421875, 'hp_metric': 10570.32421875}
spotpython tuning: 10570.32421875 [####------] 41.67% 

In fun(): config:
{'act_fn': Sigmoid(),
 'batch_norm': True,
 'batch_size': 16,
 'dropout_prob': np.float64(0.06779536482775718),
 'epochs': 32,
 'initialization': 'Default',
 'l1': 16,
 'lr_mult': np.float64(8.216392212995512),
 'optimizer': 'Adagrad',
 'patience': 8}

train_model result: {'val_loss': 22684.01953125, 'hp_metric': 22684.01953125}
spotpython tuning: 10570.32421875 [#####-----] 50.00% 

In fun(): config:
{'act_fn': Swish(),
 'batch_norm': True,
 'batch_size': 4,
 'dropout_prob': np.float64(0.25),
 'epochs': 16,
 'initialization': 'Default',
 'l1': 128,
 'lr_mult': np.float64(8.60815951788443),
 'optimizer': 'RAdam',
 'patience': 8}

train_model result: {'val_loss': 22166.294921875, 'hp_metric': 22166.294921875}
spotpython tuning: 10570.32421875 [######----] 58.33% 

In fun(): config:
{'act_fn': Swish(),
 'batch_norm': True,
 'batch_size': 16,
 'dropout_prob': np.float64(0.25),
 'epochs': 64,
 'initialization': 'Default',
 'l1': 32,
 'lr_mult': np.float64(9.341773134812628),
 'optimizer': 'RAdam',
 'patience': 8}
train_model result: {'val_loss': 23517.87109375, 'hp_metric': 23517.87109375}
spotpython tuning: 10570.32421875 [#######---] 66.67% 

In fun(): config:
{'act_fn': Swish(),
 'batch_norm': True,
 'batch_size': 16,
 'dropout_prob': np.float64(0.0),
 'epochs': 32,
 'initialization': 'Default',
 'l1': 128,
 'lr_mult': np.float64(5.944385753623345),
 'optimizer': 'Adagrad',
 'patience': 8}
train_model result: {'val_loss': 13746.05078125, 'hp_metric': 13746.05078125}
spotpython tuning: 10570.32421875 [########--] 75.00% 

In fun(): config:
{'act_fn': Swish(),
 'batch_norm': True,
 'batch_size': 8,
 'dropout_prob': np.float64(0.0),
 'epochs': 32,
 'initialization': 'Default',
 'l1': 64,
 'lr_mult': np.float64(7.625070160838597),
 'optimizer': 'RAdam',
 'patience': 8}

train_model result: {'val_loss': 22999.564453125, 'hp_metric': 22999.564453125}
spotpython tuning: 10570.32421875 [########--] 83.33% 

In fun(): config:
{'act_fn': Swish(),
 'batch_norm': True,
 'batch_size': 8,
 'dropout_prob': np.float64(0.25),
 'epochs': 64,
 'initialization': 'Default',
 'l1': 128,
 'lr_mult': np.float64(9.273567842021258),
 'optimizer': 'Adagrad',
 'patience': 8}
train_model result: {'val_loss': 3871.7509765625, 'hp_metric': 3871.7509765625}
spotpython tuning: 3871.7509765625 [#########-] 91.67% 
| name           | type   | default   |   lower |   upper | tuned             | transform             |   importance | stars   |
|----------------|--------|-----------|---------|---------|-------------------|-----------------------|--------------|---------|
| l1             | int    | 3         |     3.0 |     8.0 | 7.0               | transform_power_2_int |         0.00 |         |
| epochs         | int    | 4         |     4.0 |     9.0 | 6.0               | transform_power_2_int |         0.00 |         |
| batch_size     | int    | 4         |     1.0 |     4.0 | 3.0               | transform_power_2_int |         0.00 |         |
| act_fn         | factor | ReLU      |     0.0 |     5.0 | Swish             | None                  |         0.00 |         |
| optimizer      | factor | SGD       |     0.0 |    11.0 | Adagrad           | None                  |         0.00 |         |
| dropout_prob   | float  | 0.01      |     0.0 |    0.25 | 0.25              | None                  |         0.00 |         |
| lr_mult        | float  | 1.0       |     0.1 |    10.0 | 9.273567842021258 | None                  |         0.00 |         |
| patience       | int    | 2         |     2.0 |     6.0 | 3.0               | transform_power_2_int |         7.21 | *       |
| batch_norm     | factor | 0         |     0.0 |     1.0 | 1                 | None                  |       100.00 | ***     |
| initialization | factor | Default   |     0.0 |     0.0 | Default           | None                  |         0.00 |         |
l1:  0.004087690277892819
epochs:  0.004087690277892819
batch_size:  0.004087690277892819
act_fn:  0.004087690277892819
optimizer:  0.004087690277892819
dropout_prob:  0.004087690277892819
lr_mult:  0.004087690277892819
patience:  7.206038810372332
batch_norm:  100.0

32.2 Summary

This section showed how to use user-specified data sets for the hyperparameter tuning process with spotpython. The user needs to provide the data set and load it as a spotpython CSVDataset() class.