nn.optimizer

nn.optimizer

Torch optimizer factory with per-optimizer base learning rates.

Ported from spotpython.hyperparameters.optimizer. Instead of tuning one learning rate across optimizers with very different scales, a single lr_mult hyperparameter multiplies each optimizer’s individual base learning rate (e.g. 1.0 for Adadelta, 0.001 for Adam). The constructor arguments replicate spotPython exactly so that tuning results remain comparable; in particular SparseAdam and RAdam ignore lr_mult.

Requires the torch optional extra (pip install 'spotoptim[torch]').

Functions

Name	Description
optimizer_handler	Return a torch optimizer with spotPython’s base learning-rate convention.

optimizer_handler

nn.optimizer.optimizer_handler(optimizer_name, params, lr_mult=1.0, **kwargs)

Return a torch optimizer with spotPython’s base learning-rate convention.

Parameters

Name	Type	Description	Default
optimizer_name	str	Name of the optimizer. Options: - “Adadelta”: base lr 1.0 (times `lr_mult`) - “Adagrad”: base lr 0.01 (times `lr_mult`) - “Adam”: base lr 0.001 (times `lr_mult`) - “AdamW”: base lr 0.001 (times `lr_mult`) - “SparseAdam”: fixed lr 0.001 (`lr_mult` ignored) - “Adamax”: base lr 0.002 (times `lr_mult`) - “ASGD”: base lr 0.01 (times `lr_mult`) - “LBFGS”: base lr 1.0 (times `lr_mult`) - “NAdam”: base lr 0.002 (times `lr_mult`) - “RAdam”: fixed lr 0.001 (`lr_mult` ignored) - “RMSprop”: base lr 0.01 (times `lr_mult`) - “Rprop”: base lr 0.01 (times `lr_mult`) - “SGD”: base lr 0.001 (times `lr_mult`)	required
params	Union[Iterable, list, `torch`.`Tensor`]	The parameters to optimize, e.g. `model.parameters()`.	required
lr_mult	float	Multiplier for the optimizer’s base learning rate. Defaults to 1.0.	`1.0`
**kwargs	Any	Ignored. Accepted for call-site compatibility.	`{}`

Returns

Name	Type	Description
	`torch`.`optim`.`Optimizer`	torch.optim.Optimizer: An instance of the specified optimizer.

Raises

Name	Type	Description
	ValueError	If `optimizer_name` is not one of the supported names.

Examples

import torch
from spotoptim.nn.optimizer import optimizer_handler

weights = [torch.nn.Parameter(torch.zeros(3))]
opt = optimizer_handler("Adam", weights, lr_mult=0.5)
print(type(opt).__name__, opt.param_groups[0]["lr"])

Adam 0.0005