optimizer.schedule_free.AdamWScheduleFree

optimizer.schedule_free.AdamWScheduleFree(
    params,
    lr=0.0025,
    betas=(0.9, 0.999),
    eps=1e-08,
    weight_decay=0,
    warmup_steps=0,
    r=0.0,
    weight_lr_power=2.0,
)

Schedule-Free AdamW in PyTorch.

As the name suggests, no scheduler is needed with this optimizer. To add warmup, rather than using a learning rate schedule you can just set the warmup_steps parameter.

This optimizer requires that .train() and .eval() be called on the optimizer before the beginning of training and evaluation respectively.

Reference: https://github.com/facebookresearch/schedule_free

Methods

Name	Description
eval	Switch model parameters to evaluation mode (using averaged parameters ‘x’).
step	Performs a single optimization step.
train	Switch model parameters to training mode (using current iterate ‘y’).

eval

optimizer.schedule_free.AdamWScheduleFree.eval()

Switch model parameters to evaluation mode (using averaged parameters ‘x’).

step

optimizer.schedule_free.AdamWScheduleFree.step(closure=None)

Performs a single optimization step.

train

optimizer.schedule_free.AdamWScheduleFree.train()

Switch model parameters to training mode (using current iterate ‘y’).