Why constant learning rates are suboptimal
A constant learning rate assumes the same step size is good throughout training, but this is rarely true. Early in training, large steps are safe because the model is far from good solutions. Near convergence, large steps overshoot and oscillate. Schedules adapt the learning rate to this phase of training.
Without schedules, you often compromise: pick a rate high enough for fast progress but low enough to avoid divergence at the end. A schedule lets you be aggressive early and conservative late.
Linear warmup and cosine decay
Modern default: linear warmup over 5-10% of training (ramping from near-zero to max LR), then cosine decay to near-zero over the rest of training. The cosine is smooth, avoiding sharp steps that might harm convergence. This schedule is inspired by optimization theory but works empirically across many domains.
Cosine decay means the learning rate follows a cosine curve. At 50% of training, it's at 50% of max LR; at 75%, it's at 15%, giving aggressive early decay followed by a long tail of fine-tuning.
Cyclical schedules and restarts
Cyclical learning rates (SGDR) periodically restart from a high rate, allowing the optimizer to explore new basins. This can escape local minima and sometimes find flatter solutions. Modern large models rarely use cycles, but they're valuable for smaller models and irregular data distributions.