Penalizing complexity to prevent overfitting
Weight decay and L2 regularization both add a penalty proportional to the magnitude of weights to the loss function. The effect is to push weights toward zero, simplifying the model. A simple model with smaller weights has fewer degrees of freedom, making it less likely to memorize training noise.
The trade-off is expressiveness: with too much regularization, the model lacks capacity to fit even true patterns. The regularization coefficient (lambda) controls this balance.
Why L2 penalizes large weights
L2 regularization adds (lambda / 2) * sum(w^2) to the loss. Minimizing this term pushes weights toward zero. Gradient descent on this term produces a penalty proportional to each weight's magnitude. Large weights incur heavy penalties, while small weights are nearly free.
This implicit bias toward small weights often finds solutions that generalize better. A model with weight norm 1.0 is 'smoother' than one with norm 100.0, less sensitive to input noise.
Weight decay in modern optimizers
In AdamW (the 'decoupled' variant), weight decay is applied separately from gradient-based updates, not mixed into the adaptive learning rate. This distinction matters: mixed weight decay in plain Adam interacts poorly with per-parameter learning rates, sometimes skipping the weight decay entirely on large-gradient parameters. Decoupled weight decay is more interpretable and often more effective.