Three optimizers, three paths through loss space
SGD (Stochastic Gradient Descent) takes steps proportional to the gradient. With momentum, it accumulates past gradients, smoothing out noise. Simple, memory-light, but requires careful learning rate tuning. Adam (Adaptive Moment Estimation) maintains per-parameter learning rates, scaling each dimension by its gradient history. This adapts the step size automatically, trading off memory for robustness. Lion (EvoLved Sign Momentum Optimizer) uses the sign of the gradient (direction only, not magnitude) combined with exponential moving averages, requiring even less memory while often outperforming Adam on large language models.
All three are variants on the same idea: adjust weights by some function of gradients. They differ in how they compute and scale those adjustments.
Convergence and generalization tradeoffs
SGD with momentum often finds sharper minima (solutions that are sensitive to weight perturbations), which can generalize better in some cases. Adam's per-parameter rates tend toward flatter minima, which generalize more robustly but may be over-regularized. Lion balances the two.
For large language models at modern scale, Lion and AdamW (Adam with weight decay decoupled) are standard because the memory cost of Adam (storing two moments per parameter) becomes prohibitive at billions of parameters.
Practical choice criteria
If memory is tight, use Lion. If fine-tuning a pretrained model, use AdamW with a small learning rate. If training from scratch on large scale and memory is available, Adam or Lion are both fine. SGD with momentum is useful for studying optimization but rarely the best choice in practice.