Why pretraining loss follows a power law
Large language model pretraining exhibits a consistent pattern: loss decreases smoothly as the model ingests more tokens, following a power-law relationship rather than exponential decay. This shape is not coincidental. Early in training, the model is still learning to parse basic syntax and vocabulary; loss drops rapidly. As training progresses and the model captures increasingly subtle linguistic patterns, each additional token produces smaller gains. The underlying mathematics of gradient descent on high-dimensional data naturally produces this curve.
The billion-token regime and diminishing returns
Modern LLM pretraining runs over hundreds of billions to trillions of tokens. The power-law curve means that while loss continues to improve, the rate of improvement slows predictably. This regime defines practical compute budgets: doubling tokens typically reduces loss by a consistent percentage, not by half. The curve also reveals why stopping early wastes compute (steep part of the curve still has room to descend), and why running indefinitely produces marginal gains at exponential cost. The sweet spot is where the cost to reduce loss by one unit matches the value that unit of reduced loss creates.