The power-law relationship between model size and performance
Scaling laws describe how model loss improves as you increase parameters, training data, or compute. The relationship is predictable: loss decreases as a power of the scale factor. Doubling model size does not halve the loss; instead, loss drops by a consistent fraction (typically around 15-20%). This pattern holds across diverse architectures and domains, suggesting it captures something fundamental about learning in high dimensions. The Kaplan scaling laws, derived from GPT-3 experiments, established these relationships empirically and made it possible to predict the loss of a model you have not yet trained.
Chinchilla's reallocation of compute toward data
The original Kaplan scaling laws suggested compute should be split unevenly: roughly 2x more toward data than toward model size for optimal performance. Chinchilla research in 2022 re-examined this question and found the optimal split was much closer to even: parameters and data tokens should scale roughly equally for a fixed compute budget. This meant contemporary models like GPT-3 were actually undertrained relative to their size. The insight redirected the field toward models with more data per parameter, improving sample efficiency and final performance. Chinchilla and its successor Gopher demonstrated that more careful allocation of compute produces smaller, more capable models.