Tracking the growth of AI models
Parameter counts have grown exponentially across AI milestones. AlexNet (2012) had 60 million parameters and marked the start of the deep learning revolution. ResNet-50 (2015) had 25 million. VGG-19 (2014) had 144 million. GPT-1 (2018) introduced a 117 million parameter language model. GPT-2 (2019) jumped to 1.5 billion. GPT-3 (2020) reached 175 billion. Modern models like GPT-4 and Llama 2 are estimated in the hundreds of billions to low trillions. This growth reflects increasing compute availability and the discovery that scale consistently improves performance across tasks. Scaling laws predict this growth: loss improves predictably with parameters, and this property persists across orders of magnitude.
The relationship between size and capability
Larger models are not just better versions of smaller ones; they exhibit emergent behaviors. GPT-1 was a language model; GPT-3 became a general-purpose few-shot learner. Parameter count correlates strongly with downstream capability on diverse benchmarks. However, scaling laws also show diminishing returns: doubling parameters does not double capability. The tradeoff is compute cost: training GPT-3 required many times the FLOPs of training BERT, even adjusting for dataset size. Modern research focuses on efficiency: Chinchilla and subsequent work showed that prior models were undertrained relative to their size, and more careful allocation of parameters and data yields better results. The field is shifting toward efficiency alongside scale.