One recipe scaled in three dimensions
Decoder-only models like the GPT series follow a single architectural recipe: a stack of transformer blocks with causal attention, applied to autoregressive text generation. The scaling law is well-characterized: double any of three dimensions (width, depth, or data), and you improve performance with predictable diminishing returns.
Width is the hidden dimension size (more parameters per layer). Depth is the number of layers. Data is the number of training tokens. Optimal allocation is roughly balanced: a 1 trillion-token model should use about 70B parameters, not 100B or 50B.
Empirical scaling law exponents
Loss improves roughly as N^-a, where N is the number of parameters and a is around 0.07 (smaller a means faster improvement with more parameters). This means going from 1B to 10B parameters (10x) improves loss by a factor of roughly 10^0.07 or 1.8x. This predictability is powerful: teams can estimate final performance before training, choosing model size accordingly.
The Chinchilla scaling (2020) and recent work suggest that for a given compute budget, most benefit comes from a larger batch size and longer training, not a larger model.
Why decoder-only won
Encoder-decoder models like T5 split parameters between encoding and decoding. Decoder-only consolidates everything into one stack, simplifying the architecture and enabling very long-context training (most of the model's capacity is devoted to sequence understanding). This unified approach has become dominant in recent years.