What perplexity measures
Perplexity is a standard metric for language models. It is the exponential of the average negative log-likelihood the model assigns to a held-out text, and it can be read as the effective number of equally likely choices the model is deciding among at each token. Lower perplexity means the model is less surprised by real text and is therefore predicting it better.
Because it is anchored to next-token prediction, perplexity offers a clean, comparable signal of raw modelling quality on a given corpus, independent of any downstream task.
The scaling relationship and diminishing returns
Across model families, increasing the parameter count reliably lowers perplexity, and the relationship is close to a straight line when both axes are plotted on log scales. This power-law behavior is the heart of neural scaling laws described in work by Kaplan and colleagues and later refined by the Chinchilla study.
The returns diminish: each doubling of parameters yields a smaller absolute drop in perplexity, and gains depend on training on enough data. The curve is consistent and predictable, which is what lets researchers forecast how much a larger model will improve before training it.