Benchmark diversity and saturation
Machine learning benchmarks are the most studied datasets in the field. MMLU (Massive Multitask Language Understanding) covers 57 subjects in multiple-choice format. GSM8K has middle-school math word problems. HumanEval tests code generation. MATH has competition-level problems. BBH (Big Bench Hard) is a collection of hard reasoning tasks. ARC is a set of science questions.
Each benchmark has different characteristics. MMLU tests breadth of knowledge; HumanEval tests coding ability; MATH tests mathematical reasoning. A model can excel on one and fail on another. No single benchmark tells the whole story. Modern LLM papers report performance on many benchmarks to show capability across different domains.
Saturation and the scaling plateau
As models improve, benchmarks eventually saturate: performance approaches 100% and stops improving. MMLU, once a separating benchmark, is now saturated by large frontier models (90%+ performance). HumanEval, too, has seen ceiling effects at scale. MATH and other hard reasoning tasks show more separation between models but are also starting to saturate.
Saturated benchmarks lose discriminative power. If benchmark A is 95% saturated and benchmark B is 50%, then B is more informative for ranking models. Researchers constantly need new benchmarks to distinguish progress. This arms race between models and benchmarks is necessary but creates an incentive for overfitting to specific benchmarks. The broader lesson: no single metric or benchmark fully captures model capability. Real-world performance often diverges from benchmark rankings, especially for tasks unlike the benchmark's distribution.