Ranking models via pairwise human judgment
Measuring model quality with metrics (BLEU, perplexity) can be misleading. A language model might score well on BLEU but produce stilted, unnatural text. For subjective tasks, human judgment is more reliable. Pairwise evaluation asks humans to compare two model outputs for the same input and pick the better one. Repeat this across many input-output pairs and models.
This pairwise preference data feeds an Elo rating system (borrowed from chess). Each model starts with an equal rating. When a model wins a comparison, its rating increases; when it loses, it decreases. The amount depends on the current ratings: an upset (a lower-rated model beating a higher-rated one) changes ratings more than an expected outcome. After many comparisons, the Elo ratings converge to a stable ranking.
Efficiency and bias in human evaluation
Pairwise evaluation is more efficient than rating each output on an absolute scale (1-5), because humans are better at relative judgment ('A is better than B') than absolute scoring ('A is a 3.7'). However, pairwise evaluation is slower than automated metrics, so sample sizes are smaller.
Bias is a risk. Evaluators might favor longer outputs, outputs from well-known models, or outputs that match their prior beliefs. Anonymization (hiding which model produced which output) and detailed rubrics reduce bias. Disagreement between evaluators is common and often informative: outputs with low inter-rater agreement might be genuinely ambiguous or reflect gaps in the rubric. Modern LLM evaluations increasingly use Elo-based human ratings as the ground truth, especially for open-ended generation tasks.