Mapping confidence to accuracy
A well-calibrated classifier's predicted probabilities match reality. If a model predicts 70% confidence on 1000 samples, approximately 700 should be correct and 300 incorrect. A poorly-calibrated model might predict 70% on samples where only 50% are actually correct. The reliability diagram (calibration curve) plots predicted probability on the x-axis against actual frequency of the positive class on the y-axis. A perfect diagonal line indicates perfect calibration.
Calibration is different from accuracy. A model can be 95% accurate but poorly calibrated, and vice versa. For applications where you act on predicted probabilities (setting confidence thresholds, cost-weighted decisions), calibration matters as much as raw accuracy.
Diagnosing and correcting miscalibration
The reliability diagram reveals overfitting and underfitting. If the curve bulges above the diagonal, the model is overconfident (predicts high probability but is wrong more often than it thinks). If it stays below, the model is underconfident. Tree-based methods (Random Forest, Gradient Boosting) are often overconfident; logistic regression is usually well-calibrated.
Calibration can be improved post-hoc without retraining. Platt scaling (fitting a logistic regression on the model's predictions) or isotonic regression (a non-parametric calibration) can map predicted probabilities to true frequencies. For critical applications, a held-out calibration set is mandatory: never use the same test set for both model evaluation and calibration, or the calibration will be optimistic.