Splitting data multiple ways for stable estimates
A single train-test split can be lucky or unlucky. If the test set happens to be easy, reported accuracy is optimistic. If it is hard, accuracy is pessimistic. k-Fold cross-validation hedges this by splitting data into k disjoint folds, then training k models: for each fold, train on the remaining k-1 folds and evaluate on the held-out fold. Average the k test scores to get a more stable estimate.
Typical k is 5 or 10. With k=5, you get five independent test estimates from 80% training data each. This is more informative than a single 80/20 split. Standard deviation of the k scores also quantifies variance: high variance means performance depends heavily on which fold is held out, signaling an unstable model or small dataset.
Variations and pitfalls
Stratified k-fold ensures each fold has similar class distribution (important for imbalanced data). Leave-one-out cross-validation (k=n, one sample per fold) is the most thorough but computationally expensive for large n. Time-series data requires special care: never shuffle; use forward-chaining (train on past, test on future) to avoid data leakage.
A critical pitfall: hyperparameter tuning on the full cross-validation loop inflates reported performance. Always split into three parts: train, validation (for hyperparameter tuning), and test. Or use nested cross-validation: outer loop for testing, inner loop for hyperparameter selection. Cross-validation is not a substitute for test data; it is a way to estimate the test error when data is scarce. With abundant data, a single large held-out test set is simpler and avoids the risk of accidentally fitting the validation process itself.