Gaming the public test set
Kaggle and other machine learning competitions publish leaderboards where teams submit predictions on a public test set and see immediate scores. The problem: if hundreds of teams submit thousands of predictions, each trying to maximize public score, someone will eventually find patterns in the public data that do not generalize to the hidden test set (used for final ranking). The public score is optimistic.
Leaderboard overfitting happens when a team optimizes for the specific public test examples. Maybe the public set has more of a certain class, or examples from a particular source. A model tuned to exploit this quirk will rank high on the public leaderboard but crash on the hidden set. The gap between public and private leaderboard position is the smoking gun.
Mitigation: train/val/test discipline
The safest approach is to treat public score as one data point, not the objective. Create your own validation set from the training data, and primarily optimize on that. Submit to the public leaderboard sparingly, once you have confidence in the model. Avoid iterating thousands of submissions, each tweaked by 0.001 points.
Organizations running competitions can also dampen leaderboard overfitting by holding back half the test set (private) for final ranking, or by adding noise to public scores. But participants cannot control this. The lesson: leaderboards are noisy and incomplete signals. Trust your own validation more than public scores, especially late in a competition when public leaderboards are heavily optimized by many teams. Generalization to held-out data is what matters.