Detecting when training stops improving
Early stopping monitors a holdout validation set during training. Training loss generally decreases monotonically (the model sees each training example and overfits to it), but validation loss often falls for a time, then rises as the model begins to memorize irrelevant patterns.
When validation loss starts climbing, training should stop. Continuing beyond this point trades generalization for memorization.
The patience parameter and plateau detection
In practice, validation loss is noisy: a single batch might degrade it temporarily. Most implementations use a patience window (e.g., stop if validation loss doesn't improve for 10 consecutive evaluations). This allows the model to escape temporary plateaus without halting prematurely.
The patience hyperparameter is a tradeoff: high patience risks overfitting; low patience risks underfitting. A typical choice is 5-20 evaluations, depending on validation frequency.
Saving the best checkpoint
The key detail: always save the model checkpoint at the best validation loss, not the final checkpoint. The best checkpoint may be 100 steps before the stopping signal fires. Without this, you'd deploy a worse model.