Learning from mistakes: residual targeting
Gradient boosting is a sequential ensemble method: train a weak learner (shallow tree), compute its prediction errors (residuals), then train the next tree to predict those residuals. Add this second tree's predictions to the first tree's. Now train a third tree to predict the errors of the combined first two, and so on. Each new tree corrects the mistakes of all its predecessors.
This is more sophisticated than averaging many independent trees (Random Forest). Boosting concentrates learning on hard examples: samples where earlier trees were wrong get higher weight in the next iteration, forcing the ensemble to improve on difficult cases. The result is a smaller ensemble that often outperforms much larger random forests.
Speed, regularization, and practical implementation
Gradient boosting is sequential, so it cannot be easily parallelized like Random Forest. But it needs fewer trees to reach the same accuracy, offsetting this. Implementation matters: XGBoost, LightGBM, and CatBoost use approximations (quantile sketching, leaf-wise instead of level-wise tree growth) to scale to millions of rows in seconds.
Regularization is crucial. Learning rate (shrinkage) makes each tree's contribution small so the ensemble isn't overfit by a single tree. Maximum tree depth, minimum samples per leaf, and early stopping (halt boosting if validation error stops improving) prevent overfitting. Well-tuned gradient boosting wins many tabular data competitions, but careless tuning easily overfits.