Lazy learning: no training, just memory
k-Nearest Neighbors is a non-parametric, instance-based classifier. At training time, it simply stores the training data. At prediction time, for a new point, it finds the k nearest points in the training set (typically using Euclidean distance) and counts which class is most common among them. If k=5 and 4 of the 5 nearest points are 'red' and 1 is 'blue', the prediction is 'red'.
KNN is a lazy learner: it defers all computation to prediction time, unlike eager methods (tree, logistic regression) that build a model during training. This makes it simple to understand and implement, but slow and memory-intensive on large datasets.
Trade-offs: bias, variance, and hyperparameters
Small k (say, k=1) gives low bias but high variance; the model is sensitive to individual training points and noisy data. Large k (say, k=n) averages many points and is stable but biased toward the global class prevalence, ignoring local structure. The optimal k depends on data density and signal-to-noise.
Distance metric matters too. Euclidean distance works in many cases, but Manhattan distance or custom metrics can fit specific domains better. KNN also suffers from the curse of dimensionality: in high dimensions, all points are far apart and the concept of 'nearest' becomes fuzzy. Despite these limitations, KNN is surprisingly competitive on tabular data and baseline comparisons; it is often the first algorithm to try in a new problem.