The margin: maximizing certainty
Support Vector Machines find the hyperplane (a line in 2D, a plane in 3D, a hyperplane in higher dimensions) that separates two classes with maximum margin. The margin is the distance from the hyperplane to the nearest point on either side. A larger margin is preferable because it indicates greater separation and typically means the classifier will generalize better to new data.
Not all data is linearly separable, so SVMs allow some misclassification (controlled by a regularization parameter C). Points that fall on the 'wrong' side of the margin incur a penalty. The name 'support vectors' refers to the critical points that define the margin; the rest of the data could be ignored.
Kernels: tackling nonlinear boundaries
In the original feature space, many datasets are not linearly separable. SVMs solve this with the kernel trick: instead of finding a hyperplane in the original space, they implicitly work in a high-dimensional transformed space where a linear boundary might exist. Common kernels are polynomial (captures interactions) and RBF (Radial Basis Function, captures local similarity).
SVMs excel at high-dimensional problems, binary classification, and small-to-medium datasets. They can be slow on large datasets (quadratic complexity in the number of support vectors) and require careful scaling and kernel selection. For multiclass problems, SVMs must be extended (one-vs-rest or one-vs-one). Despite these limitations, SVMs remain powerful for structured data and are a solid choice when you have moderate data and interpretability or theoretical grounding matters.