Learning soft targets from a teacher model
Knowledge distillation trains a small student model to mimic a large teacher model. Rather than matching only the hard labels (dog, cat, bird), the student is trained to match the teacher's probability distributions (soft targets). A teacher might output 0.8 for dog, 0.1 for cat, 0.1 for bird; the student learns to reproduce this softer distribution, not just the winner.
Soft targets carry more information than hard labels. Even incorrect predictions from the teacher carry signal: if the teacher confidently says 'wolf' for a husky, the student learns that huskies are wolf-like, which is useful generalization.
Achieving compression without accuracy loss
A naive small model trained from scratch often loses 5-15% accuracy compared to the teacher. With distillation, that gap shrinks to 1-3%. The student model, though much smaller (1-10x fewer parameters), maintains nearly teacher-level accuracy because the soft targets provide a richer training signal than the original labels alone.
This makes distillation practical for deployment: a 10x smaller model runs faster and uses less memory, with minimal quality loss.
Temperature scaling and loss weighting
The temperature parameter controls softness: high temperature (e.g., 20) makes the teacher's distribution smoother; low temperature (e.g., 1) keeps it sharper. Mixing distillation loss with the original classification loss is also crucial: pure distillation can overfit to the teacher's quirks.