Converting logits to probabilities and losses
Classification networks output logits (raw unnormalized scores) from the final layer. Softmax converts them to probabilities: each logit is exponentiated, then all are normalized to sum to 1. Softmax is the standard because it is smooth, differentiable, and the probabilities are interpretable. Cross-entropy loss measures how well the predicted probabilities match the true distribution. For a multi-class problem, it is the negative log probability of the true class: -log(p_correct). If the model assigns 90% probability to the correct class, loss is -log(0.9) = 0.10. If it assigns 10%, loss is -log(0.1) = 2.30. This penalty structure makes sense: slightly wrong predictions are cheaper than wildly wrong ones.
Why softmax plus cross-entropy is standard
The combination is mathematically elegant. The gradient of cross-entropy loss with respect to logits is simply (predicted_probability - true_probability), leading to simple and stable gradient updates. Softmax ensures probabilities sum to 1 and are bounded between 0 and 1, preventing extreme values. Alternatives like Hinge loss (for SVMs) or Focal loss (for imbalanced data) exist and suit specific problems, but softmax-cross-entropy is the default for multi-class classification because it handles most cases well, is numerically stable (when implemented carefully), and benefits from decades of optimization. The pairing is so common that many frameworks provide a fused implementation that computes both efficiently and prevents numerical issues that can arise from computing them separately.