The path from input to prediction
The forward pass executes the computational graph from input to output. For a typical feedforward network: embed the input, apply a linear layer (matrix multiply) to project to a hidden dimension, apply a nonlinearity (activation function), apply another linear layer, and so on. Each operation is deterministic given the weights and input. The final layer produces logits (raw scores before softmax), which are converted to probabilities and used for loss computation. The forward pass is the inference mechanism: it runs at train time to compute loss and at test time to make predictions.
Computational cost and the role of nonlinearities
Linear layers are cheap (just matrix multiply), but stacking them without nonlinearities collapses to a single linear operation, reducing expressiveness to shallow models. Nonlinearities like ReLU break this linearity, allowing networks to learn nonlinear functions. The interleaving of linear and nonlinear operations is what gives deep networks their power. Modern architectures also interleave normalization (batch norm, layer norm) and residual connections to control information flow. Understanding the forward pass clarifies why depth matters: more linear-nonlinear alternations capture more complex decision boundaries. Debugging training often requires tracing activations through the forward pass to spot where gradients might vanish.