Blocking future context during autoregressive training
A causal attention mask is an upper-triangular matrix of zeros and negative infinities applied to attention weights. It prevents each token from attending to tokens that come after it. At position i, attention can only see positions 0 to i, not i+1 to n.
This constraint is essential for autoregressive training, where the model must predict the next token given only the past. Without the mask, the model could 'cheat' by looking at the answer before predicting, learning a trivial mapping instead of genuine sequential reasoning.
Difference between training and inference
During training, the mask enables efficient batch processing: a single forward pass computes loss for all positions simultaneously. During inference (generation), you don't apply the mask; you generate one token at a time, sequentially. Each new token is computed in isolation, attending to all previous tokens.
This asymmetry is a common point of confusion. The mask forces the learning problem; inference has no mask because you're not trying to predict future tokens, just generate the next one.
Encoder-decoder models and bidirectional attention
Encoders (like in BERT or the encoder half of a T5) do not use causal masks; they attend bidirectionally because they're not generating, just encoding. Decoders (like in GPT or the decoder of T5) use causal masks during both training and inference to enforce left-to-right generation.