Self-attention: internal relationship mapping
Self-attention allows each token to directly attend to all other tokens within the same sequence, creating a global dependency map. The query, key, and value vectors all come from the same sequence, which means every token can learn relationships to its neighbors, distant context, and even itself. This is the core mechanism in encoder stacks, where each layer refines token representations by incorporating information from the entire sequence.
Cross-attention: bridging separate sequences
Cross-attention computes attention between two different sequences. Typically, the query comes from one sequence (the decoder or target stream) while the key and value come from another sequence (the encoder or source stream). This asymmetry allows the decoder to learn which parts of the input are relevant to each output position. Machine translation, image captioning, and question-answering all rely on cross-attention to align information from disparate sources.
A standard transformer encoder-decoder uses self-attention in both the encoder (to process the input) and the decoder (to process previously generated tokens), with cross-attention in the decoder to bind the output to the input context.