Sequence generation: one token at a time vs. parallel denoising
Autoregressive models generate left to right, predicting the next token conditioned on all previous tokens. Each token depends on its predecessors, creating a sequential dependency chain. Diffusion models start with pure noise and iteratively refine it over many steps, each step reducing noise across the entire sample in parallel.
Autoregressive is simple and has strong theoretical grounding; diffusion is more parallelizable and can benefit from efficient denoising strategies.
Speed and parallelism tradeoffs
Autoregressive generation cannot be parallelized: generating token N requires token N-1. A 100-token sequence takes 100 forward passes. Diffusion also takes many steps (typically 20-1000), but each step is a single matrix operation on the full sequence, highly parallelizable on GPUs.
In practice, autoregressive is faster for text (fewer tokens, fast forward passes), while diffusion can be faster for images (many pixels, more parallelism on denoising).
Quality and exposure bias
Autoregressive models suffer from exposure bias: they're trained on ground-truth histories but generate with predicted histories. A single early mistake compounds down the sequence. Diffusion models have no exposure bias because they're trained on noisy versions of ground truth throughout.
For text, autoregressive remains dominant because exposure bias is manageable with proper training techniques. For images, diffusion has become standard because of superior visual quality.