Draft-and-verify acceleration for LLM inference
Standard autoregressive generation requires the full model to process one token at a time, creating a latency bottleneck where each step depends on the previous output. Speculative decoding breaks this chain by running a smaller draft model in parallel with the main model. The draft model rapidly proposes the next 4-6 tokens, and the main model verifies them all at once in a single forward pass.
If the draft predictions match the main model's top-1 choices, they are accepted and the process repeats. If verification fails, the pipeline rewinds and continues from the last accepted position using the main model's output. This maintains generation quality while reducing the number of main-model forward passes.
Throughput gains with zero quality loss
Because the draft model proposes speculative tokens and the main model validates them, the final outputs are identical to traditional decoding (the main model always has the final say). The speedup depends on draft accuracy and cost ratio. With a 3-5x cheaper draft model achieving 80-90% acceptance rate, speculative decoding delivers 2-3x higher throughput on latency-sensitive applications like chat interfaces. Production LLM services commonly use this technique to serve more users with the same hardware.