Learn / What is speculative decoding?
What is speculative decoding?
Speculative decoding is an inference speed-up in which a small, fast draft model proposes several tokens ahead and the large target model verifies them all in a single parallel pass. Accepted tokens are kept, the first rejected one is replaced with the target model's own choice, and the output distribution stays exactly what the large model alone would have produced. The result is lower per-token latency at the same quality, when the conditions are right.
Why decoding is slow in the first place
Generation is sequential: each new token requires a full forward pass, and each pass must stream the model's entire weights from GPU memory. At low batch sizes that streaming, not arithmetic, is the bottleneck, so the GPU's compute units sit mostly idle while tokens come out one at a time (the mechanics are in inference latency & throughput). Speculative decoding is a bet that this idle compute can be spent checking several cheap guesses at once instead of producing one expensive token at a time.
How it works
- Draft. A small model (or a lightweight head reading the large model's hidden states, as in EAGLE) proposes the next tokens, cheaply.
- Verify. The large model processes all proposals in one forward pass, the same parallel trick that makes prefill fast, and produces its own probabilities for each position.
- Accept or reject. Proposals are accepted left to right while they agree with what the target model would sample; at the first rejection, the target's own token is used and drafting restarts from there. The acceptance rule is constructed so the final text follows the target model's distribution exactly.
When the draft model guesses well, several tokens land per large-model pass and generation speeds up by two to three times. When it guesses poorly, the overhead of drafting and verifying is paid for nothing, which is why acceptance rate is the number that decides whether the technique pays off.
The catch: it spends compute you may not have
Speculative decoding converts spare compute into lower latency. That trade is excellent when one or a few users are waiting on a single stream, and it evaporates under load: with many concurrent requests, continuous batching has already put the idle compute to work serving other users, and verification passes now compete with them.
Netra's published benchmark shows the effect in practice: SGLang with EAGLE or DFlash, advanced open-source speculative setups, was neck-and-neck with Netra at a handful of users, then stalled around 3,500 words per second from 32 users up while Netra kept climbing past 7,000. A clever decoding trick on its own isn't enough at scale; memory capacity and scheduling decide who keeps accelerating.
Frequently asked questions
Does speculative decoding change output quality?
No. The verification step accepts draft tokens only when they match what the large model would have sampled, using a rejection rule that preserves the target model's output distribution exactly. You get the same answers, faster.
Do I need a separate draft model?
Classically yes: a much smaller model from the same family. Newer methods like EAGLE avoid a full second model by drafting from the target model's own hidden states with a lightweight head.
When is speculative decoding worth using?
When the GPU has spare compute: low concurrency, latency-sensitive workloads. Under heavy concurrent traffic the spare compute it relies on disappears, and in Netra's published benchmark, engines using speculative techniques stalled at roughly half Netra's throughput once real load arrived.