One Blocking Memcpy Per Verify Step

After two articles about prefill (kernel surgery, then the burst-admission stall), the frontier moved to the other half of serving: decode. Our production Qwen3.6-35B deployment runs block speculative decoding: a small draft model proposes a block of B = 12 tokens, the 35B target model verifies the whole block in one forward pass, and on average τ ≈ 4.4 of the 12 land per round (the accepted prefix plus the bonus token the verify pass yields for free). Every profile of that loop showed the same signature: at single-request decode the GPU was roughly 25% busy, and even under a decode-heavy concurrent load it idled a quarter of the time, in a metronome of 1-to-3-millisecond host gaps between kernel bursts, one per verify round. The kernel tables said decode was cheap. The wall clock said otherwise.

We had proof this host-boundedness was real and expensive: an earlier experiment quantized the draft model to fp8, removed 7% of decode GPU work, and moved end-to-end latency by 0.24%, which is to say not at all. When shaving kernels does nothing, the kernels are not the bottleneck. This article is the story of finding what was: one blocking device-to-host copy per verify round, plus ninety eager host ops of per-layer glue hiding behind it, plus a red herring generated by the profiler itself. It comes with the arithmetic this time: the round timing model, the upper-bound proof that made the sync deletable, and the padding math behind three kernel optimizations we then measured and rejected. It ends with a correction to the previous article's closing guess.

TL;DR

• Per-token latency in block speculative decoding is TPOT = Tround / τ. With overlap scheduling, Tround = max(TGPU, Thost); with even one synchronization point it degrades to TGPU + Tserial. Our worker had exactly one: it copied accept-dependent sequence lengths to the CPU every verify round, so the host could never run ahead and every host microsecond became GPU-idle time.
• The fix is planning with a provable upper bound instead of the exact value: every CPU-side consumer of that copy turned out to be sizing metadata, and for the draft's sliding-window cache the bound min(Lplan, W + P − 1) dominates the exact windowed length (proof below). Two data-dependent gathers beneath the sync became one fixed-shape gather. The device-side lengths stay exact, so outputs are bit-identical: 24/24 temp-0 probes matched, in both gate runs.
• A second capture then exposed the next layer: the linear-attention state commit ran ~90 eager host ops per round (2 strided-view copies + 2 weight dtype conversions + 1 index rebuild, times 30 layers). Folding the copies into the verify CUDA graph and caching the rest removed all of it from the host path.
• Net effect: decode-window GPU occupancy 74.8% → 93.7%, verify round 20.9 → 16.2 ms, decode-heavy mean TPOT 4.13 → 3.88 ms (−6.1%), throughput 1,588 → 1,690 tok/s (+6.4%).
• The scariest number in the trace, a 2.5 ms cudaGraphLaunch, was the profiler measuring itself: launch host-time fits t(N) ≈ t0 + κN over graph node count N, with the same κ ≈ 2.5 µs/node for both our graphs (~100-node draft, ~940-node target). Real launch cost is an order of magnitude smaller.
• With decode finally GPU-bound we re-measured three kernel ideas. All lost, each with arithmetic attached: a grouped-GEMM expert route (−15%: 3 tokens per expert against 64-row tiles is ≥95% padding), draft fp8 (neutral: launch-bound matmuls do not care about bytes), and an fp8 draft sampling head (−3.8%: it traded 2.8% of accept length for 2% of round time, and TPOT = Tround/τ is unforgiving about the denominator).

The round timing model

One verify round at batch size 8 does the following work. On the GPU: the draft model's forward over the 12-token block (captured in a ~100-node CUDA graph), the target model's verify forward over all 8 × 12 = 96 draft tokens (a ~940-node graph), and the eager kernels between and after them: proposal sampling against the LM head, the accept computation, the recurrent-state commit for the 30 linear-attention layers, the draft-cache KV append. Measured GPU time per round: TGPU ≈ 15.6 ms. On the host: building the draft block, replay metadata for both graphs, two graph launches, and roughly a hundred eager kernel launches; call it Thost ≈ 5 ms of dispatch work.

Overlap scheduling exists to make those two overlap: while the GPU executes round N, the host prepares and enqueues round N+1, and the round period becomes max(TGPU, Thost) = 15.6 ms. What we measured instead was Tround ≈ 18.3 ms unprofiled (TPOT 4.13 ms × τ 4.43), i.e. the serialized form TGPU + Tserial with about 2.6 ms of host work sitting rigidly after each round's GPU completion. The stacked capture (torch profiler with Python stacks, armed once decode is steady) explained why in one number: 91% of all GPU-idle gap time sat under a single cudaMemcpyAsync, called once per round from the speculative worker, blocking the host for 1.8 ms on average and up to 4.9 ms. During that block the GPU was 80-to-88% busy finishing the previous round. The host was not slow; it was pinned to the GPU's clock, and everything it did afterwards (metadata, two launches, the eager tail) executed against an idle device.

The sync, and the bound that replaced it

The copy was not gratuitous. Our draft model attends over a compact sliding window: it keeps only the most recent W = 4096 tokens of context, in a paged KV cache with page size P = 64. Rebuilding that window's metadata each round needs the request's current length L, and L is the one thing the host cannot know early: it advanced by last round's accept count, which the GPU computed. The worker therefore materialized the windowed length on the CPU with an exact device-to-host copy. Formally, with pages, the windowed (compact) length is not simply min(L, W): the window's left edge must land on a page boundary to keep the page table valid, so the code computes

s(L) = ⌊(L − min(L, W)) / P⌋ · P  (the page-aligned window start), and c(L) = L − s(L)  (the compact length actually resident).

Note c(L) can exceed W by up to P − 1 tokens of alignment slack, and it is not even monotone in L (it sawtooths every time the aligned start jumps a page). Two more syncs hid directly beneath this one, on the same accept-dependent tensor: the window's page-table rebuild used a gather whose output size is data-dependent (a lengths.max().item() to size the loop and a boolean masked_select to pack the result), and each is its own device round-trip. Fixing the top sync alone would just hand the stall to the next one; all three had to go.

The way out is to ask what the CPU-side value is actually for. Tracing every consumer gave one answer: sizing. The CPU lengths decide how many bytes of attention metadata to prepare and how many page-table columns to copy; the exact masking inside the kernels is always driven by the device-side tensor, which never left the GPU. A sizing input does not need to be exact, it needs to dominate the true value. And the scheduler already maintains a sync-free dominator: for overlap allocation it tracks a planning length Lplan = Lcommitted + B, where the CPU-side committed length lags the device value by at most one block, hence Lplan ≥ L always. The claim that makes the sync deletable:

Claim. c(L) ≤ min(L, W + P − 1), and therefore c(L) ≤ min(Lplan, W + P − 1) since min(·, k) is monotone.

Proof. Case L ≤ W: the window start is 0, so c(L) = L = min(L, W + P − 1). Case L > W: the unaligned start is L − W, and flooring to a page boundary discards at most P − 1, so s(L) ≥ L − W − (P − 1), giving c(L) = L − s(L) ≤ W + P − 1; and s(L) ≥ 0 gives c(L) ≤ L. ∎

The data-dependent gathers got the same treatment. The old rebuild packed the window's page-table entries into a flat tensor whose length was Σ c(Li), forcing the size onto the CPU. The replacement is a fixed-shape rectangle: for request i and column j ∈ [0, W + P − 1), write draft_map[i][j] = target_map[i][s(Li) + j] where j < c(Li), and a don't-care slot otherwise, all computed with masked device-side arithmetic. The rectangle's width is a compile-time constant, so nothing about its shape depends on the accept outcome. Columns beyond c(Li) hold garbage by construction, and that is fine for the same reason the bound is fine: attention reads are bounded by the exact device-side lengths, which we never touched. The change is metadata-sizing only, value-identical by construction, and the gate can therefore demand exact equality: temp-0 outputs matched the baseline on all 24 probes, in both gate runs, alongside a GSM8K score inside the usual band.

A red herring from the instrument

The same capture contained a number that looked like the real villain: cudaGraphLaunch taking 2.5 ms of host time for the target graph, every round. A driver call three orders of magnitude slower than its reputation deserves suspicion, and the trace itself provided the control experiment: the draft model's graph launch, in the same rounds, took 0.25 ms. Model launch host-time as affine in node count, t(N) = t0 + κN, and fit the two points: (N ≈ 100, 0.25 ms) and (N ≈ 940, 2.5 ms) give κ ≈ 2.6 µs/node with t0 ≈ 0. That per-node coefficient is not the CUDA driver; it is CUPTI instrumenting every kernel node at launch when profiling is attached. The unprofiled numbers corroborate: rounds run ~2.6 ms faster without the profiler, almost exactly the instrumentation total for both launches plus the eager tail.

We almost scheduled engineering against that number. The general lesson: when a profiler says a launch API is slow, first check whether the cost scales with the thing the profiler instruments. Two graphs with a 9.4x node ratio and a 10x launch-time ratio is the profiler describing itself. The gaps the launches sat in were real, but their cause was the sync upstream; once the host ran ahead, the true launch cost (well under half a millisecond for both graphs) disappeared under 15.6 ms of GPU work.

The second layer: ninety eager ops of state-commit glue

With the sync gone, the follow-up capture showed occupancy at 80.3%: better, not done. The new top host cost, visible now that the big one was dead, was the recurrent-state commit. Our target model interleaves 30 linear-attention (gated delta-rule) layers with full-attention layers; after each verify, every linear layer's recurrent state must be committed at the accepted position. The wrapper doing that ran, per layer per round: two real .contiguous() copies (the k/v tensors captured for the commit are strided views into each layer's fused QKV projection buffer), fp32 conversions of two per-layer gating parameters (frozen weights, converted to the same values every round), and a freshly allocated identity index tensor. Thirty layers times five ops that allocate, dispatch, and launch: ninety-odd eager host operations per round, every round, all pure Python-dispatch latency.

Each fix is deliberately boring, and all are value-identical. The contiguous copies moved to graph-capture time: materializing k/v inside the verify CUDA graph means the copy kernels re-execute on every replay with zero host involvement. That also fixed a memory problem we had not noticed: holding a strided view keeps the whole viewed buffer alive, so each captured graph was pinning every layer's full fused-projection buffer (12,288 hidden columns wide) just to retain the k/v slice of it; the contiguous copies retain less than half the bytes per layer per graph. The fp32 gating parameters became a cache keyed by weight pointer (frozen weights cannot change under us); the identity index became a persistent buffer. The third capture closed the story: 93.7% GPU-busy, one residual gap in the whole window, zero blocking memcpys, and the profiled verify round down from 20.9 ms to 16.2 ms.

The serving gate

Three kernel ideas, measured and rejected

Making decode GPU-bound had a second purpose: it un-hides kernel wins. The draft-fp8 experiment that moved nothing at 75% occupancy deserved a re-trial at 94%, along with two ideas the new kernel table suggested (the expert layers at 45% of decode GPU time, the bf16 sampling head at 4.4%). We ran all three through the same same-host gates. All three lost, and each loss has arithmetic worth keeping:

• Grouped-GEMM experts for decode batches: −15% throughput. A verify step feeds 8 × 12 = 96 tokens, each routed to 8 of 256 experts: 96 · 8 / 256 = 3 tokens per expert on average. The grouped fp8 kernel pads every expert's token count to its tile height (64 rows at minimum), so the padded-to-useful ratio is ≥ 64/3 ≈ 21×: over 95% of the MACs computed are padding. The fused Triton path that keeps per-expert work dense wins the regime, and no routing threshold fixes tile geometry. (The accuracy gate passed cleanly, so the loss is purely shape economics. Beating it needs a kernel specialized for 1-to-8-row expert batches, not a dispatch flip.)
• Draft fp8, re-measured: still neutral (−0.06%). The host-bound excuse is gone and it still does not pay. The reason generalizes: the draft's matmuls are tiny (tens of rows against a 2,048-wide hidden), so their runtime is dominated by kernel launch and setup latency, not bytes moved. Halving the bytes does not help a kernel that spends its time starting up.
• fp8 draft sampling head: −3.8% throughput. The subtle one. Quantizing the head the draft samples from preserves output parity (the target's verify decides every emitted token), so it looked free: halve a 0.62 GB weight read per round for nothing. But TPOT = Tround/τ, so a change wins only if ΔTround/Tround > Δτ/τ. fp8 logits flipped enough argmax ties to cut accept length from 4.28 to 4.16 (Δτ/τ = 2.8%) while the halved read saved about 2% of round time. Net loss, exactly as the inequality predicts. In speculative decoding, accept length is the currency; proposal quality is worth more than proposal speed.

All three knobs stay in the tree, default-off, next to the notes that record why. Negative results with gates attached are how you stop re-litigating the same ideas every quarter.

Closing the previous article's onion

The burst-admission article ended with a caveat: a residual ~0.7 s ramp gap whose stack "points at the request-delivery pipeline". We owe that sentence a correction. Re-analyzing the same capture with the gap-clipped stack method used here put the gap inside the KV allocator: a Triton JIT compile on the scheduler thread, at the first burst-admitted two-request extend. The allocator kernel specializes on the batch-size bucket (a compile-time constant, next_power_of_2(bs)) and on the pointer dtypes of its length arguments, and our prefill path and speculative planner pass different integer widths (int64 versus int32). The warmup's single-shape pass could never cover that product space; the first mixed batch after a deploy paid 684 ms of compilation, on the scheduler thread, mid-serving. Same class as the TileLang stall from that article, one abstraction level down.

The fix is the same medicine: prewarm the variant space at allocator init: all 8 batch-size buckets × the 3 dtype combinations the live call sites produce × 2 kernels = 48 variants, driven by masked single-element dummy tensors (safe because every out-of-range lane is masked). Cost: 57 seconds once per compile-cache volume, 0.6 seconds every boot after. One more chunk-boundary stall fell in the same pass: the scheduler rebuilt the request's full token list (origin + output, a pure Python list concatenation of up to 64k elements) on every 16k-token prefill chunk, while the output half was still empty. Caching one owned copy per request cut 64k cold prefill by another 5.1%, again with bit-identical outputs. Final ramp capture: 95.0% GPU-busy in the ramp window, mega-gaps gone. Five layers deep (TileLang JIT, idle-loop guards, tokenization, position-id construction, allocator JIT), the burst-admission onion is now, actually, closed.

What we took away

• In an overlapped scheduler, one sync costs more than any amount of async work. The round period is max(TGPU, Thost) pipelined and TGPU + Tserial synced; the difference was 6% of TPOT. Hunt syncs before you hunt milliseconds.
• Exactness is often a sizing requirement wearing a correctness costume. Every consumer of the synced value needed a dominator, not the value. Prove the bound (ours took four lines and an exhaustive check), keep the exact tensor on the device, and the gate can still demand bit-identical outputs.
• Calibrate the profiler against itself. Two graphs with a 9.4x node-count ratio and a 10x launch-time ratio is a linear fit for the instrument's own overhead: κ ≈ 2.6 µs per node. The draft graph was the control group we did not know we had.
• Re-measure rejected ideas when the regime changes, and expect them to lose anyway. Occupancy going from 75% to 94% revived three kernel hypotheses; the gates killed all three, each with a transferable reason: tile padding at 3 tokens per expert, launch-bound matmuls, and the accept-length inequality ΔT/T > Δτ/τ.
• Publish corrections. Our "request-delivery pipeline" guess was wrong, and the wrong guess was load-bearing: it pointed at another process entirely. The stacked capture pointed at a compiler, again. Untraced waits keep being compilers.

FAQ

Why was the sync only worth 3% on its own? Because behind it sat the state-commit glue: once the host stopped waiting, it was still spending milliseconds of eager dispatch per round, so occupancy only rose to 80.3%. Serialization costs stack like the ramp layers did: each fix exposes the next. The two fixes together were worth the 6%.

Is an upper bound really safe for attention metadata? For sizing, yes, and only for sizing: buffer capacities and page-table column counts can be over-provisioned by up to one page harmlessly, and the kernels bound their actual reads with the exact device-side lengths, which never left the GPU. The claim is not "approximate lengths are fine", it is that the CPU-side value participates in no computation whose result depends on it beyond allocation size. We verified that by tracing every consumer before writing the bound, checked the bound exhaustively over the reachable parameter space, and then let the exact-match output gate be the final referee.

Why not capture the whole verify round in one CUDA graph and delete the glue wholesale? It is the logical endpoint, and at batch size 1 (where decode remains host-bound) probably the right one. At production concurrency the two-graph-plus-eager structure now overlaps fully, so the remaining win is small; whole-round capture is real engineering (the accept computation and state commit have data-dependent shapes that would need graph-friendly reformulations, like the fixed-shape rectangle above) and it sits behind higher-leverage work.

What is actually left in decode? The GPU floor: expert layers at 45% of decode GPU time (needs a grouped kernel specialized for a few rows per expert), the linear-attention verify kernels at 11%, and, above everything, the denominator: at τ ≈ 4.3 of a 12-token block, every 1% of proposal quality is 1% of TPOT, which makes draft-model training the highest-leverage line item in the whole decode budget.

What stack does this apply to? Single-H100 serving of Qwen3.6-35B-A3B on our SGLang-based runtime: block speculative decoding (12-token draft blocks, greedy block verification), a compact sliding-window draft cache, overlap scheduling, CUDA graphs for the draft and target forwards. The specific sync is ours; the classes (accept-dependent metadata syncs, data-dependent gathers in per-step paths, per-layer eager glue in state commits, profiler-inflated graph launches) apply to any speculative decoder with an overlapped scheduler.

Free tools from Netra

All of these run in your browser. No signup, and your data is not uploaded to us.

• LLM VRAM Calculator: size the GPU for any model, precision and context length.
• Fine-Tuning Memory Calculator: the training-time memory budget.
• Token Counter: measure prompts under the GPT, Claude and Llama tokenizers.
• Free OCR: turn scans and PDFs into LLM-ready text.
• Web to Markdown: strip a page to clean Markdown before it reaches a prompt.
• Limbus: local-first image segmentation.
• sam3.c: SAM3 inference in pure C.

References

• Three Layers of a Burst-Admission Stall: the previous entry, whose closing caveat this article corrects.
• Kernel Surgery on a Production Qwen Deployment: the prefill-side kernel work and the gating methodology used throughout.
• How To Speed Up GDN Kernels for Qwen Models: background on the gated delta-rule linear-attention layers whose state commit stars in the second half.
• Leviathan et al., Fast Inference from Transformers via Speculative Decoding: the acceptance framework behind "accept length is the currency".
• Up to 4x faster than vLLM: the serving benchmark methodology, including same-host serial A/Bs.