Kernel Surgery on a Production Qwen Deployment

Our production Qwen3.6-35B-A3B deployment is a dense stack of specializations: a hybrid model that interleaves 30 Gated DeltaNet (linear attention) layers with 10 full-attention layers, all 40 carrying a fine-grained MoE (256 experts, top-8, intermediate 512); FP8 weights with FP8 KV cache; block-parallel speculative decoding; a hierarchical prefix cache; and CUDA graphs over both the decode ladder and the torch.compiled extend path. It serves 32–64k-token agent prompts on single H100s at roughly 24,000 total tokens/second per GPU.

We spent two days doing a full profiling pass over this deployment with one question: where does the GPU time actually go, and what can kernel-level surgery reclaim? The answer turned out to be a story in three acts: a memory-traffic bug hiding inside kernel glue that cost more than the kernel it wrapped, a MoE runner unlock that required fixing three separate latent bugs, and (the part we did not expect) a one-character upstream bug that had been silently deleting one expert's output from every layer, invisible until our changes revived the dead code path it lived on.

This article walks the full pass: the profiling method, the kernel tables, both optimizations with their same-host A/B numbers, and the systematic bug hunt. The numbers are reported with their variance, including the ones that did not go our way.

TL;DR

• Steady-state profiling at production concurrency showed the top costs are MoE expert GEMMs (22.4%), full-attention prefill (15.2%), and, surprisingly, 13.6% of GPU time in pure memory movement around our fastest kernel: transpose-and-materialize glue whose output was 99.6% discarded.
• Fix #1 (lean glue): return the transposes as strided views and let the one consumed row fuse into its gather. Same-host serial A/B, twice: 64k-token cold prefill −11–12%, concurrent throughput +4.7% and +17.8%, mean TTFT −6% to −20%, outputs bit-identical.
• Fix #2 (DeepGEMM MoE on TP1): the runner's standard-dispatch path is dead code upstream, and reviving it surfaced three bugs: a kernel that assumed at least one thread per expert (fails for fine-grained MoE), a masked-layout scratch tensor that scales with tokens × experts (16.25 GiB at our 16k bucket), and a combine kernel that silently dropped expert 0's contribution via an expert_id > 0 guard that should have been >= 0.
• The expert-0 bug cost 20 GSM8K points (0.46 → 0.26) while producing perfectly coherent text. We localized it with a chain of one-boot discriminator experiments and three tensor-level probes: both GEMMs were bit-faithful (0.0017 relative error), the per-slot outputs were correct (0.015), and the error appeared only after the combine (0.061). After the fix, the DeepGEMM route matches the ideal quantized chain exactly as well as the Triton baseline (0.0146).
• Verdict: lean glue shipped to production. The DeepGEMM MoE runner is accuracy-clean and 1.26× faster at our 16k-token chunk size in isolation, but end-to-end throughput-neutral until small-batch dispatch improves, so it stays off. The expert-0 fix is being upstreamed regardless.

Profiling a deployment, not a model

Kernel tables lie when the capture does not look like production. Our harness boots the verbatim production configuration (same flags, same caches, same speculative block size) on one H100, and takes three kinds of captures:

• Stage-separated traces (prefill 32,768→1 token; decode 1→2,048 tokens), matching the shape of our agent workload rather than a synthetic default.
• A graph-off "mapping" boot: with CUDA graphs on, every kernel in a trace attributes to cudaGraphLaunch and nothing else. A second boot with graphs disabled recovers the kernel → Python-callsite mapping, which the analyzer joins back onto the graph-on trace.
• A steady-state concurrent window: a 40-request 32k/2k benchmark runs in the background, and the profiler arms only once ≥12 requests are in flight, capturing 32 scheduler steps of true steady state.

Even the harness taught us something: our first concurrent capture recorded an empty window because the readiness gauge polled a metrics endpoint the production config does not expose: the poll silently read 0.0 against a 404, armed the profiler after the benchmark had already finished, and produced a trace of an idle GPU. Every gauge in a gate needs a liveness check of its own.

Where the time goes

At steady state under the production workload (19 concurrent requests, 32k in / 2k out), the GPU is ~97% occupied after ramp, so kernel share is throughput. The mix:

Two findings framed everything after. First, single-request decode is 75% GPU-idle: speculative decoding at batch 1 is host-orchestration-bound, so no amount of kernel work moves it; that is a scheduler problem for another day. Second, the third-largest GPU consumer was not a compute kernel at all. It was at::native::elementwise_kernel doing direct_copy, plain tensor copies, attributed to the wrapper around our GDN prefill kernel.

Optimization 1: the glue that cost more than the kernel

Our GDN prefill runs on FlashQLA, a TileLang chunked kernel that is 2–2.9× faster than the Triton alternative on Qwen's shapes. But FlashQLA uses a [K, V] state layout while our memory pool stores [V, K], so the integration wrapped it in adapters: gather the state, transpose, run the kernel, transpose back, materialize, scatter. The profile said this glue cost 12.9% of prefill GPU time, twice the 6.5% of the kernel it wraps.

The dominant term was a tensor most of which nobody reads. For prefix caching, the kernel emits its per-chunk recurrent states: for a 16k-token forward, that is 256 chunk states of shape $[32,128,128]$, about 0.5 GB, which the wrapper transposed and materialized with .contiguous() every layer, every chunk. The consumer, the radix-cache state tracker, then index-selects at most one chunk state per request:

Because nothing downstream requires contiguity, the fix is to return the transposes as strided views. We validated it with two independent same-host serial A/B runs (boot baseline, measure, shut down, boot patched, measure; same GPU, because identical configs on different hosts varied by ±10% in our runs, which would drown the effect):

The 64k prefill number is the one we trust most: it replicated within 0.6 points across runs. The concurrency benchmark's spread (+4.7% to +17.8%) is mostly benchmark variance, not effect variance; the honest claim is "+5–18% depending on load mix, +11–12% on long-prompt prefill." At 32k the effect disappears into fixed host overhead (~±0). This fix is now enabled in the production preset.

Optimization 2: the 22% nobody routes to the fast GEMM

The single largest kernel family, at 22.4%, is the routed-expert GEMMs, running on Triton's fused_moe. Our stack ships DeepGEMM, whose grouped FP8 GEMMs are excellent, but the MoE runner's auto selection only engages DeepGEMM for expert-parallel deployments; on single-GPU tensor parallelism it always picks Triton. Before building anything, we measured whether the switch would even pay, with a microbenchmark of the full pipelines (quantization, scatter, both GEMMs, activation, combine) at production shapes:

A 1.26× win on a 22% share is worth roughly 4–6% of end-to-end prefill, worth building, with routing by forward size baked in from the start. Then came the three bugs.

Bug 1: one thread per expert. DeepGEMM's masked-layout activation kernel launches intermediate/8 threads per block and used one thread per expert to scan the per-expert token counts. Qwen3.6's fine-grained experts give 64 threads against 256 experts, and a host-side check refuses to launch. We rewrote the scan to walk experts in blockDim-sized chunks with a carried prefix, validated against an fp32 reference across four shapes including a worst-case eight-chunk configuration.

Bug 2: scratch that scales with tokens × experts. The masked layout pads every expert to the full token count: at our 16,384-token compile bucket that is a $[256,16384,2048]$ bf16 tensor of 16.25 GiB inside CUDA-graph capture, an instant OOM. Worse, every sub-threshold bucket also pins masked scratch into the shared capture pool at ~1.2 MB per token, so even a 3840-token bucket holds ~4.5 GB hostage. The contiguous route we added caps its scratch with a static worst-case bound,

rows (every expert padded to the next 128-block at most once), with unused rows masked out of the GEMM by a −1 group index, capture-safe because every shape is static per bucket. Routing all buckets ≥1024 tokens through the contiguous layout confines the masked route to the decode graphs (≤720 tokens at our concurrency cap), where its scratch is bounded and its layout is the right design. The scatter workspace is allocated once and sliced per bucket, so nothing transient inflates the capture pools.

With those two fixed, the server booted, captured all 74 piecewise buckets plus the decode ladder, and served. Then the accuracy gate failed.

The hunt: 20 GSM8K points, perfectly coherent text

Same-config GSM8K baseline: 0.465. The DeepGEMM arms: 0.245–0.29, with fluent, plausible outputs and no error anywhere. A drop like that with coherent text means something systematic and mid-sized is wrong in the math. We localized it with a chain of experiments, each one boot or one probe, each designed to cut the hypothesis space in half:

The combine kernel, the weighted sum that folds each token's top-8 expert outputs back together, guarded each slot with expert_id > 0 instead of >= 0. Every token routed to expert 0 simply lost that expert's entire contribution. With 256 experts and top-8 routing, that is ~3% of all token-slots, silently zeroed, in all 40 layers. The model stays fluent, because a residual stream shrugs off a missing summand, but it reasons measurably worse. And because the kernel's only caller is the standard-dispatch DeepGEMM path that upstream never exercises on TP1, the bug was unobservable until our contiguous route revived the path. The sibling kernels in the same file guard correctly; this one had also been accumulating in bf16 where every sibling accumulates in fp32, so we fixed both.

End-to-end confirmation: with the fix, the DeepGEMM arms scored GSM8K 0.505 and 0.555 against the 0.465 baseline. The gate flipped from a 20-point deficit to at-or-above parity (the positive deltas are within ~2.5σ of the 200-question noise; we claim "no longer worse," not "better," though the combine's new fp32 accumulation plausibly helps).

Results, and what actually shipped

The final same-host gate, all three configurations serially on one H100:

Our shipping decisions follow the numbers rather than the effort invested:

• Lean glue: shipped. Replicated, bit-identical, and the largest validated win of the pass.
• DeepGEMM MoE: built, correct, held. The 1.26× kernel-level win at 16k tokens gets diluted by the neutral 8k and losing sub-4k buckets, and the end-to-end throughput deltas sit inside benchmark noise (identical baseline configs spanned 23,976–27,490 tok/s across the day; any <10% single-run bench delta on this stack is weather, not climate). It goes to production when a mixed dispatch (Triton kernels for small forwards inside the DeepGEMM runner) lets the large-bucket win survive contact with the full workload.
• The expert-0 fix: upstreaming regardless. It silently corrupts any future TP1 user of this path, and it costs one character to fix.

What we took away

• Profile the glue, not just the kernels. The fastest kernel in our stack was wrapped in adapters costing 2× its own runtime. Integration seams (layout adapters, .contiguous() calls, dtype casts) deserve their own line in the kernel table.
• Dead code paths downstream are unexercised code paths upstream. Enabling a configuration nobody runs means becoming its first tester. Budget for that: three latent bugs in one unlock is not bad luck, it is the expected density.
• Accuracy gates on every perf change, no exceptions. The expert-0 bug produced fluent text at −20 GSM8K points. A throughput-only gate would have shipped it.
• Discriminators beat debuggers for numeric bugs. Seven binary experiments took us from "accuracy collapsed somewhere in serving" to one guilty kernel; the ideal-quantized-chain reference then convicted a single stage by arithmetic: a weighted average of correct inputs cannot create error.
• Same-host serial A/B or it did not happen. With ±10% host-to-host variance, parallel arms on different machines measure the hardware lottery, not the patch.

FAQ

Why was the glue so expensive relative to the kernel? Prefill GEMM-heavy kernels run near the GPU's compute roof; a plain copy runs at memory bandwidth on bytes that do no work. Moving ~1 GB per layer-forward of state the consumer discards costs as much as computing attention over thousands of tokens.

Couldn't the expert-0 bug be caught by unit tests? Ours passed: the kernel matched its reference on the routes' components. The guard bug lived in a combine kernel whose test coverage upstream is implicit in end-to-end runs that never execute it on TP1. The stage-level probe against an independent fp32 chain is the test that caught it; we've kept all three probes as permanent gate entrypoints.

Why does a missing expert only cost 20 points instead of destroying the model? The MoE output is added to a residual stream, and the router's top-8 weights spread mass across experts; losing one of eight summands ~3% of the time degrades the computation smoothly. That is precisely what makes this bug class dangerous: there is no crash, only a model that is quietly worse.

Is the 1.26× DeepGEMM win real if end-to-end is neutral? Both are real. The microbenchmark isolates the MoE layer at one forward size; production runs a distribution of forward sizes plus everything else. A 26% win on 22% of GPU time at the dominant bucket is ~4–6% end-to-end ceiling, and the sub-4k buckets currently give a chunk of it back. Closing that gap is dispatch engineering, not GEMM engineering.

What hardware and stack does this apply to? Single-H100 serving of Qwen3.6-35B-A3B (FP8 block-quantized weights, FP8 KV) on our SGLang-based runtime. The expert-0 fix and the chunked expert-scan fix apply to any fine-grained MoE served through the DeepGEMM standard-dispatch path; the lean-glue pattern applies anywhere a layout adapter materializes more than its consumer reads.

Free tools from Netra

All of these run in your browser. No signup, and your data is not uploaded to us.

• LLM VRAM Calculator: size the GPU for any model, precision and context length.
• Fine-Tuning Memory Calculator: the training-time memory budget.
• Token Counter: measure prompts under the GPT, Claude and Llama tokenizers.
• Free OCR: turn scans and PDFs into LLM-ready text.
• Web to Markdown: strip a page to clean Markdown before it reaches a prompt.
• Limbus: local-first image segmentation.
• sam3.c: SAM3 inference in pure C.

References

• Fusing RMSNorm and FP8 Quantization Under torch.compile: the previous entry in this optimization series, with the compile-safety design this stack builds on.
• Radix Islands: How Eager SWA Eviction Doubled Our Serving Concurrency: the caching-side companion, including the byte-equality gating methodology contrasted here.
• How to Speed Up GDN Kernels for Qwen Models: background on the Gated DeltaNet kernels whose integration glue Optimization 1 repairs.
• DeepGEMM: the grouped FP8 GEMM library behind Optimization 2.
• Up to 4x faster than vLLM: the serving benchmark methodology, including why we insist on same-host serial A/Bs.