How to Speed Up GDN Kernels for Qwen Models

The newest Qwen models replace most of their attention layers with Gated DeltaNet (GDN), a linear attention variant whose state stays a fixed size no matter how long the context grows. That trades the KV cache problem for a different one: a recurrent kernel that is genuinely hard to make fast on a GPU, and that now sits on the critical path of every prefill and every training step.

The Qwen team's FlashQLA (MIT licensed) makes that kernel 2 to 3x faster forward and about 2x faster backward than the Flash Linear Attention Triton baseline on Hopper. The interesting part is not the number, it is the three techniques behind it, because each one generalizes to other recurrent kernels. This article works through them: why the GPU sits idle in the first place, how the gate's own decay is exploited to parallelize a sequential recurrence, and how the kernels are scheduled. The measured results come from the repo's published H200 benchmarks.

TL;DR

• GDN's chunked prefill launches roughly batch x heads thread blocks. Under TP8 that is 8 blocks on a 132-SM H200, so 94% of the GPU is idle.
• The fix exploits the gate itself: contributions decay as $e^{\sum g}$, so once accumulated decay passes e^-10 ≈ 4.5 x 10^-5 (far below bf16 resolution), history can be truncated and one sequence split into parallel segments.
• The split length falls out of a latency model: $L_{cp}^{\ast }\propto \sqrt{HN/P}$, balancing sequential depth against parallel width.
• Kernels are fused into a few warp-specialized TileLang kernels: 512 threads as one producer plus three consumer warpgroups, overlapping data movement, Tensor Core and CUDA core work.
• Measured on H200: 0.31 ms vs 0.91 ms against FLA for a 32K prefill at TP8, with backward at 1.4 to 1.9x across 16K to 256K tokens.

What GDN actually computes

Standard attention answers every query by re-reading the whole history, which is why its cache grows without bound (we derived that cost in how much VRAM you need to run an LLM). GDN instead maintains a fixed-size state matrix $S\in {\mathbb{R}}^{d_k\times d_v}$ per head, updated once per token by a gated delta rule:

Two scalars per head drive it: ${\alpha }_t=e^{g_t}$ is a decay gate that forgets old state, and ${\beta }_t$ is a write strength. The $(I-{\beta }_t{k}_t{k}_t^{\top })$ factor is the delta rule: before writing a new value, it erases whatever the state already stored in the direction of $k_t$. The state is 128 x 128 per head, forever, regardless of context length.

Token-by-token application is hopeless for prefill, so the practical formulation processes chunks of 64 tokens: matrix work inside each chunk, one state hand-off between chunks. The awkward part is intra-chunk causality, which requires inverting a lower-triangular interaction matrix. FlashQLA solves

as a dedicated triangular-solve kernel (kkt_solve), one 64 x 64 system per chunk per head. That structure, chunked matmuls plus a sequential state chain, is what everything below is optimizing.

The real enemy: an idle GPU

Here is the part most kernel writeups skip. The fused forward kernel launches a grid of roughly batch x heads thread blocks, because the value head dimension (128) fits in a single block's tile. Now count what tensor parallelism leaves behind. Qwen3.5's largest models have 64 value heads; TP slices heads across GPUs, so at TP8 each GPU holds 8 heads. One long sequence, batch 1, is then 8 thread blocks, on an H200 with 132 streaming multiprocessors:

No amount of instruction-level tuning fixes a kernel that occupies 6% of the machine. The chunk chain is sequential in time, the heads are sliced away by TP, and the batch is 1 because it is one user's long prompt. Something has to manufacture parallelism, and the only direction left is along the sequence.

Lever 1: the gate is also a license to parallelize

Splitting a recurrence along the sequence looks illegal: segment two needs the state left by segment one. FlashQLA's observation is that the gate makes the dependency local. Every step multiplies the state by ${\alpha }_t=e^{g_t}$ with $g_t<0$, so a chunk's influence on the state decays exponentially with distance:

So each split does not need the true state from the beginning of time. It needs a warmup window: just enough preceding chunks that everything older has decayed past the threshold $\sum g<-10$. The kernel get_warmup_chunks scans the gates to find that window per head, fused_gdr_h recomputes the carry states, and correct_initial_states stitches them into each segment. The error introduced sits below bf16's own rounding noise, and the repo's test suite hammers the kernel a thousand runs in a loop to confirm the results stay within tolerance of the reference.

How long should each segment be? Too short and the warmup overhead and sequential stitching dominate; too long and the SMs starve again. FlashQLA models one step's latency as a sequential term plus a parallel term and minimizes it:

where $H$ is heads, $N$ total chunks in the batch, and $P$ the SM count. The implementation scales the optimum by an empirical factor of 3, rounds to a power of two for scheduling and alignment, and floors it at 4 chunks so the pipeline stages inside the kernel stay full. All of this switches on automatically only when $B\cdot H$ fails to saturate the SMs, so short-sequence and large-batch cases pay nothing.

Lever 2: do less arithmetic, not just faster arithmetic

The second lever is algebraic. GDN's naive dataflow is generous with exponentials and elementwise traffic: gates are exponentiated per use, decay factors are re-applied per chunk, and intermediate products are materialized between kernels. On Hopper the special function units that evaluate $e^x$ are a scarce resource compared to Tensor Cores, so SFU-heavy inner loops stall the pipeline in ways a matmul never would.

FlashQLA reformulates the forward and backward flows so that gate cumulative sums are computed once per chunk (chunk_local_cumsum) and folded into the matrix operands, the triangular inverse is solved once into $A$ and reused by both the output path and the state update, and the backward pass recomputes hidden states in one dedicated pass (fused_gdr_h) instead of storing every intermediate. Same numerics, strictly fewer Tensor Core, CUDA core and SFU operations. Nothing here is exotic, which is rather the point: profile which unit is actually the bottleneck, then move work off it by rewriting the algebra rather than the schedule.

Lever 3: warp specialization, the middle path on fusion

There are two failure modes for structuring this computation. Decompose it into a dozen small kernels and you pay launch latency and round-trips through HBM between every step. Fuse everything into one mega-kernel and register pressure collapses your occupancy. FlashQLA takes the middle path: a few fused kernels, chosen so the context-parallel preprocessing and the backward pass can reuse them, each internally scheduled by hand.

Each fused kernel runs 512 threads as four warpgroups: one producer and three consumers. The producer does nothing but issue TMA copies, feeding tiles from HBM into shared memory. The consumers run the Tensor Core matmuls and the CUDA core epilogues. Named barriers with hand-set arrival counts coordinate them, so while one tile is being multiplied, the next is already in flight and the previous one's epilogue is retiring. Data movement, Tensor Core work and CUDA core work overlap instead of taking turns, which is the entire trick of Hopper-era kernel engineering, and TileLang lets the repo express it in Python-ish source instead of raw PTX.

One detail worth stealing: the thread geometry is not a tunable. The barrier arrival counts encode the schedule, and the repo treats them as part of the algorithm. Autotuning knobs are for tile sizes; the pipeline shape is a design decision.

What it adds up to

The repo publishes full H200 benchmarks against FLA Triton 0.5.0 and FlashInfer 0.6.9 across the Qwen3.5 family's head configurations. The single-sequence 32K prefill, the case the occupancy chart above says should be worst, is where the context parallelism shows most clearly:

Backward, which matters for pretraining and fine-tuning, runs 1.4 to 1.9x faster than FLA from 16K to 256K total tokens at 32 heads. If you are budgeting a fine-tune, that multiplies directly into step time; the memory side of that budget is our Fine-Tuning Memory Calculator's job.

Using it, and the fine print

FlashQLA is a drop-in for the FLA kernel it replaces: pip install from source, call chunk_gated_delta_rule(q, k, v, g, beta, ...), done. The constraints are real, though: Hopper (SM90) or newer only, CUDA 12.8+, PyTorch 2.8+, head dimension fixed at 128, chunk size 64, bf16 or fp16 inputs, and grouped-query layouts where value heads are a multiple of key heads. On Ampere it simply raises.

The wider lesson travels further than the library. Linear attention moved the cost of long context from memory (a KV cache that grows per token) to compute shape (a recurrence that resists parallelism), and the winning moves were an occupancy model, a numerically-justified truncation, and a hand-scheduled pipeline. That is the kind of work that decides whether the same GPU serves one user or three. It is also exactly the layer Netra works at: we fine tune, accelerate and deploy models on infrastructure you own, so improvements like these turn into latency and cost you can actually see. If you are sizing that hardware, start with the LLM VRAM Calculator and the arithmetic in our VRAM guide.

FAQ

Does the context-parallel split change the model's output? It introduces error below e^-10 ≈ 4.5 x 10^-5, which is under bf16's own rounding at the state's scale. The repo's accuracy tests run the kernel a thousand times against the FLA reference to keep it that way.

Why is FlashInfer sometimes close? When batch x heads already fills the SMs (many heads, or many short sequences), occupancy is not the bottleneck and everyone's Tensor Cores are similarly busy. The gap opens exactly where occupancy collapses: long single sequences under TP.

Does this help decode too? The headline kernels are chunked prefill, which also covers the training forward and backward. Decode is a different regime (one token, recurrent step) with its own kernel in the repo.

Can I run it on an A100? No. The kernels check for SM90 and raise otherwise; the producer-consumer schedule is built on Hopper's TMA and warpgroup MMA.

Do I still need a KV cache with GDN? For the GDN layers, no: the state is a fixed 128 x 128 per head. Qwen's hybrid models keep some standard attention layers, and those still cache normally, so the sizing maths in the VRAM guide still applies to that fraction.

Free tools from Netra

All of these run in your browser. No signup, and your data is not uploaded to us.

• LLM VRAM Calculator: size the GPU for any model, precision and context length.
• Fine-Tuning Memory Calculator: the training-time memory budget, where the 2x backward speedup here matters.
• Token Counter: measure prompts under the GPT, Claude and Llama tokenizers.
• Free OCR: turn scans and PDFs into LLM-ready text.
• Web to Markdown: strip a page to clean Markdown before it reaches a prompt.
• Limbus: local-first image segmentation.
• sam3.c: SAM3 inference in pure C.

References

• FlashQLA, the Qwen team's kernel library, including the H200 benchmark logs quoted here.
• Yang et al., Gated Delta Networks: Improving Mamba2 with Delta Rule, the GDN formulation.
• Yang et al., Parallelizing Linear Transformers with the Delta Rule over Sequence Length, the chunked delta-rule algorithm.
• Flash Linear Attention, the Triton baseline.
• TileLang, the kernel DSL the fused kernels are written in.