Radix Islands: How Eager SWA Eviction Doubled Our Serving Concurrency

Hybrid attention models like Gemma 4 make a simple promise: most layers use sliding-window attention (SWA), which only ever looks at the last $w$ tokens, so most of the KV cache should stay small no matter how long the context grows. In our production deployment, that promise was quietly broken. Each running request was holding roughly 15,500 tokens of SWA KV against a 1,024-token window, a 14x over-retention that made the SWA pool the resource that decided how many requests could run at once.

The culprit is an interaction between two features that are each individually correct: window attention wants to discard old KV, and radix prefix caching wants to keep it. This article walks through why the interaction goes wrong, the fix (tombstone "islands" in the radix tree), how we proved it byte-exact on the full production stack, and what it bought: +32% KV capacity and 21 → 46 concurrently running requests (+42.5% throughput) on the same GPU. It is a companion piece to our explainer on paged attention and continuous batching, which covers the machinery this builds on.

TL;DR

• Radix prefix caching takes tree references on every committed prefill chunk, including SWA KV the window will never read again. Measured hold: ~15,500 tokens per request vs the 1,088 actually needed.
• The fix restructures cached paths into tombstones (full-attention KV kept, SWA slots freed eagerly) plus a live island of the last $w+p$ tokens, so prefix reuse still works.
• Decode islands additionally make generated text prefix-reusable every $cp/4$ tokens; previously only finished requests contributed decode tokens to the cache.
• Outputs are byte-identical at temperature 0 on the full production stack (FP8, FA3, fp8 KV, speculative decoding, piecewise CUDA graphs), verified with a two-boot equality gate.
• The freed hold let us cut the SWA pool from ratio 0.32 to 0.17 and grow total KV capacity 740k → 980k tokens (+32%). Under pool pressure: 21 → 46 running requests, +42.5% throughput, −37% mean latency, zero retractions.

The hybrid KV layout, and who pays for what

Gemma 4 26B-A4B is a 30-layer MoE with hybrid attention: 25 layers use sliding-window attention with window $w=1024$, and 5 layers are full attention. The serving engine materializes this as two device pools sharing one global token index space of size $T$: a full pool holding all $T$ tokens for the 5 full layers, and an SWA pool sized $r\cdot T$ for the 25 SWA layers, where the ratio $r$ is a boot-time knob. (For why KV dominates the memory budget in the first place, see our VRAM guide.)

With fp8 KV on this architecture, a token of index space costs 9.5 KB in the full pool, 95 KB in the SWA pool if fully retained, and 10.24 KB for the speculative-decoding draft KV riding the same index space. The all-in marginal cost is

and given the device byte budget $B\approx 37$ GB left for KV on an H100-80GB, capacity is $T_{\max }(r)=B/c(r)$. The equation says something uncomfortable: every point of SWA ratio is bought with total capacity. At $r=0.32$ a token costs 50.1 KB and the machine holds 740k tokens; at $r=0.17$ it costs 35.9 KB and holds ~1M. If the window only needs 1,024 tokens per running request, why were we paying for $r=0.32$?

The lazy-eviction hold

Ideally the SWA pool holds the last $w$ tokens per running sequence; beyond the window, SWA KV is dead by definition. Two mechanisms prevented that ideal. First, with chunked prefill (chunk size $cp=16384$), every committed chunk is inserted into the radix prefix tree, and the insert takes a reference on the entire chunk's KV (including the SWA KV) and locks the path while the request runs. Tree-owned, lock-protected KV cannot be window-evicted. Second, the decode-time evictor only frees out-of-window KV in the request-owned region past the cached prefix; it never touches what the tree owns.

The steady-state result: roughly one full prefill chunk of SWA KV pinned per running request.

With a hold $H$, the SWA pool binds admission at roughly $N\approx rT/H$ running requests. At $r=0.17$ that predicted ~10.7; the measured bind was 10–11 running requests while the full pool idled at 50%. The mitigation at the time was to over-provision: raise $r$ to 0.32, lifting the bind to ~15–18 at the cost of shrinking total KV capacity by 24%. That is the configuration this work set out to retire: a tax paid on every token to paper over lazy eviction.

The mechanism: tombstones and islands

The island mechanism restructures what the radix tree stores for SWA layers. A cached path becomes three regions:

The key move is that this structure is created eagerly, at chunk-commit time, not lazily under memory pressure. When a chunk is stashed into the tree at total length $L$, the tombstone frontier is placed at

page-aligned with page size $p=64$, and the insert itself frees the SWA slots before $t$. The retained tail of at least $w+p$ tokens is exactly what the next chunk's earliest token attends to. The per-request hold drops from one prefill chunk (~16k tokens) to the window plus whatever single chunk is currently in flight.

A second insertion path, decode islands, fires every $cp/4$ committed tokens during generation and inserts the request's prompt-plus-generation prefix into the tree, with tombstones starting where the decode-time evictor already freed. This makes generated text prefix-reusable while the request is still running: an agent that re-sends its growing transcript now hits cache on its own previous turn's output, not just its prompt.

One design detail is worth recording. The natural cadence trigger (fire when the sequence length is an exact multiple of $cp/4$) silently never fires under speculative decoding, because sequence length advances by variable accept lengths (ours averages ~8.5 tokens per step; see the accept-length discussion in our benchmark article) and essentially never lands on an exact multiple. Our trigger fires on accumulated progress since the last island instead. If you combine decode-region caching with speculative decoding, check this; modulo-based triggers and variable step sizes do not mix.

Why eager freeing is safe under overlap scheduling

The stash for chunk $N$ runs while chunk $N$'s forward pass may still be executing on the GPU. Freeing KV out from under an in-flight kernel sounds alarming, but the reasoning closes: the freed region ends at least $w+p$ tokens before the retained tail, and the previous stash already guaranteed the in-flight chunk's window was retained. Any freed slot that gets re-allocated is only written by kernels enqueued later on the same compute stream, so stream ordering makes read-before-overwrite an invariant rather than a race. And a write-write collision on a re-allocated slot can only clobber SWA KV that is beyond every future token's window: dead by construction. The allocator additionally makes double-frees structural no-ops.

Proving it byte-exact, and a gate-design lesson

Because the mechanism only ever frees KV that no future token attends to, it should be semantically invisible. That is a testable claim. Our end-to-end gate boots the verbatim production stack (FP8 W8A8 weights, FlashAttention-3 at head-dim 512, fp8-e4m3 KV, page size 64, block speculative decoding, piecewise CUDA graphs) twice on one H100, differing only in the island switch, and demands byte-equality at temperature 0 across cold serves, cache-hit re-serves, and extensions that reach 9,000+ tokens deep into decode-region islands (the strongest corruption probe available, since their last-window KV is island data). All classes passed byte-exact, alongside allocator-level conservation tests: after inserting and evicting everything, both pools return exactly to baseline: no leak, no double-free.

The gate's first revision, however, produced a negative result worth sharing. It compared cache-hit re-serves against cold references, and "failed" 3 of 4 prompts with islands off. A control run showed the same per-prompt flip pattern in the unmodified engine: a cache-hit serve recomputes only the suffix past the cached prefix, under different prefill kernel tiling and different speculative draft context, and that flips borderline temperature-0 argmaxes. This is a property of radix-cached serving in general, not of islands. If you build temperature-0 equality gates on a cached engine: compare cache-hit against cache-hit, never cache-hit against cold.

Re-deriving the pool split

With the hold eliminated, the SWA pool floor at $N$ running requests is one un-stashed in-flight chunk plus $N(w+p)$: about 68k tokens at our concurrency cap, versus the 237k the old ratio provisioned. That opens the capacity trade back up:

All three satisfy the floor with margin, and the benchmarks below confirm no admission bind even at $r=0.10$, validating that the hold, not the window, was the binding term all along.

Results: the latency-bound regime

All benchmarks run a seeded 32k-in/16k-out random workload (identical token totals across arms) against a full production boot on one H100-80GB. At client concurrency 24, where admission never binds, the mechanism's raw effect is visible in isolation:

The +14.7% at identical configuration comes from per-request speed: less SWA pool pressure means the scheduler stops throttling chunk admission mid-request. And the B → C step is the payoff of the re-derivation: nothing measurable changes while total KV capacity grows by a third.

Results: the pool-bound regime

The latency-bound arms conceal the admission story, because the client cap of 24 is below where the pool binds. Raising the offered load to 96 requests at the server's own concurrency cap of 48 exposes it:

This is the headline: +42.5% output throughput (3,995 → 5,695 tok/s) and −37% mean E2E latency (73.4 → 46.5 s), with zero retractions in both arms. The speculative accept length also rises from 8.22 to 9.13 as a side effect: healthier admission yields fuller verify batches. Notably, the historical benchmark that originally chose $r=0.32$ measured only 15–18 running requests at that ratio; islands move the leaner 0.17 partition to full-cap admission while carrying 32% more KV.

Honest caveats: each arm is one machine (H100 unit variance is small against 14–42% deltas, but not zero; read the B/C gap above as a tie), and the correctness gate ran with a reduced chunk size so both island types fire within gate-sized requests; production's larger chunks fire islands less often through the same code path.

Deployment notes

• Rollback is coupled. The kill switch restores the old freeing behavior, but the new 0.17/980k partition assumes eager eviction: disabling islands requires reverting the ratio too, or admission re-binds at ~11 running requests. Kill switches and capacity knobs that move together should be documented together.
• Scope it to the cache that supports it. Our staging tier uses a different (unified, hierarchical) cache class that also reports SWA support but has no island path; the guard is an exact type check, not a capability flag. Duck-typing a memory-safety feature is how you corrupt KV in the one configuration you didn't test.
• Cadence under speculative decoding needs the accumulated-progress trigger described above, or decode islands never fire.

FAQ

Does this change model outputs? No. The mechanism only frees SWA KV beyond every future token's attention window. We verified byte-equality at temperature 0 on the full production stack, including requests that extend 9,000+ tokens into island-cached decode regions.

Why keep full-attention KV in tombstones at all? Prefix matching walks the full-attention KV. Freeing it would destroy radix cache hits; keeping it costs 9.5 KB/token against the 95 KB/token the SWA slots cost. The asymmetry is the whole design.

Does this apply to non-hybrid models? No. A model that is all full attention has no out-of-window KV to free. It applies to any hybrid SWA architecture served with radix caching (the Gemma family, and the growing set of models mixing local and global attention).

Is this the same thing as paged attention? No: paged attention removes allocation waste (reserving memory never written). Islands remove retention waste: KV that was legitimately written but can never be read again, yet is pinned by the prefix cache.

What would I check before enabling something like this? Three things: that your temperature-0 equality gate compares cache-hit to cache-hit; that eviction cadence survives your decoding mode (speculative decoding breaks modulo-based triggers); and that any memory-saver / snapshot path in your stack has been exercised against island-populated trees.

Free tools from Netra

All of these run in your browser. No signup, and your data is not uploaded to us.

• LLM VRAM Calculator: size the GPU for any model, precision and context length.
• Fine-Tuning Memory Calculator: the training-time memory budget.
• Token Counter: measure prompts under the GPT, Claude and Llama tokenizers.
• Free OCR: turn scans and PDFs into LLM-ready text.
• Web to Markdown: strip a page to clean Markdown before it reaches a prompt.
• Limbus: local-first image segmentation.
• sam3.c: SAM3 inference in pure C.

References

• SGLang RadixAttention: the radix-tree prefix cache the mechanism restructures.
• Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention: the paged KV substrate.
• How Paged Attention and Continuous Batching Actually Work: our explainer on the underlying machinery.
• Up to 4x faster than vLLM: the benchmark methodology these serving numbers follow.