Up to 4x faster than vLLM: benchmarking Netra on real long-context workloads

TL;DR

• Up to 4.4x more output per second than vLLM on a 32K-in / 16K-out workload, same H200, same weights.
• 5.5x faster token streaming, and ~3.4x faster to finish a full answer at production load.
• Beats even advanced open-source speed-up techniques by ~2x once real traffic arrives.
• On the speed-vs-latency trade-off, only Netra is consistently worth choosing at scale.

The setup: one GPU, one model, five engines

We wanted a fair fight, so everything was held identical except the engine itself.

• The model. Qwen3.6-35B-A3B, a 35-billion-parameter model. The same compressed weights were loaded into every engine.
• The hardware. A single NVIDIA H200 GPU. No multi-GPU tricks.
• The task. A large input and a long answer: roughly a 50-page document going in, and a multi-page, reasoning-length answer coming out (32,000 tokens in, 16,000 out). This is the shape of modern agentic, document-analysis, and coding workloads, and it's the hardest case for a serving engine.
• The load. Everything from 1 user at a time up to 512 concurrent users, so you can see how each engine holds up as traffic grows.

The contenders:

And a quick vocabulary, if you don't live in this world:

A "token" is roughly three-quarters of a word.

Far more output, at every level of load

Here's the headline. As users pile on, Netra pulls away from the entire field. Netra produced 4.4x more output per second than vLLM at a single request, and still 3x more at 256 concurrent users. The numbers behind the curve (words per second; higher is better):

Netra wins every single load level against vLLM, by 3 to 4x. Just as important, it keeps climbing as users grow (987 up to 7,111 words/sec) while the others flatten out early. That gap, and why it widens, is the rest of this post.

Faster to start, faster to stream, faster to finish

Raw throughput is one dimension. Production teams also care about latency: how long until the first word, how smoothly the answer streams, how long the whole thing takes. Netra leads on all of them at once. In human terms, at a busy 256-users-at-once load:

• Words stream 5.5x faster. Netra emits a new token every 7.4 ms; vLLM takes 41 ms. The answer visibly "types" five times faster.
• The whole answer finishes ~3.4x sooner, about 6 minutes for Netra versus nearly 20 minutes for vLLM on the same work.
• The first word arrives sooner, too, at the loads where it matters.

Faster to start, faster to stream, faster to finish.

The full throughput picture: total work and prompt-reading speed

"Output speed" is what your users watch stream, but a serving engine also has to read the prompt (prefill), and on a 32K-token input that's a lot of reading. Two more views complete the picture.

• Total throughput (every token processed, input plus output) peaks at 22,400 tokens/sec for Netra, versus ~7,100 for vLLM and ~11,000 for the best open-source speed-up. This is the number that maps most directly to GPU cost: more total tokens per second per GPU means fewer GPUs for the same work.
• Prefill speed (how fast the engine digests the incoming prompt) peaks at 14,900 tokens/sec for Netra, 3x vLLM. On long-context workloads, fast prefill is what keeps time-to-first-token low when many big prompts arrive at once.

Both lenses tell the same story as output speed: Netra leads by roughly 2 to 4x over the open-source field, and 3 to 4x over vLLM, with the margin widening under load.

It even beats the best open-source speed-up tricks

It's easy to look good against a default config, so we also put Netra up against SGLang + DFlash, an advanced open-source speed-up technique well beyond an out-of-the-box setup and the strongest non-proprietary competitor we could find.

The interesting part is the shape of the result.

• At a handful of users, the two are neck-and-neck. When the GPU is barely busy, both are excellent and there's little to separate them.
• As real traffic arrives, Netra pulls ~2x ahead. From 32 users up, the advanced competitor stalls out around 3,500 words/sec, while Netra keeps climbing past 7,000.

The reason is memory. Netra fits roughly twice as many simultaneous conversations into the same GPU, so when traffic spikes the other engines run out of room and slow down while Netra keeps going. A clever decoding trick on its own isn't enough. It's the whole engine that scales.

The efficiency frontier: who's actually worth choosing

There's always a trade-off in serving: you can usually buy more throughput by tolerating more delay, and vice-versa. So the fairest comparison isn't a single number. It's asking, across every speed-vs-delay setting, which engine gives you the best deal.

That set of best-possible trade-offs is the efficiency frontier. An engine sits "on the frontier" when nothing else beats it on both throughput and delay at once. And if an engine is never on the frontier, you'd never rationally pick it, because something always beats it on both counts.

Three of the five engines (vLLM, default SGLang, and SGLang+EAGLE) never reach the frontier at all. For any latency target, something always beats them on both speed and delay.

Netra owns the frontier wherever real traffic lives. The advanced DFlash technique shares a small slice at very light load, where everything is fast anyway. The most practical way to read it is to ask: if you commit to a "words must stream at least this fast" guarantee, how much throughput can each engine deliver without breaking it?

Words per second deliverable while keeping the latency promise.

At any reasonable guarantee, Netra delivers about 2x the throughput of the next-best engine and 4 to 7x that of vLLM. That's the same traffic, at the same responsiveness, on a fraction of the GPUs.

So how does Netra do it?

Here's the honest version, and it's not the obvious one.

Speculative decoding (guessing several words ahead and checking them in one pass) is the trick that makes fast decoding possible, and several engines have it now, including the open-source ones we tested. At a single request, the good ones are genuinely close to Netra. So our advantage isn't that we guess better. It's that our advantage survives when real traffic shows up.

That's the catch with most speculative engines. The trick is expensive to run, and the cost grows with every concurrent user. Verifying all those guesses takes extra compute and extra memory, so as the batch fills up, other engines hit a wall: their throughput flattens (or even falls), and the wait for the first word balloons. You can see it in the charts, where the speculative competitors top out around 3,500 to 4,000 words/sec and their latency spikes while Netra keeps climbing past 7,000.

Netra is engineered specifically for that high-load regime. Two things make the difference.

1. It's far more memory-efficient. A GPU has only so much memory, and every active conversation takes up space. Netra packs about 2x more conversations into the same GPU without hurting answer quality, so when traffic spikes it still has room to keep everyone moving while other engines start queuing.
2. It schedules the work much better. Keeping a big batch busy and answering new arrivals quickly is a hard balancing act. Netra's scheduler keeps the GPU fully fed at high concurrency while still admitting new requests fast, which is why it holds high throughput and low time-to-first-token at the same time, exactly where other speculative engines are forced to trade one for the other.

Put simply: everyone can go fast with one user. Netra stays fast with hundreds.

How to trust these numbers

Benchmarks are easy to game, so here's exactly how we kept this one honest.

• Same model, same GPU, same benchmark. Every engine ran identical FP8 weights on the same single H200, driven by the same open-source, industry-standard serving benchmark. The only variable was the runtime.
• One engine at a time. No two engines ever shared the GPU, so no result is inflated or dragged down by a noisy neighbor.
• 200 operating points, not one. We swept two input sizes, three answer lengths, and ten concurrency levels. That's the full picture, not a single flattering data point.
• We tested against the strongest competition, not the weakest. That includes speculative-enabled SGLang and an advanced open-source speed-up technique, not just out-of-the-box defaults that are easy to beat.
• We show where we don't win. At very short prompts under saturating concurrency, a batching-optimized engine like vLLM can edge ahead, because speculation is a tax when the GPU is already full. Netra is built for long context, and we say so plainly.

Want to verify it on your own workload? That's the real test. See the offer below.

What this means for you

If your product looks like modern AI, with long prompts, reasoning-length answers, and many concurrent users, the engine underneath decides how much GPU you rent and how fast your users feel the product. On that workload, Netra delivers:

That's the difference between one GPU and several for the same traffic, and a noticeably faster product for your users.

Want to run your own workload on Netra? Get in touch and we'll benchmark your exact model and traffic, on your numbers.

Benchmark environment

Every engine was benchmarked sequentially, one at a time, each on its own dedicated H200 container on Modal. No two engines ever ran on the same GPU at the same time, so no result is inflated or dragged down by a noisy neighbor. Each engine was given the same dataset and the same request schedule, and each concurrency level ran to a steady state before measurement.

Throughput is output tokens per second; latency figures (time to first token, time per output token, end-to-end) are means. "Words" is a reader-friendly stand-in for tokens (about 0.75 words each).

Learn more

• Netra Runtime vs vLLM: an at-a-glance comparison
• LLM inference latency and throughput explained
• What is LLM inference?