Learn / What is continuous batching?
What is continuous batching?
Continuous batching is an LLM-serving technique where the engine continuously adds new requests to, and removes finished requests from, an in-flight batch — instead of waiting for a whole batch to finish before starting the next. It keeps the GPU busy and is one of the main reasons modern inference engines reach high throughput under concurrent load.
Static vs continuous batching
| Approach | How it batches | Effect |
|---|---|---|
| Static batching | Fixed group; waits for all to finish | GPU idles on the slowest request |
| Continuous batching | Adds and removes requests every step | Keeps the GPU full; higher throughput |
Why it raises throughput
In LLM serving, requests start and finish at different times because answers vary in length. With a fixed batch, the whole group waits for the slowest request, leaving GPU capacity unused. Continuous batching fills those gaps by admitting new work as soon as room opens up, so the hardware spends more time generating tokens and less time waiting.
How Netra helps
Netra Runtime is a high-performance inference platform built for production load. In a published benchmark its output throughput kept climbing as concurrency grew — from 987 words per second at one user to 7,111 at 256 — while other engines flattened out early. See inference latency and throughput or the Netra vs vLLM comparison.
Frequently asked questions
What is the difference between static and continuous batching?
Static batching groups a fixed set of requests and waits for all of them to finish before starting the next batch. Continuous batching adds new requests and removes finished ones every step, so the GPU does not sit idle waiting for the slowest request.
Why does continuous batching improve throughput?
Because requests finish at different times, a fixed batch leaves the GPU partly idle. Continuous batching keeps the batch full by swapping requests in and out, raising GPU utilization and total tokens produced per second.
Does batching increase latency?
Batching trades a little per-request latency for much higher throughput, and the trade-off grows with concurrency. A well-designed engine keeps latency low even as it batches aggressively.