Learn / Inference latency & throughput
LLM inference latency and throughput
Latency and throughput are the two core measures of LLM inference performance. Throughput is how many tokens the system produces per second; latency is how long a user waits — broken down into time-to-first-token, time-per-output-token, and end-to-end completion time. Concurrency, the number of users hitting the system at once, determines how these trade off in production.
The key metrics
Five metrics describe almost everything about LLM serving performance.
| Metric | In plain English | Better when |
|---|---|---|
| Throughput | Words (tokens) per second produced | higher |
| Time to first token (TTFT) | How long before the first word appears | lower |
| Time per output token | How fast words stream after the first | lower |
| End-to-end latency | Total time to finish the whole answer | lower |
| Concurrency | How many users hit the system at once | the load level |
A token is roughly three-quarters of a word.
Latency vs throughput trade-off
Latency and throughput pull against each other. Batching more requests together raises throughput but can increase the latency any single user feels, and the tension grows as concurrency rises. A serving engine that looks fast for one user can fall behind under production load, so the metrics that matter are the ones measured while many users are active at once.
How to reduce latency and raise throughput
Common techniques include continuous batching, careful KV-cache management, quantization, efficient request scheduling, choosing an appropriately sized model, and using a serving engine optimized for the workload. The right combination depends on model size, context length, and how many users you serve concurrently.
How Netra performs
In a published benchmark at 256 concurrent users, Netra Runtime streamed tokens about 5.5x faster than vLLM (a new token every 7.4 ms versus 41 ms) and finished a full long answer roughly 3.4x sooner — about 6 minutes versus nearly 20. Its throughput kept climbing as load grew, from 987 to 7,111 words per second, while other engines flattened out early. See the Netra vs vLLM comparison or the full benchmark.
Frequently asked questions
What is time-to-first-token (TTFT)?
Time-to-first-token is the delay between sending a prompt and receiving the first generated token. It is the main driver of perceived responsiveness in chatbots and agents.
What is the difference between latency and throughput?
Latency is how long one request takes; throughput is how many tokens the system produces per second across all requests. A system can have high throughput but still feel slow if per-request latency is high under load.
How do you reduce LLM inference latency?
Common methods include continuous batching, KV-cache optimization, quantization, efficient request scheduling, choosing an appropriately sized model, and using a high-performance serving engine.