Learn / What is a KV cache in LLM inference?
What is a KV cache in LLM inference?
A KV cache (key-value cache) stores the attention keys and values computed for previous tokens, so the model doesn't recompute them for every new token. It makes token generation much faster, but it grows with context length and consumes GPU memory — which is why managing it well is central to efficient long-context inference.
What it stores and why
When an LLM generates text, each new token attends to all previous tokens. Without a cache, the model would recompute the keys and values for the entire sequence at every step. The KV cache saves those tensors after they're first computed, so each new token only does the work for itself — turning repeated, quadratic work into something far cheaper.
The memory trade-off
The speed-up isn't free: the cache grows with every token in context and with every concurrent request. Long prompts and many simultaneous users can fill GPU memory quickly, so the way an engine allocates, reuses, and evicts KV-cache memory is one of the biggest factors in long-context and high-concurrency performance.
How Netra helps
Netra Runtime is built for demanding long-context serving. Its published benchmark used a 32K-token input and 16K-token output on a single NVIDIA H200 GPU, and Netra led the field on throughput and latency from a single user up to 512 concurrent users. See LLM inference for AI agents or inference latency and throughput.
Frequently asked questions
What does the KV cache store?
The KV cache stores the attention key and value tensors computed for the tokens already processed, so the model can attend to previous tokens without recomputing them for every new token.
Why does the KV cache use so much memory?
The cache grows with the number of tokens in context and with the number of concurrent requests, so long prompts and high concurrency can consume large amounts of GPU memory.
How does the KV cache affect long-context inference?
Long-context workloads create large KV caches, so how efficiently an engine manages that memory largely determines how well it handles long inputs and many concurrent users.