Learn / How much GPU memory does an LLM need?
How much GPU memory (VRAM) does an LLM need?
The GPU memory (VRAM) an LLM needs is mostly two things: the model weights and the KV cache. Weights are the parameter count times the bytes per parameter, which quantization reduces; the KV cache grows with context length and concurrency. A useful estimate is weights + KV cache + roughly 10-20% overhead for activations and the runtime.
Model weights
Weight memory is simply the number of parameters times the bytes used to store each one. At 16-bit precision that's 2 bytes per parameter, so a 7-billion-parameter model needs about 14 GB just for weights. Lower precisions shrink this: 8-bit is about 1 byte per parameter (~7 GB) and 4-bit is roughly 0.5 bytes (~3.5-4 GB). This is why quantization is the first lever for fitting a model on a smaller GPU.
The KV cache
During generation the model stores the attention keys and values for every token so it doesn't recompute them — the KV cache. Its size is roughly 2 × layers × kv_dimension × context_tokens × bytes_per_value (the 2 is for keys and values). It grows linearly with context length and with the number of concurrent requests, so long-context or high-traffic serving can make the KV cache as large as the weights, or larger.
Overhead and the total
On top of weights and KV cache, the runtime needs memory for activations and framework buffers — commonly another 10-20%. So a working estimate is: total VRAM ≈ weights + KV cache + overhead. The components each scale with different things, which is what the calculator below makes concrete.
| Component | What drives it | How to reduce it |
|---|---|---|
| Weights | Parameter count × bytes per parameter | Quantize; use a smaller model |
| KV cache | Context length × concurrency | Shorter context; KV-cache quantization |
| Overhead | Activations and runtime buffers | Efficient serving engine |
Try it yourself
Free tool
LLM VRAM calculator
Paste a GGUF model URL and estimate its VRAM from the real metadata: weights plus KV cache at your chosen context length.
Frequently asked questions
How much VRAM do I need to run a 7B model?
At 16-bit precision a 7B model's weights are about 14 GB, plus a few GB for the KV cache and overhead. Quantized to 4-bit, the weights drop to roughly 3.5-4 GB, so a 4-bit 7B model can run in about 6-8 GB of VRAM depending on context length.
What uses the most GPU memory in LLM inference?
The model weights usually dominate, but at long context lengths or high concurrency the KV cache can grow large enough to exceed the weights. Activations and framework overhead add a smaller amount on top.
Does quantization reduce GPU memory?
Yes. The weight memory is the parameter count times the bytes per parameter, so moving from 16-bit (2 bytes) to 8-bit (1 byte) or 4-bit (about 0.5 bytes) cuts the weight memory roughly in half or to a quarter.