How Much VRAM Do You Need to Run an LLM?

This is the first question anyone asks before self hosting a model: how much VRAM do you need to run an LLM? Most answers give you a rule of thumb and stop there. The rule of thumb is fine until the day it is not, and the day it is not is usually the day someone sends a long prompt and your server returns an out-of-memory error.

So this guide derives the number instead of guessing it. There are only four things in GPU memory at inference time, and three of them are easy. The fourth, the KV cache, is the one that decides whether your deployment survives contact with real traffic. If you would rather skip the arithmetic, the LLM VRAM Calculator runs every formula below for you.

TL;DR

Weights cost bytes per parameter times parameter count. At 4-bit, a 70B model is about 33 GiB. That part is fixed.
The KV cache costs 320 KiB per token for a 70B model like Llama 3.1, and it grows linearly with context length and with the number of concurrent requests.
At 128K context that is 40 GiB of KV cache, which is larger than the quantized weights. Context, not model size, is what usually kills you.
Grouped-query attention is why this is survivable at all. The same 70B model with classic multi-head attention would need 320 GiB of KV cache at 128K.
A useful way to size a deployment is to ask how many tokens fit, not how many users: an 80 GB card running a 4-bit 70B has room for roughly 119,000 tokens of KV cache in total, however you divide them up.

The memory budget

Everything resident on the GPU during inference falls into one of four buckets:

M_{total} = weights P \cdot b_{w} + grows with context M_{KV} + activations M_{act} + framework M_{fw}

Here $P$ is the parameter count and $b_{w}$ the bytes spent on each parameter. The weights term is static: once the model is loaded it never changes size. The KV cache term is dynamic and depends on what your users are doing right now. Activations are transient buffers, and the framework term is the CUDA context plus allocator fragmentation.

One note on units before the numbers, because it is a genuine source of confusion. GPU vendors advertise in gigabytes ( $1 0^{9}$ bytes) while allocators report gibibytes ( $2^{30}$ bytes). An "80 GB" H100 shows up as roughly 79 GiB of usable memory. That 1% matters when you are trying to fit a model into the last gigabyte, so the figures below are all in GiB.

The weights: the easy part

Weight memory is just the parameter count multiplied by the size of each parameter:

M_{weights} = P \cdot b_{w}

The only decision is $b_{w}$ , which is set by your precision:

Precision	Bytes per parameter	8B model	70B model
FP32	4	29.8 GiB	260.8 GiB
FP16 / BF16	2	14.9 GiB	130.4 GiB
FP8	1	7.5 GiB	65.2 GiB
4-bit (nominal)	0.5	3.7 GiB	32.6 GiB

Treat the 4-bit row as a floor rather than a promise. A 4-bit quantization does not store only 4 bits per weight: it stores 4-bit integers plus a scale (and often a zero point) for every block of weights. With the common block size of 32, a scheme like Q4_K_M lands nearer 4.5 effective bits, so a real 70B checkpoint is closer to 37 to 40 GiB on disk than to the 32.6 GiB the arithmetic suggests. This is why a downloaded GGUF is always a little bigger than you expected.

The KV cache: the part that actually bites

During generation the model attends to every previous token. Recomputing the key and value projections for the whole prefix on every new token would be quadratic and hopeless, so they are computed once and cached. That cache is the KV cache, and it is the term that grows.

For every token, in every layer, the model stores one key vector and one value vector. That gives:

M_{KV} = 2 \cdot L \cdot n_{kv} \cdot d_{head} \cdot b_{kv} \cdot s \cdot B

The 2 is for K and V. $L$ is the number of layers, $n_{kv}$ the number of key/value heads, $d_{head}$ the dimension of each head, $b_{kv}$ the bytes per element, $s$ the sequence length in tokens and $B$ the number of sequences in flight.

Notice what is not in that formula: the parameter count. The KV cache does not care how many parameters your model has. It cares about its shape. Take Llama 3.1 70B, which has 80 layers, 8 key/value heads and a head dimension of 128, served in FP16:

\frac{M _{KV}}{token} = 2 \cdot 80 \cdot 8 \cdot 128 \cdot 2 bytes = 327, 680 bytes = 320 KiB

320 KiB per token sounds harmless. Multiply it by a 128K context and it is 40 GiB, which is more memory than the 4-bit weights of the very same model. Multiply it again by eight concurrent users and no single GPU on the market will hold it.

This is also the clearest way to see what grouped-query attention bought us. In classic multi-head attention every query head has its own KV head, so $n_{kv} = 64$ for this model instead of 8. Every term in the formula is unchanged except that one, so the cache is exactly 8 times larger: 2.5 MiB per token, and 320 GiB at 128K context. GQA is not a minor optimization. It is the reason long-context serving is possible on hardware you can actually buy.

KV cache against context length, computed from the formula above. Grouped-query attention (8 KV heads) keeps a 70B model at 40 GiB even at 128K tokens. The same model with multi-head attention (64 KV heads) would need 320 GiB, or four 80 GB cards for the cache alone.

Putting the budget together

Now the whole picture. Take that 70B at 4-bit, add roughly 2 GiB for the CUDA context and allocator, and vary only the context length:

The same model on the same card at three context lengths. Weights are fixed at 32.6 GiB. The KV cache goes from 1.2 GiB to 40 GiB, and at 128K the total reaches roughly 75 GiB, which is uncomfortably close to what an 80 GB card can actually give you.

The shape of that chart is the whole lesson. Sizing a GPU by model weights alone tells you whether the model loads. It tells you nothing about whether it serves.

How many users actually fit?

Serving engines such as vLLM do not think in users. They allocate the weights, reserve a slice of memory for the framework, and hand whatever is left to the KV cache as a fixed pool of blocks. So the honest capacity question is not "how many users" but "how many tokens", and you can solve the budget for it directly:

N_{tokens} = \frac{u \cdot M _{GPU} - P \cdot b _{w} - M _{fw}}{2 \cdot L \cdot n _{kv} \cdot d _{head} \cdot b _{kv}}

where $u$ is the fraction of the card you let the engine use (vLLM's gpu_memory_utilization, typically 0.90). For the 4-bit 70B on a single 80 GB card:

N_{tokens} = \frac{0.90 \cdot 79 - 32.6 - 2}{320 KiB} \approx 119, 000 tokens

That single number answers a lot of questions at once. It is about 14 concurrent users at 8K context each, or 3 users at 32K, or one user at 119K and nobody else. Concurrency and context length are the same resource, spent differently, and that is the tradeoff you are actually making when you advertise a long context window.

Two levers widen it. Quantizing the KV cache to FP8 halves $b_{kv}$ and therefore doubles $N_{tokens}$ , usually for very little quality loss, because attention is tolerant of noise in the cache. Quantizing the weights harder frees space in the numerator, but with diminishing returns: below 4 bits you pay in quality faster than you gain in capacity.

Since $N_{tokens}$ is denominated in tokens, it is worth knowing what your prompts really cost. A 4,000 word document is not 4,000 tokens. Our Token Counter gives you the true figure for GPT, Claude and Llama tokenizers, and if your inputs start life as PDFs or web pages, Free OCR and Web to Markdown will turn them into text you can measure.

Activations and framework overhead

The last two terms are small but not zero, and forgetting them is how you end up 2 GiB short.

Activations are the intermediate tensors of a forward pass. During decoding you process one token per sequence, so they are negligible. During prefill you process the entire prompt at once, and the activation buffers scale with the number of tokens you push through together. This is why a server that has been happily streaming replies falls over the moment somebody pastes in a long document: the prefill spike, not the steady state, is what exceeds the budget. Engines cap this with chunked prefill for exactly that reason.

Framework overhead is the CUDA context, the kernels, the allocator's fragmentation and the engine's own bookkeeping. Budget 1 to 2 GiB and stop worrying about it.

A recipe you can follow

Compute the weights: $P \cdot b_{w}$ . Add 10% if you are using a 4-bit GGUF, for the scales.
Look up $L$ , $n_{kv}$ and $d_{head}$ in the model's config.json. They are num_hidden_layers, num_key_value_heads and hidden_size / num_attention_heads.
Compute the KV cost per token, then multiply by the longest context you intend to advertise, times the concurrency you intend to support.
Add 2 GiB. Compare against the card's real capacity in GiB, not the marketing number in GB.
If it does not fit: quantize the KV cache before you quantize the weights further, and shorten the advertised context before you buy another GPU.

The LLM VRAM Calculator does all five steps and lets you drag the context length to watch the KV term move.

Training is a different question

Everything above is inference. Fine-tuning adds gradients (one per parameter) and optimizer state, and Adam keeps two moments per parameter in FP32. Full fine-tuning therefore needs roughly 16 bytes per parameter before activations, which is why a 70B model that infers comfortably on one card needs a small cluster to train, and why LoRA (which freezes the base weights and trains a small adapter) is the default for most teams. Our Fine-Tuning Memory Calculator covers that case properly.

Sizing the hardware is only half the job. Making the model fast and cheap once it fits is the other half, and it is where most of the money hides. Netra helps teams fine tune, accelerate and deploy models on their own infrastructure, so you get low latency without renting more GPU than the arithmetic says you need.

FAQ

How much VRAM do I need to run a 7B or 8B model? About 15 GiB at FP16 or 4 GiB at 4-bit for the weights. An 8 GiB card runs it quantized, but remember the KV cache: an 8B model costs 128 KiB per token, so a 32K context adds another 4 GiB and that "8 GB is enough" advice stops being true.

Can I run a 70B model on one GPU? Yes, at 4-bit, on an 80 GB card, with about 119,000 tokens of KV cache to share between your users. At FP16 the weights alone are 130 GiB and you need at least two cards.

Why does my server OOM only sometimes? Almost always the KV cache or the prefill spike. Both scale with what your users send, not with what you deployed, so the failure arrives with the traffic rather than at startup.

Does quantizing the KV cache hurt quality? Much less than quantizing the weights by the same amount. FP8 KV cache is close to free in practice and doubles your token capacity, so it is usually the first lever to pull.

Do MoE models change the maths? Yes, in one place. All experts must be resident in memory, so $P$ in the weights term is the total parameter count, not the active one. The active count governs speed; the total count governs whether it fits.

Free tools from Netra

All of these run in your browser, with no signup and no upload of your data to us.

LLM VRAM Calculator: runs the formulas on this page for any model, precision, context length and batch size.
Fine-Tuning Memory Calculator: the training-time equivalent, including optimizer states, gradients and LoRA.
Token Counter: measure how many tokens a prompt really costs, which is what sizes your KV cache.
Free OCR: turn scans and PDFs into LLM-ready text.
Web to Markdown: convert any page into clean Markdown for a prompt or a RAG index.
Limbus: local-first image segmentation.
sam3.c: SAM3 inference in pure C.

References

Ainslie et al., GQA: Training Generalized Multi-Query Transformer Models, the paper that introduced grouped-query attention.
Kwon et al., Efficient Memory Management for LLM Serving with PagedAttention, the vLLM paper, on why the KV cache is managed as a block pool.
vLLM documentation, gpu_memory_utilization and KV cache sizing.
Modal, How much VRAM do I need for LLM inference?

Blog / Guides