How much VRAM do I need to run an LLM?

As a rough planning guide at 4-bit quantization with a 4K context: a 7B model needs about 6 GB of VRAM, a 13B about 10 GB, a 32B about 24 GB, and a 70B about 45 GB. The exact number depends on quantization, context length, and KV cache settings; this calculator reads a model's GGUF metadata and computes it for your configuration.

What uses GPU memory when running an LLM?

Three things: the model weights (fixed by parameter count and quantization), the KV cache (grows with context length and can rival the weights at long context), and runtime overhead such as compute buffers and activation scratch space.

Does quantization reduce the VRAM an LLM needs?

Yes, substantially. A 7B model is roughly 14 GB at F16, about 7.5 GB at Q8, and about 4.4 GB at Q4_K_M. Quantizing the KV cache reduces the context-dependent part of memory the same way.

Tools / Model Memory Calculator

LLM VRAM calculator

GitHub

Calculate how much GPU memory an LLM needs to run. Paste a direct link to a .gguf file (Hugging Face resolve URLs work best) and this reads the model's metadata to compute the weight footprint, KV cache, and runtime overhead. Free, no signup.

GGUF file URL Or choose a local GGUF file

Context size (tokens)

KV cache quantization

Verbose log

Tip: many hosts (like Hugging Face) support the HTTP range requests needed to avoid downloading the whole file.

How this tool works

GGUF is the single-file format used by llama.cpp and Ollama to ship a quantized model. Every GGUF file begins with a metadata header that describes the architecture before any weights appear. This tool reads only that header: it issues HTTP range requests for the first slices of the file, parses the key-value metadata, and stops as soon as the required keys are found, never downloading the multi-gigabyte weight body. From the header it reads the block (layer) count, embedding length (hidden size), attention head count, key/value head count, parameter count, and split count. It then asks the server for the total file size with a HEAD request, falling back to a one-byte range request and the Content-Range response. For a quantized GGUF, the file size on disk is essentially the weight footprint in memory. You can also pick a local .gguf file instead of a URL; only its header is read from disk.

The two numbers that matter

Memory to run a model is dominated by two things: the weights and the KV cache. The weights are fixed once you pick a quantization: a 7B model at Q4_K_M is roughly 4.4 GB, the same model at Q8 about 7.5 GB, and at F16 about 14 GB. That is the file size the tool reads directly. The KV cache is the part people forget: it grows with context length and holds the attention keys and values for every token in the window, so a long prompt can cost as much memory as the model itself.

How much VRAM do you need to run an LLM?

As a rough planning guide at 4-bit quantization (Q4_K_M) with a 4K context: a 7B model needs about 6 GB of VRAM, a 13B about 10 GB, a 32B about 24 GB, and a 70B about 45 GB, covering weights, KV cache, and overhead. Those numbers shift with quantization and grow with context length, which is exactly what this calculator computes from the real GGUF metadata instead of a lookup table. For the full breakdown of where the memory goes, read our guide on how much GPU memory an LLM needs.

How the KV cache is estimated

For each token the model stores a key and a value vector in every layer. The size of those vectors depends on grouped-query attention: modern models share key/value heads across query heads, so the per-token cost is set by the KV heads, not the full attention width. The tool computes the cache as bytes_per_value × hidden_size × layers × context × (kv_heads / heads), where the last factor is the GQA ratio. The KV cache quantization selector sets bytes_per_value, which covers both the key and the value: FP16/BF16 is 4 bytes per token position, FP32 doubles that, and INT8, Q6, Q5, and Q4 shrink it at a small quality cost.

Runtime overhead

Beyond weights and KV cache, a runtime allocates compute buffers, activation scratch space, and bookkeeping structures. When the header includes the model's parameter count, the tool adds an overhead estimate of 0.02 GB per billion parameters + 0.15 GB, a regression fit over measured llama.cpp deployments. When the parameter count is missing from the metadata, the overhead line shows zero and you should leave extra headroom yourself.

Reading it in your browser

All of this runs client-side; nothing is sent to a server and no file is fully downloaded. That only works when the file host allows cross-origin range requests, which Hugging Face resolve URLs do; they are the supported path, and blob URLs are rewritten to resolve automatically. Other hosts often block cross-origin reads, in which case the tool shows a clear message rather than a wrong number. Sharded models (split into multiple .gguf parts) are detected from the header or the file name: for URLs the tool sums the size of every shard, reading metadata from the first shard even when you paste a later one, and for local files it multiplies the chosen shard's size by the shard count.

Accuracy and limits

The KV-cache math is deliberately simplified. Real runtimes differ in how they lay out and page the cache, some reserve extra buffers, and features like sliding-window attention change the totals. The overhead term is a fitted average, not a guarantee for your runtime and settings. Treat the result as a solid planning estimate for whether a model fits a given GPU, then leave headroom.

Source

See the implementation on GitHub: KolosalAI/model-memory-calculator.

Learn more

New to this? Read how much GPU memory an LLM needs.