What are the GPU requirements for fine-tuning an LLM?

Full fine-tuning with AdamW needs roughly 12 to 16 bytes per parameter before activations, which works out to about 12 to 16 times the parameter count in GB: on the order of 100 GB for an 8B model. LoRA trains under 1% of the parameters and cuts gradient and optimizer memory to a fraction, and QLoRA also stores the frozen base in 4-bit, letting 7B to 13B models fine-tune on a single 24 GB GPU.

Can I fine-tune an LLM on a single GPU?

Yes, with parameter-efficient methods. QLoRA stores the frozen base model in 4-bit and trains small 16-bit adapters, which typically fits a 7B to 13B model on a single 24 GB consumer GPU. Full fine-tuning of the same models usually requires multiple data-center GPUs.

What consumes GPU memory during fine-tuning?

Five things: the model weights, the gradients, the optimizer states (the largest share with AdamW), the activations saved for the backward pass, and framework overhead. This calculator breaks each one down for full fine-tuning, LoRA, and QLoRA.

Tools / Training Memory Calculator

Fine-tuning GPU memory calculator

GitHub

Estimate the peak GPU memory needed to fine-tune an LLM. Paste a Hugging Face model and this reads its architecture and parameter count, then breaks memory down across model weights, gradients, optimizer states, and activations for full fine-tuning, LoRA, or QLoRA.

Hugging Face model

Method Precision Optimizer Micro-batch size Sequence length Gradient checkpointing LoRA rank

How this tool works

You give it a Hugging Face model, and it does two lookups. First it calls the Hugging Face models API for that repo and reads safetensors.total, the exact parameter count published in the model's weight index. Then it fetches the repo's config.json and reads the architecture: hidden size, number of layers, attention heads, key/value heads (for grouped-query attention), and the feed-forward intermediate size. Everything else you choose in the form. Nothing is uploaded and no weights are downloaded; it only reads a few kilobytes of metadata. If a repo does not publish a parameter count, the tool falls back to estimating it from the architecture and vocabulary size.

GPU requirements for fine-tuning: quick rules of thumb

Before running the numbers, it helps to know the ballpark. Full fine-tuning with AdamW costs roughly 12 to 16 bytes per parameter before activations, which is about 12 to 16 times the parameter count in GB: on the order of 100 GB for an 8B model, so multiple data-center GPUs. LoRA trains under 1% of the parameters and cuts gradient and optimizer memory to a fraction while the frozen 16-bit base still sits in memory. QLoRA also stores the frozen base in 4-bit, which is what typically fits a 7B to 13B model on a single 24 GB consumer GPU. The calculator above turns these rules of thumb into exact numbers for your model, method, batch size, and sequence length.

Why fine-tuning needs so much memory

Running a model for inference only needs the weights in memory. Training needs far more, because every trainable parameter is accompanied by several other tensors of the same size. Peak training memory is the sum of five things: the model weights, the gradients, the optimizer states, the activations saved for the backward pass, and a slice of framework and allocator overhead. The calculator adds these up and applies a 10% headroom factor for the overhead that is hard to model precisely.

Model weights, gradients, and optimizer states

In mixed-precision training the weights are held in 16-bit (BF16 or FP16), so a parameter costs 2 bytes. Each trainable parameter also needs a gradient of the same precision, another 2 bytes. The optimizer is where it adds up: AdamW keeps two running averages per parameter (the first and second moments) stored in FP32 at 4 bytes each, or 8 bytes per parameter. If you enable an FP32 master copy of the weights for stable mixed-precision updates, that is another 4 bytes. So a fully-trained parameter with AdamW costs roughly 2 (weight) + 2 (gradient) + 8 (moments) = 12 bytes, or 16 bytes with an FP32 master copy. This is why full fine-tuning of an 8B model needs on the order of 100 GB before activations. 8-bit AdamW stores the moments in 8-bit and cuts the optimizer share to about 2 bytes per parameter; paged AdamW keeps the 8-byte moments but can offload them to CPU RAM under pressure.

LoRA and QLoRA

LoRA freezes the full base model and inserts small low-rank adapter matrices into the attention and MLP projections. Only those adapters are trainable, so gradients and optimizer states are computed on a tiny fraction of the parameters (often well under 1%) while the frozen base still sits in memory at 16-bit. The adapter parameter count is rank × (d_in + d_out) summed over every targeted projection (q, k, v, o, gate, up, down) across every layer, using the model's real dimensions including grouped-query attention. QLoRA goes further and stores the frozen base in 4-bit NF4 with double quantization, roughly 0.52 bytes per parameter instead of 2, which is what lets large models fit on a single consumer GPU. The adapters themselves stay in 16-bit.

Activations and gradient checkpointing

During the forward pass the framework saves intermediate tensors so it can compute gradients on the way back. This activation memory scales with batch size, sequence length, hidden size, and depth, and it can dominate at long sequence lengths. The calculator uses the standard transformer estimate of s · b · h · (34 + 5 · a · s / h) per layer without checkpointing. Gradient checkpointing trades compute for memory by recomputing activations during the backward pass instead of storing them: selective checkpointing keeps the estimate near 34 · s · b · h · L, and full checkpointing drops it to roughly 2 · s · b · h · L. That is why increasing sequence length or batch size can blow up memory, and why checkpointing is the first lever to pull.

Accuracy and limits

These are planning estimates, not a guarantee. Real usage depends on the framework (PyTorch, DeepSpeed, FSDP), the attention kernel (FlashAttention and fused kernels store far less than the naive estimate), CPU offloading, and memory fragmentation. The activation formula assumes a standard dense transformer; mixture-of-experts models, unusual attention layouts, and very long context change the picture. Treat the number as a lower-to-middle bound and keep headroom on the actual GPU.

Source

See the implementation on GitHub: NetraRuntime/training-memory-calculator.

Learn more

New to this? Read how much GPU memory it takes to fine-tune an LLM and what LoRA and QLoRA are.