Learn / How much GPU memory to fine-tune an LLM?
How much GPU memory do you need to fine-tune an LLM?
Fine-tuning an LLM needs far more memory than running it, because on top of the weights you must store gradients, optimizer states, and activations. A rule of thumb for full fine-tuning in mixed precision is about 12-16 bytes per parameter, so a 7B model can need well over 100 GB. LoRA and QLoRA cut this sharply by training only small adapter matrices on top of a frozen base.
What uses the memory
Fine-tuning memory has four main parts beyond the base weights: gradients (one value per trainable parameter), optimizer states (Adam keeps two — momentum and variance — so about 8 bytes per parameter in fp32), activations saved for the backward pass (these grow with batch size, sequence length, hidden size, and layer count), and a slice of overhead. Add them up and the training footprint is a multiple of the model itself.
The rule of thumb
For full fine-tuning, the weights, gradients, and optimizer states alone come to roughly 12 bytes per parameter in bf16 (2 for weights + 2 for gradients + 8 for Adam), or about 16 bytes when keeping an fp32 master copy. That puts a 7B model near 84-112 GB before activations — which is why full fine-tuning usually needs multiple high-memory GPUs.
How LoRA and QLoRA cut it
Instead of training every parameter, LoRA freezes the base model and trains small low-rank adapters, so gradients and optimizer states are only needed for a tiny fraction of the parameters. QLoRA goes further by 4-bit-quantizing the frozen base, shrinking the largest remaining piece. Together they can bring a job that needed a multi-GPU node down to a single GPU.
| Method | What's trainable | Relative memory |
|---|---|---|
| Full fine-tuning | All parameters | Highest (~12-16 bytes/param) |
| LoRA | Small adapters; base frozen (16-bit) | Much lower |
| QLoRA | Small adapters; base frozen (4-bit) | Lowest |
Try it yourself
Free tool
Fine-tuning GPU memory calculator
Estimate GPU memory to fine-tune a model with full, LoRA, or QLoRA, broken down by weights, gradients, optimizer, and activations.
Frequently asked questions
Why does fine-tuning need more memory than inference?
Inference only needs the weights and the KV cache. Fine-tuning also stores gradients for every trainable parameter, optimizer states (two per parameter for Adam), and activations for the backward pass, which together can add up to several times the model's size.
Can I fine-tune a large model on a single GPU?
Often yes, using QLoRA. QLoRA quantizes the frozen base model to 4-bit and trains only small LoRA adapters, which in the original paper made it possible to fine-tune a 65B model on a single 48GB GPU while preserving 16-bit fine-tuning quality.
What is the memory difference between full fine-tuning and LoRA?
Full fine-tuning trains every parameter, so it needs gradients and optimizer states for all of them. LoRA trains only small adapter matrices, so gradients and optimizer state are computed on a tiny fraction of the parameters, cutting the extra memory dramatically.
References
- Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021). LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106.09685.
- Dettmers, T., Pagnoni, A., Holtzman, A., & Zettlemoyer, L. (2023). QLoRA: Efficient Finetuning of Quantized LLMs. arXiv:2305.14314.