Learn / What is LoRA (and QLoRA)?
What is LoRA (and QLoRA)?
LoRA (Low-Rank Adaptation) is a fine-tuning method that freezes a model's original weights and trains small low-rank adapter matrices instead. This cuts the number of trainable parameters — by up to 10,000x on GPT-3 175B in the original paper — and the GPU memory needed, while matching full fine-tuning quality on many tasks. QLoRA goes further by 4-bit-quantizing the frozen base, letting you fine-tune very large models on a single GPU.
How LoRA works
Instead of updating a weight matrix directly, LoRA keeps the original frozen and learns a small update expressed as the product of two low-rank matrices. Only those two matrices are trained, so the number of trainable parameters — and therefore the gradients and optimizer states — is a tiny fraction of the full model. At inference the adapter can be merged back into the weights, so there is no added latency. In the original paper, LoRA reduced trainable parameters on GPT-3 175B by up to 10,000x and GPU memory by about 3x, while performing on par with full fine-tuning.
How QLoRA works
QLoRA combines LoRA with aggressive quantization: it stores the frozen base model in 4-bit and trains the LoRA adapters on top, backpropagating gradients through the quantized weights. Because the base — the largest memory component — is now 4-bit, QLoRA made it possible to fine-tune a 65B-parameter model on a single 48GB GPU while preserving full 16-bit fine-tuning quality.
When to use which
Use LoRA when the base model already fits comfortably in memory and you want fast, cheap adaptation with full-precision weights. Reach for QLoRA when the base is too large to fit otherwise, or when you want to fine-tune on a single GPU. Both produce a small adapter you can deploy on top of the base model.
| Method | Base weights | Trained | Memory |
|---|---|---|---|
| Full fine-tuning | Trained, full precision | All parameters | Highest |
| LoRA | Frozen, 16-bit | Small adapters | Lower |
| QLoRA | Frozen, 4-bit | Small adapters | Lowest |
Try it yourself
Free tool
Fine-tuning GPU memory calculator
Compare full, LoRA, and QLoRA memory for a model and see the weights, gradients, optimizer, and activation breakdown.
Frequently asked questions
What is the difference between LoRA and QLoRA?
LoRA freezes the base model at its normal precision and trains small low-rank adapters. QLoRA does the same but first quantizes the frozen base to 4-bit, which shrinks the largest memory component and lets you fine-tune much larger models on a single GPU.
Does LoRA reduce accuracy?
On many tasks it matches full fine-tuning. The original LoRA paper reported quality on par with or better than full fine-tuning on several models despite training far fewer parameters.
What is a LoRA rank?
The rank is the size of the low-rank adapter matrices. A higher rank gives the adapter more capacity but uses more memory; a lower rank is smaller and cheaper. It is the main knob for trading quality against cost in LoRA.
References
- Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021). LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106.09685.
- Dettmers, T., Pagnoni, A., Holtzman, A., & Zettlemoyer, L. (2023). QLoRA: Efficient Finetuning of Quantized LLMs. arXiv:2305.14314.