Learn / What is knowledge distillation?
What is knowledge distillation?
Knowledge distillation is a technique for transferring the knowledge of a large "teacher" model into a smaller, faster "student" model. Instead of training the student only on labeled data, it also learns from the teacher's output probabilities — the "soft targets" introduced by Hinton, Vinyals, and Dean (2015) — so a compact model can approach the accuracy of a much larger one at a fraction of the cost. It is sometimes called model distillation.
How it works
A large, accurate teacher model is trained first. A smaller student model is then trained to reproduce the teacher's behavior — most commonly by matching the teacher's full output probability distribution (its soft targets), not just the single correct label. A temperature parameter softens those probabilities so the student learns the teacher's relative confidence across all options, which carries far more information than a hard label. The idea of compressing a large model into a smaller one goes back to Buciluă, Caruana, and Niculescu-Mizil (2006); Hinton, Vinyals, and Dean (2015) generalized it with soft targets and temperature, and later methods such as FitNets (Romero et al., 2015) also match the teacher's intermediate representations as hints.
Distillation and related techniques
Distillation is one of several ways to obtain a smaller, cheaper model, and it's related to but distinct from transfer learning and other compression methods.
| Technique | What it does |
|---|---|
| Knowledge distillation | Trains a small student to mimic a large teacher's outputs |
| Transfer learning | Reuses a pretrained model's knowledge for a new task (broader idea) |
| Quantization | Lowers weight precision to shrink and speed up a model |
| Pruning | Removes weights or structure the model doesn't need |
These are often combined — for example, distilling a large model into a small one and then quantizing it for further savings.
Why it matters
Distillation is widely used to ship smaller models that keep most of a larger model's quality. DistilBERT (Sanh et al., 2019), a distilled version of BERT, is about 40% smaller and 60% faster while retaining roughly 97% of the original's language-understanding performance. That trade — most of the quality at a fraction of the size — is why distillation is a core tool for cost- and latency-sensitive production systems.
How Netra helps
Distillation produces a compact model; Netra Runtime helps you put it into production. You can fine-tune or train on your own data, then deploy the resulting small model behind an OpenAI-compatible API on infrastructure you own — hosted, in a VPC, on-prem, or self-hosted. See how to serve small language models or deploy a fine-tuned LLM.
Frequently asked questions
What is the difference between knowledge distillation and transfer learning?
Transfer learning is the broad idea of reusing knowledge a model already has for a new task. Knowledge distillation is a specific form of knowledge transfer in which a smaller student model is trained to mimic a larger teacher model, so the student ends up smaller and faster while keeping much of the teacher's accuracy.
What are teacher and student models?
The teacher is the large, accurate model whose behavior you want to copy. The student is the smaller model trained to reproduce the teacher's outputs, so it can run faster and cheaper while approaching the teacher's quality.
How much smaller can a distilled model be?
It depends on the task, but distillation can shrink models substantially with limited quality loss. For example, DistilBERT is about 40% smaller and 60% faster than BERT while retaining roughly 97% of its language-understanding performance.
References
- Hinton, G., Vinyals, O., & Dean, J. (2015). Distilling the Knowledge in a Neural Network. arXiv:1503.02531.
- Buciluă, C., Caruana, R., & Niculescu-Mizil, A. (2006). Model Compression. Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD '06), 535–541. doi:10.1145/1150402.1150464.
- Romero, A., Ballas, N., Kahou, S. E., Chassang, A., Gatta, C., & Bengio, Y. (2015). FitNets: Hints for Thin Deep Nets. arXiv:1412.6550.
- Sanh, V., Debut, L., Chaumond, J., & Wolf, T. (2019). DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv:1910.01108.