The most capable AI models today are often enormous, expensive to run, and too slow for many real-world applications that need fast, cheap responses, think a model running on a smartphone rather than in a massive data center. One of the most effective ways to make a smaller model nearly as capable as a much larger one is a technique called model distillation, sometimes also called knowledge distillation.
What Is Model Distillation?
Model distillation is a technique in which a smaller, simpler model, called the student, is trained to replicate the behavior of a larger, more capable model, called the teacher. Rather than training the student only on raw labeled data, the way a model is normally trained, the student is also trained to match the teacher’s outputs, including the subtle probability distributions the teacher produces, not just its single final answer.
The goal is to transfer as much of the teacher’s learned knowledge as possible into a student model that is dramatically smaller, faster, and cheaper to run, while sacrificing as little accuracy as possible in the process. In many cases, a well-distilled student model can come remarkably close to the teacher’s performance despite having only a small fraction of its parameters.
How Model Distillation Works
The key insight behind distillation is that a trained model’s raw output probabilities carry more information than just the single predicted answer. If a teacher model looks at an image and outputs an 85% probability that it is a cat, a 10% probability it is a dog, and small probabilities for other animals, that full distribution reveals something about which categories the teacher considers similar or confusable, information a simple “correct label is cat” would not communicate at all.
The distillation process generally works like this:
- The teacher model makes predictions. The large, already-trained teacher model is run on a dataset, producing its full output probabilities, often called “soft labels,” for each example.
- The student model is trained on those soft labels. Instead of training the smaller student model only against the hard, correct answers, it is trained using a loss function that also rewards it for matching the teacher’s softer probability distributions, often alongside the true labels as well.
- Standard training proceeds. The student’s weights are updated through ordinary backpropagation, guided by this distillation-aware loss, gradually shaping the smaller network to mimic the teacher’s behavior as closely as its limited capacity allows.
Because the student is deliberately smaller than the teacher, it cannot replicate every nuance of the teacher’s behavior perfectly. Part of the art of distillation is managing this gap carefully: a student given too little capacity, or trained too briefly, risks ending up underfit relative to what it could achieve, while overly aggressive training on a narrow dataset of teacher outputs can still lead to overfitting on the specific examples used during distillation rather than genuinely absorbing the teacher’s broader capabilities.
A Simple Analogy
Imagine a seasoned master chef who has spent decades perfecting an elaborate signature dish, with dozens of subtle techniques and ingredient ratios refined through years of experience. Rather than handing a beginner cook a plain instruction like “make it taste good,” the master chef walks through their reasoning in detail, explaining not just the final recipe but why certain ingredients are balanced the way they are and which flavor combinations work best together. The beginner cook, learning from this richer guidance rather than just a basic recipe card, can produce a dish far closer to the master’s version than if they had been left to figure it out from scratch. That richer transfer of nuanced knowledge, not just the final answer but the reasoning behind it, is the essence of what distillation does between a teacher and student model.
Why Model Distillation Matters
Distillation addresses a real and growing tension in AI development: the most capable models tend to be the largest, but the largest models are often impractical to deploy at scale due to their cost, latency, and hardware requirements. Distillation offers a path to capture much of a large model’s capability in a package that is actually feasible to run in the real world.
- Faster inference: Smaller distilled models respond more quickly, which matters enormously for real-time applications like voice assistants or live translation.
- Lower costs: Running a distilled model requires far less computing power than running the original teacher model, reducing both infrastructure costs and energy consumption.
- On-device deployment: Distilled models are often small enough to run directly on phones, laptops, or embedded devices, without needing a constant connection to a powerful remote server.
- Accessibility: Smaller, cheaper models widen access to capable AI for smaller companies, independent developers, and applications operating under tight cost or hardware constraints.
The Bigger Picture
Model distillation is part of a broader family of techniques, alongside approaches like quantization and pruning, aimed at making powerful AI practical to actually deploy rather than confined to research labs with unlimited computing budgets. As AI capabilities continue to advance, the gap between what the best research models can do and what can actually run efficiently in everyday products remains a central challenge, and distillation stands as one of the most effective tools for closing it, letting the hard-won knowledge of a massive model live on inside something small enough to put in everyone’s pocket.