Knowledge distillation transfers knowledge from a large model to a smaller one without loss of validity
Image: U.S. Navy Photo by Mass Communication Specialist 3rd Class Jeff Johnstone, Public domain, via Wikimedia Commons
Knowledge distillation
Knowledge distillation transfers knowledge from a large model to a smaller one without loss of validity
Knowledge distillation is a technique used in machine learning to transfer knowledge from a large model to a smaller one. This process allows the smaller model to retain the knowledge capacity of the larger model without the computational expense.
Example
A large neural network trained on a dataset can be distilled into a smaller network that performs nearly as well on the same tasks, making it more efficient for deployment on devices with limited resources.
Remember this
Knowledge distillation enables the use of smaller, more efficient models while maintaining high performance, which is crucial for deploying machine learning applications on devices with limited computational power.
Text adapted from Wikipedia, licensed under CC BY-SA 4.0.
Related concepts
soft targets carry more information than hard labels: they encode class similarities
Why do some learning methods need to explore more than others?
Prompt engineering
The GenAI model learns tasks from examples in the prompt
instruction tuning does: fine-tunes on (instruction, response) pairs
Fine-tuning adapts pre-trained models to new tasks
Machine learning
Can we teach computers to learn like humans?
Retrieval-augmented generation
RAG enables LLMs to access new information without retraining
Reasoning model
RLMs excel in logic, math, and programming tasks
Swipe through 100 ML concepts daily
Open Pocket Polymath