Distillation - Learn Data Science with Travis

Travis chat

Turning Up the Heat: The Mechanics of Model Distillation

Towards Data Science

When I first read this paper, I was struck by twin impulses. The first was that I should absolutely write a post explaining it, because of how many of its ideas are elegant and compelling — from its…

Edge 451: Is One Teacher Enough? Understanding Multi-Teacher Distillation

TheSequence

Enhancing the distillation process using more than one teacher.

What is Knowledge Distillation?

Towards Data Science

Knowledge distillation is a fascinating concept, we’ll cover briefly why we need it, how it works.

On DINO, Self-Distillation with no labels

Towards Data Science

It has been clear for some time that the Transformers had arrived in the field of computer vision to amaze, but hardly anyone could have imagined such astonishing results from a Vision Transformer in…...

Edge 453: Distillation Across Different Modalities

TheSequence

Cross modal distillation is one of the most interesting distillation methods of the new generation.

Edge 447: Not All Model Distillations are Created Equal

TheSequence

Understanding the different types of model distillation.

Using Distillation to Protect Your Neural Networks

Towards Data Science

Distillation is a hot research area. For distillation, you first train a deep learning model, the teacher network, to solve your task. Then, you train a student network, which can be any model. While…...

Distill Hiatus

Distill

Over the past five years, Distill has supported authors in publishing artifacts that push beyond the traditional expectations of scientific papers. From Gabriel Goh’s interactive exposition of momentu...

Smaller, Faster, Smarter: The Power of Model Distillation

Towards AI

Last week, we covered OpenAI’s new series of models: o1 . TL;DR: They trained the o1 models to use better reasoning by leveraging an improved chain of thought before replying. This made us think. Open...

Knowledge Distillation : Simplified

Towards Data Science

Neural models in recent years have been successful in almost every field including extremely complex problem statements. However, these models are huge in size, with millions (and billions) of…