Distillation - Learn Data Science with Travis

Travis chat

Turning Up the Heat: The Mechanics of Model Distillation

Towards Data Science

When I first read this paper, I was struck by twin impulses. The first was that I should absolutely write a post explaining it, because of how many of its ideas are elegant and compelling — from its…

What is Knowledge Distillation?

Towards Data Science

Knowledge distillation is a fascinating concept, we’ll cover briefly why we need it, how it works.

On DINO, Self-Distillation with no labels

Towards Data Science

It has been clear for some time that the Transformers had arrived in the field of computer vision to amaze, but hardly anyone could have imagined such astonishing results from a Vision Transformer in…...

Using Distillation to Protect Your Neural Networks

Towards Data Science

Distillation is a hot research area. For distillation, you first train a deep learning model, the teacher network, to solve your task. Then, you train a student network, which can be any model. While…...

Distill Hiatus

Distill

Over the past five years, Distill has supported authors in publishing artifacts that push beyond the traditional expectations of scientific papers. From Gabriel Goh’s interactive exposition of momentu...

Smaller, Faster, Smarter: The Power of Model Distillation

Towards AI

Last week, we covered OpenAI’s new series of models: o1 . TL;DR: They trained the o1 models to use better reasoning by leveraging an improved chain of thought before replying. This made us think. Open...

Knowledge Distillation : Simplified

Towards Data Science

Neural models in recent years have been successful in almost every field including extremely complex problem statements. However, these models are huge in size, with millions (and billions) of…

Distilling Step-by-Step : Paper Review

Towards AI

Exploring one of the most recent and innovative methods in LLM compression Continue reading on Towards AI

Patient Knowledge Distillation

Towards Data Science

With the advent of deep learning, newer and more complex models are constantly improving performance on a variety of tasks. However, this improvement comes at the cost of computational and storage…

Distill Update 2018

Distill

Things that Worked Well Interfaces for Ideas Engagement as a Spectrum Software Engineering Best Practices for Scientific Publishing Challenges & Improvements The Distill Prize A Small Community Revie...

TernaryBERT: Quantization Meets Distillation

Towards Data Science

The ongoing trend of building ever larger models like BERT and GPT-3 has been accompanied by a complementary effort to reduce their size at little or no cost in accuracy. Effective models are built…

Knowledge Distillation — A Survey Through Time

Towards Data Science

In 2012, AlexNet outperformed all the existing models on the ImageNet data. Neural networks were about to see major adoption. By 2015, many state of the arts were broken. The trend was to use neural…