In this post, we will talk about feature-based knowledge distillation. One of the pioneering paper is “FitNets: Hints for Thin Deep Nets” by Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang,Carlo Gatta & Yoshua Bengio. The main idea of this paper is to introduce a new training method called FitNets (short for “Thin Deep … Continue reading Feature-Based Knowledge Distillation in Pytorch on MNIST