mdistiller
mdistiller is a PyTorch library serving as the official implementation for Decoupled Knowledge Distillation (DKD) presented at CVPR 2022 and DOT, a Distillation-Oriented Trainer from ICCV 2023. It provides a comprehensive suite of classical and state-of-the-art knowledge distillation algorithms tailored for mainstream computer vision benchmarks including CIFAR-100, ImageNet, and MS-COCO. The framework supports a wide range of methods such as FitNet, Attention Transfer (AT), Feature Matching (OFD), Contrastive Representation Distillation (CRD), and ReviewKD. Experimental results demonstrate that combining standard knowledge distillation with DOT or using DKD significantly improves student model performance across various teacher-student architectures like ResNet, VGG, and MobileNet variants. The library integrates with standard training pipelines, supports Weights and Biases for logging, and is designed for easy installation and configuration to accelerate research and deployment in model compression scenarios