E2084 - Visual Contrastive Self-Distillation

🤗 Upvotes: 41 | cs.CV, cs.AI

Authors:
Yijun Liang, Yunjie Tian, Yijiang Li, Yuqi Jia, Furong Huang, Tianyi Zhou, Di Fu

Title:
Visual Contrastive Self-Distillation

Arxiv:
http://arxiv.org/abs/2607.21556v1

Abstract:
On-policy self-distillation (OPSD) is promising as it removes the external teacher required by on-policy distillation (OPD), yet it still needs asymmetric information between teacher and student to ensure that the self-teacher provides a stronger learning signal than the student. Existing methods create this asymmetry either through privileged answers or visual evidence. We ask whether both can be...