{"ID":2854993,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2510.13182","arxiv_id":"2510.13182","title":"Information-Theoretic Criteria for Knowledge Distillation in Multimodal Learning","abstract":"The rapid increase in multimodal data availability has sparked significant interest in cross-modal knowledge distillation (KD) techniques, where richer \"teacher\" modalities transfer information to weaker \"student\" modalities during model training to improve performance. However, despite successes across various applications, cross-modal KD does not always result in improved outcomes, primarily due to a limited theoretical understanding that could inform practice. To address this gap, we introduce the Cross-modal Complementarity Hypothesis (CCH): we propose that cross-modal KD is effective when the mutual information between teacher and student representations exceeds the mutual information between the student representation and the labels. We theoretically validate the CCH in a joint Gaussian model and further confirm it empirically across diverse multimodal datasets, including image, text, video, audio, and cancer-related omics data. Our study establishes a novel theoretical framework for understanding cross-modal KD and offers practical guidelines based on the CCH criterion to select optimal teacher modalities for improving the performance of weaker modalities.","short_abstract":"The rapid increase in multimodal data availability has sparked significant interest in cross-modal knowledge distillation (KD) techniques, where richer \"teacher\" modalities transfer information to weaker \"student\" modalities during model training to improve performance. However, despite successes across various applica...","url_abs":"https://arxiv.org/abs/2510.13182","url_pdf":"https://arxiv.org/pdf/2510.13182v1","authors":"[\"Rongrong Xie\",\"Yizhou Xu\",\"Guido Sanguinetti\"]","published":"2025-10-15T06:10:10Z","proceeding":"cs.LG","tasks":"[\"cs.LG\"]","methods":"[]","has_code":false}