{"ID":2836713,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2511.19887","arxiv_id":"2511.19887","title":"Distilling Cross-Modal Knowledge via Feature Disentanglement","abstract":"Knowledge distillation (KD) has proven highly effective for compressing large models and enhancing the performance of smaller ones. However, its effectiveness diminishes in cross-modal scenarios, such as vision-to-language distillation, where inconsistencies in representation across modalities lead to difficult knowledge transfer. To address this challenge, we propose frequency-decoupled cross-modal knowledge distillation, a method designed to decouple and balance knowledge transfer across modalities by leveraging frequency-domain features. We observed that low-frequency features exhibit high consistency across different modalities, whereas high-frequency features demonstrate extremely low cross-modal similarity. Accordingly, we apply distinct losses to these features: enforcing strong alignment in the low-frequency domain and introducing relaxed alignment for high-frequency features. We also propose a scale consistency loss to address distributional shifts between modalities, and employ a shared classifier to unify feature spaces. Extensive experiments across multiple benchmark datasets show our method substantially outperforms traditional KD and state-of-the-art cross-modal KD approaches. Code is available at https://github.com/Johumliu/FD-CMKD.","short_abstract":"Knowledge distillation (KD) has proven highly effective for compressing large models and enhancing the performance of smaller ones. However, its effectiveness diminishes in cross-modal scenarios, such as vision-to-language distillation, where inconsistencies in representation across modalities lead to difficult knowled...","url_abs":"https://arxiv.org/abs/2511.19887","url_pdf":"https://arxiv.org/pdf/2511.19887v1","authors":"[\"Junhong Liu\",\"Yuan Zhang\",\"Tao Huang\",\"Wenchao Xu\",\"Renyu Yang\"]","published":"2025-11-25T03:45:37Z","proceeding":"cs.CV","tasks":"[\"cs.CV\",\"cs.AI\"]","methods":"[]","has_code":false,"code_links":[{"ID":606624,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_id":2836713,"paper_url":"https://arxiv.org/abs/2511.19887","paper_title":"Distilling Cross-Modal Knowledge via Feature Disentanglement","repo_url":"https://github.com/Johumliu/FD-CMKD","is_official":false,"mentioned_in_paper":false,"mentioned_in_github":true,"github_stars":0}]}