{"ID":2922238,"CreatedAt":"2026-06-02T02:42:49.606572591Z","UpdatedAt":"2026-06-03T00:47:32.987482086Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2606.00959","arxiv_id":"2606.00959","title":"Towards Understanding Modality Interaction in Multimodal Language Models via Partial Information Decomposition","abstract":"Understanding modality interaction in multimodal large language models (MLLMs) is central to reliable deployment. We introduce Partial Information Decomposition (PID) as a decision-level framework that separates unique, redundant, and synergistic contributions of sensory and linguistic inputs, beyond representation alignment and outcome-based evaluation. Across vision--language benchmarks, PID reveals recurring modality-use profiles: reasoning and grounding-oriented tasks tend to exhibit high synergy, whereas expert and knowledge-oriented tasks show stronger language-unique reliance. These profiles generalize across model families and predict sensitivity to modality-level interventions. We further extend PID to tri-modal systems with Sensory PID, treating language as a control variable to decompose video--audio information gain. Applied to omni-modal models, Sensory PID reveals a sensory synergy bottleneck dominated by visual information even on audio--visual fusion tasks. Finally, PID-guided reweighting provides initial evidence for improving multimodal reasoning and grounding performance.","short_abstract":"Understanding modality interaction in multimodal large language models (MLLMs) is central to reliable deployment. We introduce Partial Information Decomposition (PID) as a decision-level framework that separates unique, redundant, and synergistic contributions of sensory and linguistic inputs, beyond representation ali...","url_abs":"https://arxiv.org/abs/2606.00959","url_pdf":"https://arxiv.org/pdf/2606.00959v1","authors":"[\"Wanlong Fang\",\"Tianle Zhang\",\"Wen Tao\",\"Alvin Chan\"]","published":"2026-05-31T02:29:36Z","proceeding":"cs.AI","tasks":"[\"cs.AI\"]","methods":"[\"Large Language Model\",\"Language Model\"]","has_code":false}