{"ID":2877944,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2508.18734","arxiv_id":"2508.18734","title":"Improving Noise Robust Audio-Visual Speech Recognition via Router-Gated Cross-Modal Feature Fusion","abstract":"Robust audio-visual speech recognition (AVSR) in noisy environments remains challenging, as existing systems struggle to estimate audio reliability and dynamically adjust modality reliance. We propose router-gated cross-modal feature fusion, a novel AVSR framework that adaptively reweights audio and visual features based on token-level acoustic corruption scores. Using an audio-visual feature fusion-based router, our method down-weights unreliable audio tokens and reinforces visual cues through gated cross-attention in each decoder layer. This enables the model to pivot toward the visual modality when audio quality deteriorates. Experiments on LRS3 demonstrate that our approach achieves an 16.51-42.67% relative reduction in word error rate compared to AV-HuBERT. Ablation studies confirm that both the router and gating mechanism contribute to improved robustness under real-world acoustic noise.","short_abstract":"Robust audio-visual speech recognition (AVSR) in noisy environments remains challenging, as existing systems struggle to estimate audio reliability and dynamically adjust modality reliance. We propose router-gated cross-modal feature fusion, a novel AVSR framework that adaptively reweights audio and visual features bas...","url_abs":"https://arxiv.org/abs/2508.18734","url_pdf":"https://arxiv.org/pdf/2508.18734v1","authors":"[\"DongHoon Lim\",\"YoungChae Kim\",\"Dong-Hyun Kim\",\"Da-Hee Yang\",\"Joon-Hyuk Chang\"]","published":"2025-08-26T07:05:48Z","proceeding":"cs.CV","tasks":"[\"cs.CV\",\"cs.AI\",\"cs.MM\",\"eess.AS\",\"eess.SP\"]","methods":"[]","has_code":false}