{"ID":5438677,"CreatedAt":"2026-07-01T01:17:58.482524686Z","UpdatedAt":"2026-07-03T07:00:11.61005204Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2606.31225","arxiv_id":"2606.31225","title":"A First Exploration of Neuromorphic OT-CFM for Multi-Speaker VSR","abstract":"Visual Speech Recognition (VSR) tasks in complex multi-speaker scenarios are severely hindered by rapid head motions, occlusions, and subtle lip articulations. Traditional RGB-based methods struggle here due to low rates and motion blur of frames. To overcome these, we propose LipsFlow, a neuromorphic-inspired VSR framework that converts RGB videos into high-temporal-resolution event streams. For multi-speaker, we employ ByteTrack tracking and TalkNet active speaker detection to temporally segment scenes into single-speaker clips, enabling focused per-speaker analysis. By explicitly capturing microsecond-level articulatory dynamics via learnable event-based representations, LipsFlow achieves inherent robustness against visual degradation. To efficiently model these dense event-based features and adapt to speaker-specific articulatory patterns, we introduce Optimal Transport Conditional Flow Matching (OT-CFM). It enforces deterministic, straight-line trajectory generation in a semantic latent space, slashing inference latency to just two Ordinary Differential Equation (ODE) steps. Furthermore, we design a Dual-Level Semantic Supervision mechanism combining token-level BERT weight tying and sentence-level priors to resolve homophene ambiguities. Validated on competitive benchmarks, LipsFlow achieves a state-of-the-art WER of 22.3\\% at 240 ms latency, establishing a highly robust and efficient paradigm for event-based VSR.","short_abstract":"Visual Speech Recognition (VSR) tasks in complex multi-speaker scenarios are severely hindered by rapid head motions, occlusions, and subtle lip articulations. Traditional RGB-based methods struggle here due to low rates and motion blur of frames. To overcome these, we propose LipsFlow, a neuromorphic-inspired VSR fram...","url_abs":"https://arxiv.org/abs/2606.31225","url_pdf":"https://arxiv.org/pdf/2606.31225v1","authors":"[\"Lin Chen\",\"Jingping Fang\",\"Hairui Liu\",\"Chenyang Xu\",\"Xiaorui Li\",\"Weidong Cai\",\"Xiaoming Chen\"]","published":"2026-06-30T07:05:48Z","proceeding":"cs.MM","tasks":"[\"cs.MM\"]","methods":"[\"LoRA\"]","has_code":false}