{"ID":2868526,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2509.15492","arxiv_id":"2509.15492","title":"Beyond Video-to-SFX: Video to Audio Synthesis with Environmentally Aware Speech","abstract":"The generation of realistic, context-aware audio is important in real-world applications such as video game development. While existing video-to-audio (V2A) methods mainly focus on Foley sound generation, they struggle to produce intelligible speech. Meanwhile, current environmental speech synthesis approaches remain text-driven and fail to temporally align with dynamic video content. In this paper, we propose Beyond Video-to-SFX (BVS), a method to generate synchronized audio with environmentally aware intelligible speech for given videos. We introduce a two-stage modeling method: (1) stage one is a video-guided audio semantic (V2AS) model to predict unified audio semantic tokens conditioned on phonetic cues; (2) stage two is a video-conditioned semantic-to-acoustic (VS2A) model that refines semantic tokens into detailed acoustic tokens. Experiments demonstrate the effectiveness of BVS in scenarios such as video-to-context-aware speech synthesis and immersive audio background conversion, with ablation studies further validating our design. Our demonstration is available at~\\href{https://xinleiniu.github.io/BVS-demo/}{BVS-Demo}.","short_abstract":"The generation of realistic, context-aware audio is important in real-world applications such as video game development. While existing video-to-audio (V2A) methods mainly focus on Foley sound generation, they struggle to produce intelligible speech. Meanwhile, current environmental speech synthesis approaches remain t...","url_abs":"https://arxiv.org/abs/2509.15492","url_pdf":"https://arxiv.org/pdf/2509.15492v1","authors":"[\"Xinlei Niu\",\"Jianbo Ma\",\"Dylan Harper-Harris\",\"Xiangyu Zhang\",\"Charles Patrick Martin\",\"Jing Zhang\"]","published":"2025-09-19T00:09:59Z","proceeding":"cs.SD","tasks":"[\"cs.SD\",\"cs.MM\",\"eess.AS\"]","methods":"[]","has_code":false}