{"ID":2836288,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2511.21146","arxiv_id":"2511.21146","title":"AV-Edit: Multimodal Generative Sound Effect Editing via Audio-Visual Semantic Joint Control","abstract":"Sound effect editing-modifying audio by adding, removing, or replacing elements-remains constrained by existing approaches that rely solely on low-level signal processing or coarse text prompts, often resulting in limited flexibility and suboptimal audio quality. To address this, we propose AV-Edit, a generative sound effect editing framework that enables fine-grained editing of existing audio tracks in videos by jointly leveraging visual, audio, and text semantics. Specifically, the proposed method employs a specially designed contrastive audio-visual masking autoencoder (CAV-MAE-Edit) for multimodal pre-training, learning aligned cross-modal representations. These representations are then used to train an editorial Multimodal Diffusion Transformer (MM-DiT) capable of removing visually irrelevant sounds and generating missing audio elements consistent with video content through a correlation-based feature gating training strategy. Furthermore, we construct a dedicated video-based sound editing dataset as an evaluation benchmark. Experiments demonstrate that the proposed AV-Edit generates high-quality audio with precise modifications based on visual content, achieving state-of-the-art performance in the field of sound effect editing and exhibiting strong competitiveness in the domain of audio generation.","short_abstract":"Sound effect editing-modifying audio by adding, removing, or replacing elements-remains constrained by existing approaches that rely solely on low-level signal processing or coarse text prompts, often resulting in limited flexibility and suboptimal audio quality. To address this, we propose AV-Edit, a generative sound...","url_abs":"https://arxiv.org/abs/2511.21146","url_pdf":"https://arxiv.org/pdf/2511.21146v1","authors":"[\"Xinyue Guo\",\"Xiaoran Yang\",\"Lipan Zhang\",\"Jianxuan Yang\",\"Zhao Wang\",\"Jian Luan\"]","published":"2025-11-26T07:59:53Z","proceeding":"cs.MM","tasks":"[\"cs.MM\",\"cs.CV\",\"cs.SD\"]","methods":"[\"Diffusion Model\",\"Transformer\"]","has_code":false}