{"ID":5439465,"CreatedAt":"2026-07-01T01:17:58.482524686Z","UpdatedAt":"2026-07-07T04:35:10.381001403Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2606.30811","arxiv_id":"2606.30811","title":"AVTok: 1D Unified Tokenization for Holistic Audio-Video Generation","abstract":"Audio-video generation has recently gained unprecedented research attention, aiming to synthesize high-quality sounding video content with fine-grained synchronization and semantic alignment between the auditory and visual components. The preceding methods predominantly adopt a dual-branch design with separate tokenization and generation modules per modality, neglecting the representation gap while necessitating intensive computational resources for proper training. Inspired by recent advancements in one-dimensional visual tokenization, we present \\textbf{AVTok}, a novel unified tokenizer designated for holistic audio-video generation. AVTok features a dual-stream transformer-based architecture with shared encoder-decoder and modal-specific learnable queries to efficiently and effectively encode an audio-video pair into a compact one-dimensional latent representation with a unified codebook. To cope with the heterogeneous information imbalance that hinders AVTok from exploiting aligned audio-visual information, we devise a hierarchical training strategy to progressively realize reconstruction capabilities for each modality. Extensive experiments demonstrate that AVTok excels both in audio-video reconstruction and when integrated into downstream pipelines for audio-to-video, video-to-audio, and class-conditional joint audio-video generation. AVTok paves the way for the challenge of joint audio-video tokenization and provides a potential direction to build unified large multimodal models for audio-video generation.","short_abstract":"Audio-video generation has recently gained unprecedented research attention, aiming to synthesize high-quality sounding video content with fine-grained synchronization and semantic alignment between the auditory and visual components. The preceding methods predominantly adopt a dual-branch design with separate tokeniza...","url_abs":"https://arxiv.org/abs/2606.30811","url_pdf":"https://arxiv.org/pdf/2606.30811v1","authors":"[\"Kien T. Pham\",\"I Chieh Chen\",\"Qifeng Chen\",\"Long Chen\"]","published":"2026-06-29T18:35:17Z","proceeding":"cs.CV","tasks":"[\"cs.CV\",\"cs.MM\",\"cs.SD\",\"eess.AS\"]","methods":"[\"Transformer\"]","has_code":false}