{"ID":2886876,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2508.02317","arxiv_id":"2508.02317","title":"VeOmni: Scaling Any Modality Model Training with Model-Centric Distributed Recipe Zoo","abstract":"Recent advances in large language models (LLMs) have driven impressive progress in omni-modal understanding and generation. However, training omni-modal LLMs remains a significant challenge due to the heterogeneous model architectures required to process diverse modalities, necessitating sophisticated system design for efficient large-scale training. Existing frameworks typically entangle model definition with parallel logic, incurring limited scalability and substantial engineering overhead for end-to-end omni-modal training. We present VeOmni, a modular and efficient training framework to accelerate the development of omni-modal LLMs. VeOmni introduces model-centric distributed recipes that decouples communication from computation, enabling efficient 3D parallelism on omni-modal LLMs. VeOmni also features a flexible configuration interface supporting seamless integration of new modalities with minimal code change. Using VeOmni, a omni-modal mixture-of-experts (MoE) model with 30B parameters can be trained with over 2,800 tokens/sec/GPU throughput and scale to 160K context lengths via 3D parallelism on 128 GPUs, showcasing its superior efficiency and scalability for training large omni-modal LLMs.","short_abstract":"Recent advances in large language models (LLMs) have driven impressive progress in omni-modal understanding and generation. However, training omni-modal LLMs remains a significant challenge due to the heterogeneous model architectures required to process diverse modalities, necessitating sophisticated system design for...","url_abs":"https://arxiv.org/abs/2508.02317","url_pdf":"https://arxiv.org/pdf/2508.02317v3","authors":"[\"Qianli Ma\",\"Yaowei Zheng\",\"Zhelun Shi\",\"Zhongkai Zhao\",\"Bin Jia\",\"Ziyue Huang\",\"Zhiqi Lin\",\"Youjie Li\",\"Jiacheng Yang\",\"Yanghua Peng\",\"Zhi Zhang\",\"Xin Liu\"]","published":"2025-08-04T11:33:04Z","proceeding":"cs.CL","tasks":"[\"cs.CL\",\"cs.AI\",\"cs.DC\"]","methods":"[\"Large Language Model\",\"Language Model\"]","has_code":false}