{"ID":5935777,"CreatedAt":"2026-07-07T01:22:02.77346169Z","UpdatedAt":"2026-07-07T02:10:06.972658124Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2607.03229","arxiv_id":"2607.03229","title":"HyperParallel-Mpipe: A Composable Algebra System for Optimizing MLLM Training over Supernode Clusters","abstract":"Modern AI applications have expanded beyond text-only interaction into a wide range of multimodal scenarios, making multimodal large language models (MLLMs) crucial for both research and industry. However, compared with traditional decoder-only LLM training, large-scale MLLM training often shows much lower MFU. We analyze the key pain points in MLLM training and introduce Mpipe, which uses a schedule algebra to derive concrete runtime behavior from a compact schedule specification. From this algebra, Mpipe derives transpose, a multimodal-aware heterogeneous parallel schedule that remaps modality-encoder computation into otherwise idle pipeline regions. On Ascend 910C NPU clusters, Mpipe achieves 2.70x speedup in a small-scale setting and 1.21x speedup in a 512-card large-scale setting.","short_abstract":"Modern AI applications have expanded beyond text-only interaction into a wide range of multimodal scenarios, making multimodal large language models (MLLMs) crucial for both research and industry. However, compared with traditional decoder-only LLM training, large-scale MLLM training often shows much lower MFU. We anal...","url_abs":"https://arxiv.org/abs/2607.03229","url_pdf":"https://arxiv.org/pdf/2607.03229v1","authors":"[\"Chong Li\",\"Zhengdao Yu\",\"Nelson Lossing\",\"Thibaut Tachon\",\"Pierre Leca\",\"Etienne Filhol\",\"Yujie Yuan\",\"Chong Bao\",\"Teng Su\"]","published":"2026-07-03T11:41:14Z","proceeding":"cs.DC","tasks":"[\"cs.DC\"]","methods":"[\"Large Language Model\",\"Language Model\"]","has_code":false}