{"ID":5675214,"CreatedAt":"2026-07-03T01:40:09.565152011Z","UpdatedAt":"2026-07-05T07:54:18.289289986Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2607.01844","arxiv_id":"2607.01844","title":"Mixture-of-Parallelisms: Towards Memory-Efficient Training Stack for Mixture-of-Experts Models","abstract":"This paper showcases a memory-efficient training stack for Mixture-of-Experts (MoE) models. It is a training paradigm that combines and specializes various existing and novel parallelism techniques at different layers and stages of the Mixture-of-Experts (MoE) model training pipeline. It leverages these techniques to achieve maximal efficiency given the physical constraints of CPU, CPU memory, GPU HBM memory, and the CPU-GPU, GPU-GPU, and node-node communication bandwidth of the GPU cluster. It also contains a novel strategy for the optimizer step to achieve high throughput and memory efficiency, enabling practitioners to conduct lossless pre-training/fine-tuning of trillion-parameter scale models, at a million context length, with just under 12 8x H200 GPU nodes, with state-of-the-art throughput and memory efficiency. In our experiments, MoP delivers 4.7x--8.2x higher per-GPU throughput than a strongly-tuned FSDP2 baseline (with the gap widening at larger scale) and sustains training at context lengths up to 1M tokens, where the baseline runs out of memory beyond 64--128K.","short_abstract":"This paper showcases a memory-efficient training stack for Mixture-of-Experts (MoE) models. It is a training paradigm that combines and specializes various existing and novel parallelism techniques at different layers and stages of the Mixture-of-Experts (MoE) model training pipeline. It leverages these techniques to a...","url_abs":"https://arxiv.org/abs/2607.01844","url_pdf":"https://arxiv.org/pdf/2607.01844v1","authors":"[\"Xuan-Phi Nguyen\",\"Shrey Pandit\",\"Yiran Zhao\",\"Semih Yavuz\",\"Silvio Savarese\",\"Shafiq Joty\"]","published":"2026-07-02T08:06:57Z","proceeding":"cs.DC","tasks":"[\"cs.DC\",\"cs.AI\"]","methods":"[]","has_code":false}