{"ID":5675133,"CreatedAt":"2026-07-03T01:40:09.565152011Z","UpdatedAt":"2026-07-05T04:57:17.014105577Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2607.01710","arxiv_id":"2607.01710","title":"Generic Expert Coverage for Pruning SparseMixture-of-Experts Language Models","abstract":"Sparsely activated Mixture-of-Experts (MoE) language models contain substantial structured redundancy among routed experts, but pruning them without downstream calibration data remains challenging. Existing expert-pruning methods typically rely on a single aggregated importance score, which can bias the retained set toward experts favored by dominant calibration patterns. We propose \\textbf{Generic TB-Coverage}, a coverage-aware expert pruning method that uses only generic text corpora (WikiText2 and C4) for calibration. Instead of collapsing expert utility into one score, our method profiles per-expert utility separately on each corpus and enforces a fixed-budget coverage rule that preserves high-utility experts from each corpus before constructing the final pruning mask. Across Qwen1.5-MoE-A2.7B and DeepSeek-MoE-16B-Base at 25\\%, 50\\%, and 75\\% retention budgets, our method improves average accuracy on six common zero-shot benchmarks over random pruning, REAP, and ExpertSparsity, while also reducing perplexity degradation on WikiText2 and C4. The gains are largest under aggressive pruning (25\\% and 50\\% retain), suggesting that preserving cross-corpus expert coverage is an effective generic-data prior for MoE pruning. Our improvements hold with fixed pruning budgets and no downstream calibration data.","short_abstract":"Sparsely activated Mixture-of-Experts (MoE) language models contain substantial structured redundancy among routed experts, but pruning them without downstream calibration data remains challenging. Existing expert-pruning methods typically rely on a single aggregated importance score, which can bias the retained set to...","url_abs":"https://arxiv.org/abs/2607.01710","url_pdf":"https://arxiv.org/pdf/2607.01710v1","authors":"[\"Yongqin Zeng\",\"Sicheng Pan\",\"Jiale Wang\",\"Hai-tao Zheng\",\"Hong-Gee Kim\",\"Chunxia Ma\",\"XiuTeng Zhou\"]","published":"2026-07-02T05:02:18Z","proceeding":"cs.AI","tasks":"[\"cs.AI\"]","methods":"[\"Language Model\"]","has_code":false}