{"ID":2834309,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2512.01278","arxiv_id":"2512.01278","title":"Accelerating Large-Scale Reasoning Model Inference with Sparse Self-Speculative Decoding","abstract":"Reasoning language models have demonstrated remarkable capabilities on challenging tasks by generating elaborate chain-of-thought (CoT) solutions. However, such lengthy generation shifts the inference bottleneck from compute-bound to memory-bound. To generate each token, the model applies full attention to all previously generated tokens, requiring memory access to an increasingly large KV-Cache. Consequently, longer generations demand more memory access for every step, leading to substantial pressure on memory bandwidth. To address this, we introduce SparseSpec, a speculative decoding framework that reuses the same model as the draft and target models (i.e., self-speculation). SparseSpec features a novel sparse attention mechanism, PillarAttn, as the draft model, which accurately selects critical tokens via elegantly reusing information from the verification stage. Furthermore, SparseSpec co-designs self-speculation with three system innovations: (1) a unified scheduler to batch token drafting and verification, (2) delayed verification for CPU/GPU overlap, and (3) dynamic KV-Cache management to maximize memory utilization. Across various models and datasets, SparseSpec outperforms state-of-the-art solutions, with an up to 2.13x throughput speedup.","short_abstract":"Reasoning language models have demonstrated remarkable capabilities on challenging tasks by generating elaborate chain-of-thought (CoT) solutions. However, such lengthy generation shifts the inference bottleneck from compute-bound to memory-bound. To generate each token, the model applies full attention to all previous...","url_abs":"https://arxiv.org/abs/2512.01278","url_pdf":"https://arxiv.org/pdf/2512.01278v1","authors":"[\"Yilong Zhao\",\"Jiaming Tang\",\"Kan Zhu\",\"Zihao Ye\",\"Chi-Chih Chang\",\"Chaofan Lin\",\"Jongseok Park\",\"Guangxuan Xiao\",\"Mohamed S. Abdelfattah\",\"Mingyu Gao\",\"Baris Kasikci\",\"Song Han\",\"Ion Stoica\"]","published":"2025-12-01T04:50:55Z","proceeding":"cs.LG","tasks":"[\"cs.LG\",\"cs.AI\"]","methods":"[\"Language Model\",\"Generative Adversarial Network\"]","has_code":false}