{"ID":2856944,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2510.09942","arxiv_id":"2510.09942","title":"Conformal Sparsification for Bandwidth-Efficient Edge-Cloud Speculative Decoding","abstract":"Edge-cloud speculative decoding (SD) accelerates inference by having a cloud-based large language model (LLM) that verifies draft tokens generated by a resource-constrained small language model (SLM) at the edge. A central bottleneck is the limited bandwidth of the edge-cloud link, which necessitates efficient compression of draft token distributions. We first derive an information-theoretic bound that decomposes the token rejection rate into contributions from SLM-LLM distribution mismatch and from quantization distortion. Guided by this analysis, we propose the Sparse Quantize-and-Sample SD (SQS-SD) framework, which exploits distributional sparsity through structured sparsification and lattice-based quantization. Within this framework, K-SQS applies fixed top-K truncation, while C-SQS adaptively adjusts the retained token set via online conformal prediction to ensure bounded deviation from the dense distribution. Empirical results confirm that both approaches improve end-to-end latency and rejection rates in complimentary operating regimes.","short_abstract":"Edge-cloud speculative decoding (SD) accelerates inference by having a cloud-based large language model (LLM) that verifies draft tokens generated by a resource-constrained small language model (SLM) at the edge. A central bottleneck is the limited bandwidth of the edge-cloud link, which necessitates efficient compress...","url_abs":"https://arxiv.org/abs/2510.09942","url_pdf":"https://arxiv.org/pdf/2510.09942v1","authors":"[\"Payel Bhattacharjee\",\"Fengwei Tian\",\"Meiyu Zhong\",\"Guangyi Zhang\",\"Osvaldo Simeone\",\"Ravi Tandon\"]","published":"2025-10-11T00:56:21Z","proceeding":"cs.LG","tasks":"[\"cs.LG\",\"cs.AI\",\"cs.IT\"]","methods":"[\"Large Language Model\",\"Language Model\"]","has_code":false}