{"ID":5438826,"CreatedAt":"2026-07-01T01:17:58.482524686Z","UpdatedAt":"2026-07-03T11:20:51.789462812Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2606.31519","arxiv_id":"2606.31519","title":"RaBitQCache: Rotated Binary Quantization for KVCache in Long Context LLM Inference","abstract":"Long-context Large Language Model inference is severely bottlenecked by the massive Key-Value (KV) cache, yet existing sparse attention methods often suffer from static fixed-budget (Top-k) retrieval or rely on proxy scores that are computationally expensive and biased. To address these limitations, we propose RaBitQCache, a novel sparse attention framework that utilizes randomized rotated binary quantization and high-throughput binary-INT4 arithmetic to efficiently estimate attention weights. Our proxy score serves as an unbiased estimator with a proven error bound, enabling adaptive Top-p retrieval that dynamically adjusts the token budget based on actual attention sparsity. We further implement a hardware-aware system with asynchronous pipelining and lazy updates to mask overhead. Evaluations demonstrate that RaBitQCache significantly accelerates inference and reduces memory I/O while preserving generation quality compared to state-of-the-art baselines. Code is available at https://github.com/Sakuraaa0/RaBitQCache.git.","short_abstract":"Long-context Large Language Model inference is severely bottlenecked by the massive Key-Value (KV) cache, yet existing sparse attention methods often suffer from static fixed-budget (Top-k) retrieval or rely on proxy scores that are computationally expensive and biased. To address these limitations, we propose RaBitQCa...","url_abs":"https://arxiv.org/abs/2606.31519","url_pdf":"https://arxiv.org/pdf/2606.31519v1","authors":"[\"Wenhao Li\",\"Jinhao Dong\",\"Hailin Zhang\",\"Wenhang Shi\",\"Wei Lu\",\"Xiaoyong Du\"]","published":"2026-06-30T11:32:14Z","proceeding":"cs.LG","tasks":"[\"cs.LG\",\"cs.CL\"]","methods":"[\"Large Language Model\",\"Language Model\"]","has_code":false,"code_links":[{"ID":613783,"CreatedAt":"2026-07-01T01:17:58.482524686Z","UpdatedAt":"2026-07-01T01:17:58.482524686Z","DeletedAt":null,"paper_id":5438826,"paper_url":"https://arxiv.org/abs/2606.31519","paper_title":"RaBitQCache: Rotated Binary Quantization for KVCache in Long Context LLM Inference","repo_url":"https://github.com/Sakuraaa0/RaBitQCache.git","is_official":false,"mentioned_in_paper":false,"mentioned_in_github":true,"github_stars":0}]}