{"ID":2890742,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2507.18073","arxiv_id":"2507.18073","title":"Squeeze10-LLM: Squeezing LLMs' Weights by 10 Times via a Staged Mixed-Precision Quantization Method","abstract":"Deploying large language models (LLMs) is challenging due to their massive parameters and high computational costs. Ultra low-bit quantization can significantly reduce storage and accelerate inference, but extreme compression (i.e., mean bit-width \u003c= 2) often leads to severe performance degradation. To address this, we propose Squeeze10-LLM, effectively \"squeezing\" 16-bit LLMs' weights by 10 times. Specifically, Squeeze10-LLM is a staged mixed-precision post-training quantization (PTQ) framework and achieves an average of 1.6 bits per weight by quantizing 80% of the weights to 1 bit and 20% to 4 bits. We introduce Squeeze10LLM with two key innovations: Post-Binarization Activation Robustness (PBAR) and Full Information Activation Supervision (FIAS). PBAR is a refined weight significance metric that accounts for the impact of quantization on activations, improving accuracy in low-bit settings. FIAS is a strategy that preserves full activation information during quantization to mitigate cumulative error propagation across layers. Experiments on LLaMA and LLaMA2 show that Squeeze10-LLM achieves state-of-the-art performance for sub-2bit weight-only quantization, improving average accuracy from 43% to 56% on six zero-shot classification tasks--a significant boost over existing PTQ methods. Our code will be released upon publication.","short_abstract":"Deploying large language models (LLMs) is challenging due to their massive parameters and high computational costs. Ultra low-bit quantization can significantly reduce storage and accelerate inference, but extreme compression (i.e., mean bit-width \u003c= 2) often leads to severe performance degradation. To address this, we...","url_abs":"https://arxiv.org/abs/2507.18073","url_pdf":"https://arxiv.org/pdf/2507.18073v1","authors":"[\"Qingcheng Zhu\",\"Yangyang Ren\",\"Linlin Yang\",\"Mingbao Lin\",\"Yanjing Li\",\"Sheng Xu\",\"Zichao Feng\",\"Haodong Zhu\",\"Yuguang Yang\",\"Juan Zhang\",\"Runqi Wang\",\"Baochang Zhang\"]","published":"2025-07-24T03:55:19Z","proceeding":"cs.LG","tasks":"[\"cs.LG\"]","methods":"[\"Large Language Model\",\"Language Model\"]","has_code":false}