{"ID":2846994,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2511.00874","arxiv_id":"2511.00874","title":"Training with Fewer Bits: Unlocking Edge LLMs Training with Stochastic Rounding","abstract":"LLM training is resource-intensive. Quantized training improves computational and memory efficiency but introduces quantization noise, which can hinder convergence and degrade model accuracy. Stochastic Rounding (SR) has emerged as a theoretically attractive alternative to deterministic rounding, offering unbiased gradient estimates. However, its interaction with other training factors -- especially batch size -- remains under explored. In this paper, we present a theoretical and empirical study of mini-batch stochastic gradient descent (SGD) with SR, showing that increased batch sizes can compensate for reduced precision during back-propagation. Furthermore, we show that quantizing weights and activations impacts gradient variance in distinct ways. Our experiments validate these theoretical insights.","short_abstract":"LLM training is resource-intensive. Quantized training improves computational and memory efficiency but introduces quantization noise, which can hinder convergence and degrade model accuracy. Stochastic Rounding (SR) has emerged as a theoretically attractive alternative to deterministic rounding, offering unbiased grad...","url_abs":"https://arxiv.org/abs/2511.00874","url_pdf":"https://arxiv.org/pdf/2511.00874v1","authors":"[\"Taowen Liu\",\"Marta Andronic\",\"Deniz Gündüz\",\"George A. Constantinides\"]","published":"2025-11-02T09:49:34Z","proceeding":"cs.LG","tasks":"[\"cs.LG\",\"math.NA\"]","methods":"[\"Large Language Model\"]","has_code":false}