{"ID":2841423,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2511.12376","arxiv_id":"2511.12376","title":"BitSnap: Checkpoint Sparsification and Quantization in LLM Training","abstract":"As large language models (LLMs) continue to grow in size and complexity, efficient checkpoint saving\\\u0026loading has become crucial for managing storage, memory usage, and fault tolerance in LLM training. The current works do not comprehensively take into account the optimization of these several aspects. This paper proposes a novel checkpoint sparsification and quantization method that adapts dynamically to different training stages and model architectures. We present a comprehensive analysis of existing lossy and lossless compression techniques, identify current limitations, and introduce our adaptive approach that balances compression ratio, speed, and precision impact throughout the training process. Experiments on different sizes of LLMs demonstrate that our bitmask-based sparsification method achieves 16x compression ratio without compromising model accuracy. Additionally, the cluster-based quantization method achieves 2x compression ratio with little precision loss.","short_abstract":"As large language models (LLMs) continue to grow in size and complexity, efficient checkpoint saving\\\u0026loading has become crucial for managing storage, memory usage, and fault tolerance in LLM training. The current works do not comprehensively take into account the optimization of these several aspects. This paper propo...","url_abs":"https://arxiv.org/abs/2511.12376","url_pdf":"https://arxiv.org/pdf/2511.12376v2","authors":"[\"Yanxin Peng\",\"Qingping Li\",\"Baodong Wu\",\"Shigang Li\",\"Guohao Dai\",\"Shengen Yan\",\"Yu Wang\"]","published":"2025-11-15T22:48:59Z","proceeding":"cs.LG","tasks":"[\"cs.LG\"]","methods":"[\"Large Language Model\",\"Language Model\"]","has_code":false}