{"ID":2834656,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2512.01980","arxiv_id":"2512.01980","title":"Low-Rank Prehab: Preparing Neural Networks for SVD Compression","abstract":"Low-rank approximation methods such as singular value decomposition (SVD) and its variants (e.g., Fisher-weighted SVD, Activation SVD) have recently emerged as effective tools for neural network compression. In this setting, decomposition acts as a \"surgical\" intervention, followed by fine-tuning that serves as \"rehab\" to recover accuracy. Inspired by prehabilitation in surgery, we introduce a pre-compression fine-tuning stage, Low-Rank Prehab, that explicitly encourages low-rank structure in weight matrices while preserving task performance. By conditioning the model before SVD, Prehab steers weights toward spectrally compact regions of the parameter space, enabling smoother low-rank approximation and improved recovery. Experiments on large language models (LLMs) and other Transformer-based architectures, including Vision Transformers (ViTs), show that Prehab substantially reduces the immediate accuracy drop after compression and consistently improves post-finetuning performance. Across a wide range of compression ratios, our method outperforms state-of-the-art SVD-based techniques such as SVD-LLM, highlighting the importance of preparing models for compression rather than only improving the compression and recovery stages. Source code is available at https://github.com/niqretnuh/PREHAB-SVD","short_abstract":"Low-rank approximation methods such as singular value decomposition (SVD) and its variants (e.g., Fisher-weighted SVD, Activation SVD) have recently emerged as effective tools for neural network compression. In this setting, decomposition acts as a \"surgical\" intervention, followed by fine-tuning that serves as \"rehab\"...","url_abs":"https://arxiv.org/abs/2512.01980","url_pdf":"https://arxiv.org/pdf/2512.01980v1","authors":"[\"Haoran Qin\",\"Shansita Sharma\",\"Ali Abbasi\",\"Chayne Thrash\",\"Soheil Kolouri\"]","published":"2025-12-01T18:37:53Z","proceeding":"cs.LG","tasks":"[\"cs.LG\"]","methods":"[\"Vision Transformer\",\"Transformer\",\"Large Language Model\",\"Language Model\"]","has_code":false,"code_links":[{"ID":606431,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_id":2834656,"paper_url":"https://arxiv.org/abs/2512.01980","paper_title":"Low-Rank Prehab: Preparing Neural Networks for SVD Compression","repo_url":"https://github.com/niqretnuh/PREHAB-SVD","is_official":false,"mentioned_in_paper":false,"mentioned_in_github":true,"github_stars":0}]}