{"ID":2878903,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2508.17337","arxiv_id":"2508.17337","title":"DropLoRA: Sparse Low-Rank Adaptation for Parameter-Efficient Fine-Tuning","abstract":"LoRA-based large model parameter-efficient fine-tuning (PEFT) methods use low-rank de- composition to approximate updates to model parameters. However, compared to full- parameter fine-tuning, low-rank updates often lead to a performance gap in downstream tasks. To address this, we introduce DropLoRA, a novel pruning-based approach that focuses on pruning the rank dimension. Unlike conven- tional methods that attempt to overcome the low-rank bottleneck, DropLoRA innovatively integrates a pruning module between the two low-rank matrices in LoRA to simulate dy- namic subspace learning. This dynamic low- rank subspace learning allows DropLoRA to overcome the limitations of traditional LoRA, which operates within a static subspace. By continuously adapting the learning subspace, DropLoRA significantly boosts performance without incurring additional training or infer- ence costs. Our experimental results demon- strate that DropLoRA consistently outperforms LoRA in fine-tuning the LLaMA series across a wide range of large language model gener- ation tasks, including commonsense reason- ing, mathematical reasoning, code generation, and instruction-following. Our code is avail- able at https://github.com/TayeeChang/DropLoRA.","short_abstract":"LoRA-based large model parameter-efficient fine-tuning (PEFT) methods use low-rank de- composition to approximate updates to model parameters. However, compared to full- parameter fine-tuning, low-rank updates often lead to a performance gap in downstream tasks. To address this, we introduce DropLoRA, a novel pruning-b...","url_abs":"https://arxiv.org/abs/2508.17337","url_pdf":"https://arxiv.org/pdf/2508.17337v1","authors":"[\"Haojie Zhang\"]","published":"2025-08-24T12:45:36Z","proceeding":"cs.CL","tasks":"[\"cs.CL\",\"cs.LG\"]","methods":"[\"Language Model\",\"LoRA\"]","has_code":false,"code_links":[{"ID":610523,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_id":2878903,"paper_url":"https://arxiv.org/abs/2508.17337","paper_title":"DropLoRA: Sparse Low-Rank Adaptation for Parameter-Efficient Fine-Tuning","repo_url":"https://github.com/TayeeChang/DropLoRA","is_official":false,"mentioned_in_paper":false,"mentioned_in_github":true,"github_stars":0}]}