{"ID":2861445,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2510.01938","arxiv_id":"2510.01938","title":"StelLA: Subspace Learning in Low-rank Adaptation using Stiefel Manifold","abstract":"Low-rank adaptation (LoRA) has been widely adopted as a parameter-efficient technique for fine-tuning large-scale pre-trained models. However, it still lags behind full fine-tuning in performance, partly due to its insufficient exploitation of the geometric structure underlying low-rank manifolds. In this paper, we propose a geometry-aware extension of LoRA that uses a three-factor decomposition $U\\!SV^\\top$. Analogous to the structure of singular value decomposition (SVD), it separates the adapter's input and output subspaces, $V$ and $U$, from the scaling factor $S$. Our method constrains $U$ and $V$ to lie on the Stiefel manifold, ensuring their orthonormality throughout the training. To optimize on the Stiefel manifold, we employ a flexible and modular geometric optimization design that converts any Euclidean optimizer to a Riemannian one. It enables efficient subspace learning while remaining compatible with existing fine-tuning pipelines. Empirical results across a wide range of downstream tasks, including commonsense reasoning, math and code generation, image classification, and image generation, demonstrate the superior performance of our approach against the recent state-of-the-art variants of LoRA. Code is available at https://github.com/SonyResearch/stella.","short_abstract":"Low-rank adaptation (LoRA) has been widely adopted as a parameter-efficient technique for fine-tuning large-scale pre-trained models. However, it still lags behind full fine-tuning in performance, partly due to its insufficient exploitation of the geometric structure underlying low-rank manifolds. In this paper, we pro...","url_abs":"https://arxiv.org/abs/2510.01938","url_pdf":"https://arxiv.org/pdf/2510.01938v2","authors":"[\"Zhizhong Li\",\"Sina Sajadmanesh\",\"Jingtao Li\",\"Lingjuan Lyu\"]","published":"2025-10-02T11:59:13Z","proceeding":"cs.LG","tasks":"[\"cs.LG\"]","methods":"[\"LoRA\"]","has_code":false,"code_links":[{"ID":608815,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_id":2861445,"paper_url":"https://arxiv.org/abs/2510.01938","paper_title":"StelLA: Subspace Learning in Low-rank Adaptation using Stiefel Manifold","repo_url":"https://github.com/SonyResearch/stella","is_official":false,"mentioned_in_paper":false,"mentioned_in_github":true,"github_stars":0}]}