{"ID":2849530,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2510.23123","arxiv_id":"2510.23123","title":"Beyond Higher Rank: Token-wise Input-Output Projections for Efficient Low-Rank Adaptation","abstract":"Low-rank adaptation (LoRA) is a parameter-efficient fine-tuning (PEFT) method widely used in large language models (LLMs). LoRA essentially describes the projection of an input space into a low-dimensional output space, with the dimensionality determined by the LoRA rank. In standard LoRA, all input tokens share the same weights and undergo an identical input-output projection. This limits LoRA's ability to capture token-specific information due to the inherent semantic differences among tokens. To address this limitation, we propose Token-wise Projected Low-Rank Adaptation (TopLoRA), which dynamically adjusts LoRA weights according to the input token, thereby learning token-wise input-output projections in an end-to-end manner. Formally, the weights of TopLoRA can be expressed as $BΣ_X A$, where $A$ and $B$ are low-rank matrices (as in standard LoRA), and $Σ_X$ is a diagonal matrix generated from each input token $X$. Notably, TopLoRA does not increase the rank of LoRA weights but achieves more granular adaptation by learning token-wise LoRA weights (i.e., token-wise input-output projections). Extensive experiments across multiple models and datasets demonstrate that TopLoRA consistently outperforms LoRA and its variants. The code is available at https://github.com/Leopold1423/toplora-neurips25.","short_abstract":"Low-rank adaptation (LoRA) is a parameter-efficient fine-tuning (PEFT) method widely used in large language models (LLMs). LoRA essentially describes the projection of an input space into a low-dimensional output space, with the dimensionality determined by the LoRA rank. In standard LoRA, all input tokens share the sa...","url_abs":"https://arxiv.org/abs/2510.23123","url_pdf":"https://arxiv.org/pdf/2510.23123v1","authors":"[\"Shiwei Li\",\"Xiandi Luo\",\"Haozhao Wang\",\"Xing Tang\",\"Ziqiang Cui\",\"Dugang Liu\",\"Yuhua Li\",\"Xiuqiang He\",\"Ruixuan Li\"]","published":"2025-10-27T08:57:24Z","proceeding":"cs.CL","tasks":"[\"cs.CL\",\"cs.LG\"]","methods":"[\"Large Language Model\",\"Language Model\",\"LoRA\"]","has_code":false,"code_links":[{"ID":607715,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_id":2849530,"paper_url":"https://arxiv.org/abs/2510.23123","paper_title":"Beyond Higher Rank: Token-wise Input-Output Projections for Efficient Low-Rank Adaptation","repo_url":"https://github.com/Leopold1423/toplora-neurips25","is_official":false,"mentioned_in_paper":false,"mentioned_in_github":true,"github_stars":0}]}