{"ID":2884373,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2508.06953","arxiv_id":"2508.06953","title":"BoRA: Towards More Expressive Low-Rank Adaptation with Block Diversity","abstract":"Low-rank adaptation (LoRA) is a parameter-efficient fine-tuning (PEFT) method widely used in large language models (LLMs). It approximates the update of a pretrained weight matrix $W\\in\\mathbb{R}^{m\\times n}$ by the product of two low-rank matrices, $BA$, where $A \\in\\mathbb{R}^{r\\times n}$ and $B\\in\\mathbb{R}^{m\\times r} (r\\ll\\min\\{m,n\\})$. Increasing the dimension $r$ can raise the rank of LoRA weights (i.e., $BA$), which typically improves fine-tuning performance but also significantly increases the number of trainable parameters. In this paper, we propose Block Diversified Low-Rank Adaptation (BoRA), which improves the rank of LoRA weights with a small number of additional parameters. Specifically, BoRA treats the product $BA$ as a block matrix multiplication, where $A$ and $B$ are partitioned into $b$ blocks along the columns and rows, respectively (i.e., $A=[A_1,\\dots,A_b]$ and $B=[B_1,\\dots,B_b]^\\top$). Consequently, the product $BA$ becomes the concatenation of the block products $B_iA_j$ for $i,j\\in[b]$. To enhance the diversity of different block products, BoRA introduces a unique diagonal matrix $Σ_{i,j} \\in \\mathbb{R}^{r\\times r}$ for each block multiplication, resulting in $B_i Σ_{i,j} A_j$. By leveraging these block-wise diagonal matrices, BoRA increases the rank of LoRA weights by a factor of $b$ while only requiring $b^2r$ additional parameters. Extensive experiments across multiple datasets and models demonstrate the superiority of BoRA, and ablation studies further validate its scalability.","short_abstract":"Low-rank adaptation (LoRA) is a parameter-efficient fine-tuning (PEFT) method widely used in large language models (LLMs). It approximates the update of a pretrained weight matrix $W\\in\\mathbb{R}^{m\\times n}$ by the product of two low-rank matrices, $BA$, where $A \\in\\mathbb{R}^{r\\times n}$ and $B\\in\\mathbb{R}^{m\\times...","url_abs":"https://arxiv.org/abs/2508.06953","url_pdf":"https://arxiv.org/pdf/2508.06953v1","authors":"[\"Shiwei Li\",\"Xiandi Luo\",\"Haozhao Wang\",\"Xing Tang\",\"Ziqiang Cui\",\"Dugang Liu\",\"Yuhua Li\",\"Xiuqiang He\",\"Ruixuan Li\"]","published":"2025-08-09T11:58:39Z","proceeding":"cs.LG","tasks":"[\"cs.LG\"]","methods":"[\"Large Language Model\",\"Language Model\",\"LoRA\"]","has_code":false}