{"ID":2851759,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2510.19640","arxiv_id":"2510.19640","title":"Latent Space Factorization in LoRA","abstract":"Low-rank adaptation (LoRA) is a widely used method for parameter-efficient finetuning. However, existing LoRA variants lack mechanisms to explicitly disambiguate task-relevant information within the learned low-rank subspace, potentially limiting downstream performance. We propose Factorized Variational Autoencoder LoRA (FVAE-LoRA), which leverages a VAE to learn two distinct latent spaces. Our novel Evidence Lower Bound formulation explicitly promotes factorization between the latent spaces, dedicating one latent space to task-salient features and the other to residual information. Extensive experiments on text, audio, and image tasks demonstrate that FVAE-LoRA consistently outperforms standard LoRA. Moreover, spurious correlation evaluations confirm that FVAE-LoRA better isolates task-relevant signals, leading to improved robustness under distribution shifts. Our code is publicly available at: https://github.com/idiap/FVAE-LoRA","short_abstract":"Low-rank adaptation (LoRA) is a widely used method for parameter-efficient finetuning. However, existing LoRA variants lack mechanisms to explicitly disambiguate task-relevant information within the learned low-rank subspace, potentially limiting downstream performance. We propose Factorized Variational Autoencoder LoR...","url_abs":"https://arxiv.org/abs/2510.19640","url_pdf":"https://arxiv.org/pdf/2510.19640v1","authors":"[\"Shashi Kumar\",\"Yacouba Kaloga\",\"John Mitros\",\"Petr Motlicek\",\"Ina Kodrasi\"]","published":"2025-10-22T14:37:20Z","proceeding":"cs.LG","tasks":"[\"cs.LG\"]","methods":"[\"LoRA\",\"Variational Autoencoder\"]","has_code":false,"code_links":[{"ID":607934,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_id":2851759,"paper_url":"https://arxiv.org/abs/2510.19640","paper_title":"Latent Space Factorization in LoRA","repo_url":"https://github.com/idiap/FVAE-LoRA","is_official":false,"mentioned_in_paper":false,"mentioned_in_github":true,"github_stars":0}]}