{"ID":2885484,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2508.03999","arxiv_id":"2508.03999","title":"Tensorized Clustered LoRA Merging for Multi-Task Interference","abstract":"Despite the success of the monolithic dense paradigm of large language models (LLMs), the LoRA adapters offer an efficient solution by fine-tuning small task-specific modules and merging them with the base model. However, in multi-task settings, merging LoRA adapters trained on heterogeneous sources frequently causes \\textit{task interference}, degrading downstream performance. To address this, we propose a tensorized clustered LoRA (TC-LoRA) library targeting to address the task interference at the \\textit{text-level} and \\textit{parameter-level}. At the \\textit{text-level}, we cluster the training samples in the embedding space to capture input-format similarities, then train a specialized LoRA adapter for each cluster. At the \\textit{parameter-level}, we introduce a joint Canonical Polyadic (CP) decomposition that disentangles task-specific and shared factors across LoRA adapters. This joint factorization preserves essential knowledge while reducing cross-task interference. Extensive experiments on out-of-domain zero-shot and skill-composition tasks-including reasoning, question answering, and coding. Compared to strong SVD-based baselines, TC-LoRA achieves +1.4\\% accuracy on Phi-3 and +2.3\\% on Mistral-7B (+2.3\\%), demonstrating the effectiveness of TC-LoRA in LLM adaptation.","short_abstract":"Despite the success of the monolithic dense paradigm of large language models (LLMs), the LoRA adapters offer an efficient solution by fine-tuning small task-specific modules and merging them with the base model. However, in multi-task settings, merging LoRA adapters trained on heterogeneous sources frequently causes \\...","url_abs":"https://arxiv.org/abs/2508.03999","url_pdf":"https://arxiv.org/pdf/2508.03999v1","authors":"[\"Zhan Su\",\"Fengran Mo\",\"Guojun Liang\",\"Jinghan Zhang\",\"Bingbing Wen\",\"Prayag Tiwari\",\"Jian-Yun Nie\"]","published":"2025-08-06T01:26:43Z","proceeding":"cs.LG","tasks":"[\"cs.LG\"]","methods":"[\"Large Language Model\",\"Language Model\",\"LoRA\"]","has_code":false}