{"ID":3049961,"CreatedAt":"2026-06-04T02:13:16.786527022Z","UpdatedAt":"2026-06-06T15:28:14.12845936Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2606.05016","arxiv_id":"2606.05016","title":"TaDA: Calibrated Probe Gating for Task-Domain LoRA Merging","abstract":"Combining a task LoRA adapter with a domain LoRA adapter into a single unified model is a practical yet largely unexplored challenge. Existing methods treat both adapters as symmetric peers, applying uniform weights across all layers. We argue that task and domain adapters exhibit a consistent depth-dependent asymmetry across transformer architectures. Domain dominance increases with layer depth, while shallower layers retain stronger task-relevant signals. Motivated by this observation, we propose $\\textbf{TaDA}$ ($\\textbf{Ta}$sk-$\\textbf{D}$omain LoR$\\textbf{A}$ Merging), a training-free algorithm that exploits this structure through calibrated probe-guided per-layer gating and per-component subspace-aware merging. The gating assigns individual weights per layer and projection type using a probe signal proved invariant to adapter weight magnitude. The merging discards conflicting singular directions before combining the remaining components. $\\textbf{TaDA}$ produces a standard rank-$r$ LoRA adapter with zero inference overhead. On six scientific QA benchmarks with Llama-2-7B, TaDA achieves an average accuracy of 0.452, outperforming DARE-TIES by +3.6 percentage points and obtaining the best result on all six benchmarks. On six image classification benchmarks with ViT-L/16, TaDA reaches 85.9\\% average accuracy, improving over the strongest merging baseline while leading in three of the six individual benchmarks.","short_abstract":"Combining a task LoRA adapter with a domain LoRA adapter into a single unified model is a practical yet largely unexplored challenge. Existing methods treat both adapters as symmetric peers, applying uniform weights across all layers. We argue that task and domain adapters exhibit a consistent depth-dependent asymmetry...","url_abs":"https://arxiv.org/abs/2606.05016","url_pdf":"https://arxiv.org/pdf/2606.05016v1","authors":"[\"Huy Quoc To\",\"Fuyi Li\",\"Guangyan Huang\",\"Ming Liu\"]","published":"2026-06-03T15:39:37Z","proceeding":"cs.CL","tasks":"[\"cs.CL\"]","methods":"[\"Transformer\",\"LoRA\"]","has_code":false}