{"ID":5937694,"CreatedAt":"2026-07-07T03:14:33.014478982Z","UpdatedAt":"2026-07-08T13:43:20.341700643Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2607.04306","arxiv_id":"2607.04306","title":"SAD-LoRA: Spectral Alignment for Low-Rank Knowledge Distillation","abstract":"Distilling a fine-tuned teacher into a LoRA-adapted student is a standard recipe for parameter-efficient compression, but output-level KD does not explicitly control which rank-$r$ weight subspace the adapter occupies. We propose \\textbf{SAD-LoRA} (\\textbf{S}pectral \\textbf{A}lignment \\textbf{D}istillation), which selects this subspace from the data-weighted student-space reference update $\\DWT\\Sigx^{1/2}$ and maintains it during training via a differentiable principal-angle loss on $\\colspan(B)$. We show that the data-weighted distillation error decomposes exactly into subspace misalignment, within-subspace coefficient mismatch, and irreducible rank residual; standard KD can affect the first term only indirectly through output gradients. On controlled synthetic problems with a flat teacher spectrum, SAD-LoRA reduces the subspace-misalignment term from $51\\%$ to nearly zero and lifts final subspace alignment from $0.49$ to $1.00$. On RoBERTa-large to RoBERTa-base distillation across six GLUE tasks, SAD-LoRA improves rank efficiency: at $r{=}4$, it matches or beats the strongest included spectral baseline on five of six tasks, and at $r{=}8$ it gives the best result on SST-2 and CoLA. Ablations identify subspace alignment as the load-bearing component, while coefficient matching is auxiliary.","short_abstract":"Distilling a fine-tuned teacher into a LoRA-adapted student is a standard recipe for parameter-efficient compression, but output-level KD does not explicitly control which rank-$r$ weight subspace the adapter occupies. We propose \\textbf{SAD-LoRA} (\\textbf{S}pectral \\textbf{A}lignment \\textbf{D}istillation), which sele...","url_abs":"https://arxiv.org/abs/2607.04306","url_pdf":"https://arxiv.org/pdf/2607.04306v1","authors":"[\"Omer Tariq\",\"Syed Muhammad Raza\",\"Jeongbae Son\"]","published":"2026-07-05T13:44:37Z","proceeding":"cs.LG","tasks":"[\"cs.LG\",\"cs.CV\"]","methods":"[\"LoRA\"]","has_code":false}