{"ID":2834723,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2512.02216","arxiv_id":"2512.02216","title":"Parameter-Efficient Subspace Optimization for LLM Fine-Tuning","abstract":"This paper develops a new perspective on parameter-efficient fine-tuning (PEFT) for LLMs, inspired by classical subspace minimization. We introduce a unifying framework, Parameter-Efficient Subspace Optimization (PESO), which recovers existing methods such as LoRA and connects them to the principled algorithmic and theoretical foundations of subspace optimization. This connection highlights a natural ``exploration--exploitation'' view of subspace methods, guiding the design of new algorithms that achieve strong convergence performance while still preserving memory efficiency. We instantiate the framework into a practical algorithm, PESO-LoRA, based on a LoRA-type parameterization. Importantly, we provide convergence guarantees stated in the full-parameter space for the induced update, addressing a key limitation of LoRA-style analyses that only track low-dimensional factors. Empirically, PESO-LoRA improves over strong PEFT baselines on standard fine-tuning benchmarks.","short_abstract":"This paper develops a new perspective on parameter-efficient fine-tuning (PEFT) for LLMs, inspired by classical subspace minimization. We introduce a unifying framework, Parameter-Efficient Subspace Optimization (PESO), which recovers existing methods such as LoRA and connects them to the principled algorithmic and the...","url_abs":"https://arxiv.org/abs/2512.02216","url_pdf":"https://arxiv.org/pdf/2512.02216v2","authors":"[\"Yuchen Lou\",\"Zeqi Ye\",\"Minshuo Chen\"]","published":"2025-12-01T21:27:15Z","proceeding":"math.OC","tasks":"[\"math.OC\"]","methods":"[\"Large Language Model\",\"LoRA\"]","has_code":false}