{"ID":5554305,"CreatedAt":"2026-07-02T02:11:27.934456424Z","UpdatedAt":"2026-07-04T16:01:31.385776155Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2607.01125","arxiv_id":"2607.01125","title":"ZO-Act: Efficient Zeroth-Order Fine-Tuning via One-Shot Activation-Informed Low-Rank Subspaces","abstract":"Zeroth-order (ZO) optimization enables fine-tuning large language models when backpropagation is unavailable or memory-prohibitive, but existing methods often perturb full model weights or randomly constructed low-dimensional subspaces, yielding high-variance estimates and limited performance. We propose ZO-Act, an activation-informed ZO fine-tuning method that restricts perturbations to a fixed low-rank subspace derived from input activations. For each linear layer, ZO-Act computes a small activation basis once at initialization and optimizes only lightweight coefficient matrices using forward-only loss evaluations. This reduces the effective perturbation dimension, exposes explicit trainable variables compatible with momentum-based optimizers such as Adam, and naturally supports quantized LLM fine-tuning by keeping low-bit weights frozen. We analyze ZO-Act as zeroth-order optimization over a restricted coefficient space and show that perturbing the low-dimensional coefficients reduces both the variance-dependent convergence term and the finite-difference error of the ZO estimator, at the cost of a controlled subspace approximation bias that is mitigated by the low-rank structure of LLM activations and gradients. Experiments on Llama-3-8B, OPT-13B, and INT4 Llama-3-8B show consistent gains over strong ZO fine-tuning baselines across language understanding, question answering, and commonsense reasoning.","short_abstract":"Zeroth-order (ZO) optimization enables fine-tuning large language models when backpropagation is unavailable or memory-prohibitive, but existing methods often perturb full model weights or randomly constructed low-dimensional subspaces, yielding high-variance estimates and limited performance. We propose ZO-Act, an act...","url_abs":"https://arxiv.org/abs/2607.01125","url_pdf":"https://arxiv.org/pdf/2607.01125v1","authors":"[\"Xun Dong\",\"Yibo Xu\",\"Naigang Wang\",\"Xin Li\",\"Penghang Yin\",\"Zi Yang\"]","published":"2026-07-01T16:12:50Z","proceeding":"cs.LG","tasks":"[\"cs.LG\",\"math.OC\"]","methods":"[\"Large Language Model\",\"Language Model\"]","has_code":false}