{"ID":2842736,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2511.09156","arxiv_id":"2511.09156","title":"Zero-Order Sharpness-Aware Minimization","abstract":"Prompt learning has become a key method for adapting large language models to specific tasks with limited data. However, traditional gradient-based optimization methods for tuning prompts are computationally intensive, posing challenges for efficiency. We introduce ZOSA (Zero-Order Sharpness-Aware Minimization), a novel optimization framework that integrates zero-order optimization with sharpness-aware minimization to enhance prompt tuning. ZOSA employs Rademacher perturbation vectors to estimate gradients without requiring backpropagation. By incorporating sharpness-aware principles, it targets flat minima in the loss landscape, improving generalization. An adaptive learning rate, guided by loss variability, further ensures stable convergence. Experiments on few-shot learning tasks, such as text classification and natural language inference, show that ZOSA significantly outperforms existing methods. With its theoretical foundation and computational efficiency, ZOSA offers a practical solution for prompt-based learning in resource-limited settings.","short_abstract":"Prompt learning has become a key method for adapting large language models to specific tasks with limited data. However, traditional gradient-based optimization methods for tuning prompts are computationally intensive, posing challenges for efficiency. We introduce ZOSA (Zero-Order Sharpness-Aware Minimization), a nove...","url_abs":"https://arxiv.org/abs/2511.09156","url_pdf":"https://arxiv.org/pdf/2511.09156v2","authors":"[\"Yao Fu\",\"Yihang Jin\",\"Chunxia Zhang\",\"Junmin Liu\",\"Guang Dai\",\"Haishan Ye\"]","published":"2025-11-12T09:46:39Z","proceeding":"math.ST","tasks":"[\"math.ST\"]","methods":"[\"Language Model\"]","has_code":false}