{"ID":6023360,"CreatedAt":"2026-07-08T01:00:23.257252134Z","UpdatedAt":"2026-07-10T03:42:38.375808197Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2607.05804","arxiv_id":"2607.05804","title":"TurnOPD: Making On-Policy Distillation Turn-Aware for Efficient Long-Horizon Agent Training","abstract":"On-policy distillation (OPD) trains a student policy by matching a stronger teacher on the student's own trajectories, offering a promising framework for language agent training. However, its application to long-horizon agentic tasks remains insufficiently explored. We identify two key inefficiencies in vanilla agent OPD: (1) full-horizon rollouts often waste wall-clock resources on tail turns that provide weak and noisy KL supervision, and (2) trajectory-level KL objectives concentrate most of the loss on shallow tokens, leaving deeper decision turns under-trained once initial behaviors are aligned. To address these challenges, we propose TurnOPD, a turn-level budgeting strategy for efficient on-policy distillation of long-horizon agents. TurnOPD consists of two budget controllers: adaptive rollout-depth budgeting, which uses probe-based turn statistics to determine rollout length, and progressive turn-normalized loss budgeting, which gradually shifts KL weighting from token-level to turn-balanced supervision. Experiments on ALFWorld, WebShop, and Multi-Hop Search with task-specialized teacher models show that TurnOPD achieves superior validation accuracy under equal wall-clock training budgets and advances the accuracy--time frontier beyond vanilla OPD.","short_abstract":"On-policy distillation (OPD) trains a student policy by matching a stronger teacher on the student's own trajectories, offering a promising framework for language agent training. However, its application to long-horizon agentic tasks remains insufficiently explored. We identify two key inefficiencies in vanilla agent O...","url_abs":"https://arxiv.org/abs/2607.05804","url_pdf":"https://arxiv.org/pdf/2607.05804v1","authors":"[\"Yuhang Zhou\",\"Kai Zheng\",\"Haoling Li\",\"Dengyun Peng\",\"Can Xu\",\"Jingjing Chen\"]","published":"2026-07-07T03:56:35Z","proceeding":"cs.AI","tasks":"[\"cs.AI\",\"cs.CL\"]","methods":"[]","has_code":false}