{"ID":5675405,"CreatedAt":"2026-07-03T01:40:09.565152011Z","UpdatedAt":"2026-07-07T01:06:03.009715918Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2607.02234","arxiv_id":"2607.02234","title":"Purified OPSD: On-Policy Self-Distillation Without Losing How to Think","abstract":"On-policy self-distillation (OPSD) has emerged as a promising paradigm for improving LLM reasoning, where a privileged teacher with access to reference solutions provides token-level supervision on the student's own generated trajectories. However, we find that OPSD consistently fails on long chain-of-thought (long-CoT) reasoning models, yielding at best marginal gains while destabilizing the reflective reasoning capability these models depend on. Through a novel decomposition of the teacher's supervision signal, we identify the root cause: the teacher's supervision is dominated by a reference-induced component that drives rote memorization of reference-specific shortcuts, while the question-conditioned, inference-transferable component is ignored or actively opposed. Based on this diagnosis, we propose a two-step solution. First, we construct a reference-only teacher (the same model conditioned on the reference without the question) to isolate the non-transferable component of the supervision signal; the residual after subtracting this component captures the question-conditioned, inference-transferable correction. Second, we use pointwise mutual information (PMI) as the mechanism to transform this residual into a well-formed PMI target distribution that the student can directly distill from, filtering out the reference-induced shortcut. Experiments on four long-CoT models across two datasets demonstrate consistent improvements over both the base model and standard OPSD, while preserving the models' natural epistemic behavior throughout training.","short_abstract":"On-policy self-distillation (OPSD) has emerged as a promising paradigm for improving LLM reasoning, where a privileged teacher with access to reference solutions provides token-level supervision on the student's own generated trajectories. However, we find that OPSD consistently fails on long chain-of-thought (long-CoT...","url_abs":"https://arxiv.org/abs/2607.02234","url_pdf":"https://arxiv.org/pdf/2607.02234v1","authors":"[\"Zhanming Shen\",\"Jintao Tong\",\"Shaotian Yan\",\"Chen Shen\",\"Hao Chen\",\"Wentao Ye\",\"Xiaomeng Hu\",\"Rui Miao\",\"Haobo Wang\",\"Junbo Zhao\",\"Gang Chen\",\"Jieping Ye\"]","published":"2026-07-02T14:33:07Z","proceeding":"cs.AI","tasks":"[\"cs.AI\",\"cs.LG\"]","methods":"[\"Large Language Model\"]","has_code":false}