{"ID":2885022,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2508.05059","arxiv_id":"2508.05059","title":"Learning from Oblivion: Predicting Knowledge Overflowed Weights via Retrodiction of Forgetting","abstract":"Pre-trained weights have become a cornerstone of modern deep learning, enabling efficient knowledge transfer and improving downstream task performance, especially in data-scarce scenarios. However, a fundamental question remains: how can we obtain better pre-trained weights that encapsulate more knowledge beyond the given dataset? In this work, we introduce KNowledge-Overflowed Weights (KNOW) prediction, a novel strategy that leverages structured forgetting and its inversion to synthesize knowledge-enriched weights. Our key insight is that sequential fine-tuning on progressively downsized datasets induces a structured forgetting process, which can be modeled and reversed to recover knowledge as if trained on a larger dataset. We construct a dataset of weight transitions governed by this controlled forgetting and employ meta-learning to model weight prediction effectively. Specifically, our KNowledge-Overflowed Weights Nowcaster (KNOWN) acts as a hyper-model that learns the general evolution of weights and predicts enhanced weights with improved generalization. Extensive experiments across diverse datasets and architectures demonstrate that KNOW prediction consistently outperforms Naive fine-tuning and simple weight prediction, leading to superior downstream performance. Our work provides a new perspective on reinterpreting forgetting dynamics to push the limits of knowledge transfer. The code and pre-trained model are available at https://github.com/jjh6297/KNOW","short_abstract":"Pre-trained weights have become a cornerstone of modern deep learning, enabling efficient knowledge transfer and improving downstream task performance, especially in data-scarce scenarios. However, a fundamental question remains: how can we obtain better pre-trained weights that encapsulate more knowledge beyond the gi...","url_abs":"https://arxiv.org/abs/2508.05059","url_pdf":"https://arxiv.org/pdf/2508.05059v2","authors":"[\"Jinhyeok Jang\",\"Jaehong Kim\",\"Jung Uk Kim\"]","published":"2025-08-07T06:23:07Z","proceeding":"cs.LG","tasks":"[\"cs.LG\",\"cs.AI\",\"cs.CV\"]","methods":"[]","has_code":false,"code_links":[{"ID":611141,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_id":2885022,"paper_url":"https://arxiv.org/abs/2508.05059","paper_title":"Learning from Oblivion: Predicting Knowledge Overflowed Weights via Retrodiction of Forgetting","repo_url":"https://github.com/jjh6297/KNOW","is_official":false,"mentioned_in_paper":false,"mentioned_in_github":true,"github_stars":0}]}