{"ID":3053360,"CreatedAt":"2026-06-04T04:41:36.695875263Z","UpdatedAt":"2026-06-06T03:14:50.67780443Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2606.04401","arxiv_id":"2606.04401","title":"TANDEM: Bi-Level Data Mixture Optimization with Twin Networks","abstract":"The capabilities of large language models (LLMs) significantly depend on training data drawn from various domains. Optimizing domain-specific mixture ratios can be modeled as a bi-level optimization problem, which we simplify into a single-level penalized form and solve with twin networks: a proxy model trained on primary data and a dynamically updated reference model trained with additional data. Our proposed method, Twin Networks for bi-level DatA mixturE optiMization (TANDEM), measures the data efficacy through the difference between the twin models and up-weights domains that benefit more from the additional data. TANDEM provides theoretical guarantees and wider applicability, compared to prior approaches. Furthermore, our bi-level perspective suggests new settings to study domain reweighting such as data-restricted scenarios and supervised fine-tuning, where optimized mixture ratios significantly improve the performance. Extensive experiments validate TANDEM's effectiveness in all scenarios.","short_abstract":"The capabilities of large language models (LLMs) significantly depend on training data drawn from various domains. Optimizing domain-specific mixture ratios can be modeled as a bi-level optimization problem, which we simplify into a single-level penalized form and solve with twin networks: a proxy model trained on prim...","url_abs":"https://arxiv.org/abs/2606.04401","url_pdf":"https://arxiv.org/pdf/2606.04401v1","authors":"[\"Jiaxing Wang\",\"Deping Xiang\",\"Jin Xu\",\"Mingyang Yi\",\"Guoqiang Gong\",\"Zicheng Zhang\",\"Haoran Li\",\"Pengzhang Liu\",\"Zhen Chen\",\"Ke Zhang\",\"Ju Fan\",\"Qixiang Jiang\"]","published":"2026-06-03T03:28:46Z","proceeding":"cs.LG","tasks":"[\"cs.LG\"]","methods":"[\"Large Language Model\",\"Language Model\"]","has_code":false}