{"ID":5937180,"CreatedAt":"2026-07-07T03:14:33.014478982Z","UpdatedAt":"2026-07-09T09:13:40.815446834Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2607.04846","arxiv_id":"2607.04846","title":"Pretraining Curricula Enable Selective Fine-tuning","abstract":"Transformers follow implicit curricula whereby some tasks are learned before others. However, how explicit pretraining curricula influence learning, generalization, and the selectivity of fine-tuning is unclear. This is important for AI safety, where fine-tuning is used to selectively suppress misaligned behaviors. Here, we compare curricula that pretrain tasks in a balanced (sampled uniformly) or an imbalanced (one task early, the other late) fashion. We show that imbalanced learning of two conflicting copy tasks promotes in-context learning and improves the selectivity of refusal fine-tuning. Ablations and activation patching show that this occurs because imbalanced pretraining encourages tasks to be disentangled in separable neural circuits, whereas balanced training routes both tasks through a common pathway. We extend these findings to a synthetic language learning task involving rule-consistent and rule-violating data, where imbalanced curricula similarly lead to more localized, less entangled rule representations, resulting in more robust rule-following behavior. Together, these results suggest that imbalanced pretraining curricula may be an important tool for promoting disentangled representations, with direct consequences for the precision and reliability of safety fine-tuning.","short_abstract":"Transformers follow implicit curricula whereby some tasks are learned before others. However, how explicit pretraining curricula influence learning, generalization, and the selectivity of fine-tuning is unclear. This is important for AI safety, where fine-tuning is used to selectively suppress misaligned behaviors. Her...","url_abs":"https://arxiv.org/abs/2607.04846","url_pdf":"https://arxiv.org/pdf/2607.04846v1","authors":"[\"Sebastian A. Bruijns\",\"Jirko Rubruck\",\"Mia H. Whitefield\",\"Kai J. Sandbrink\",\"Fazl Barez\",\"Christopher Summerfield\"]","published":"2026-07-06T09:18:35Z","proceeding":"cs.LG","tasks":"[\"cs.LG\",\"cs.AI\"]","methods":"[\"Transformer\"]","has_code":false}