{"ID":5554350,"CreatedAt":"2026-07-02T02:11:27.934456424Z","UpdatedAt":"2026-07-08T06:46:20.493522673Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2607.01218","arxiv_id":"2607.01218","title":"The State-Prediction Separation Hypothesis","abstract":"Transformers use the same forward computation stream to both predict the next token and store useful state for future token predictions. We formulate the \\emph{state-prediction separation hypothesis}: disentangling the two roles yields better language modeling performance. We design a Transformer variant that uses two computation streams to separate the two functions, and conduct pretraining experiments across various scales. Our experiments show that state-prediction separation consistently offers better data and compute efficiencies, improving validation loss and outperforming standard Transformers by 2--3 percentage points on average on downstream tasks. We also conduct extensive empirical analysis that rules out potential confounders and demonstrates the fundamental difference in the gradients our design entails.","short_abstract":"Transformers use the same forward computation stream to both predict the next token and store useful state for future token predictions. We formulate the \\emph{state-prediction separation hypothesis}: disentangling the two roles yields better language modeling performance. We design a Transformer variant that uses two...","url_abs":"https://arxiv.org/abs/2607.01218","url_pdf":"https://arxiv.org/pdf/2607.01218v1","authors":"[\"Giovanni Monea\",\"Nathan Godey\",\"Kianté Brantley\",\"Yoav Artzi\"]","published":"2026-07-01T17:55:09Z","proceeding":"cs.CL","tasks":"[\"cs.CL\",\"cs.AI\",\"cs.LG\"]","methods":"[\"Transformer\",\"Language Model\"]","has_code":false}