{"ID":2834601,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2512.01868","arxiv_id":"2512.01868","title":"The Mean-Field Dynamics of Transformers","abstract":"We develop a mathematical framework that interprets Transformer attention as an interacting particle system and studies its continuum (mean-field) limits. By idealizing attention on the sphere, we connect Transformer dynamics to Wasserstein gradient flows, synchronization models (Kuramoto), and mean-shift clustering. Central to our results is a global clustering phenomenon whereby tokens cluster asymptotically after long metastable states where they are arranged into multiple clusters. We further analyze a tractable equiangular reduction to obtain exact clustering rates, show how commonly used normalization schemes alter contraction speeds, and identify a phase transition for long-context attention. The results highlight both the mechanisms that drive representation collapse and the regimes that preserve expressive, multi-cluster structure in deep attention architectures.","short_abstract":"We develop a mathematical framework that interprets Transformer attention as an interacting particle system and studies its continuum (mean-field) limits. By idealizing attention on the sphere, we connect Transformer dynamics to Wasserstein gradient flows, synchronization models (Kuramoto), and mean-shift clustering. C...","url_abs":"https://arxiv.org/abs/2512.01868","url_pdf":"https://arxiv.org/pdf/2512.01868v4","authors":"[\"Philippe Rigollet\"]","published":"2025-12-01T16:51:00Z","proceeding":"cs.LG","tasks":"[\"cs.LG\",\"math-ph\",\"math.DS\",\"math.PR\"]","methods":"[\"Transformer\"]","has_code":false}