{"ID":2864512,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2509.23024","arxiv_id":"2509.23024","title":"Tracing the Representation Geometry of Language Models from Pretraining to Post-training","abstract":"Standard training metrics like loss fail to explain the emergence of complex capabilities in large language models. We take a spectral approach to investigate the geometry of learned representations across pretraining and post-training, measuring effective rank (RankMe) and eigenspectrum decay ($α$-ReQ). With OLMo (1B-7B) and Pythia (160M-12B) models, we uncover a consistent non-monotonic sequence of three geometric phases during autoregressive pretraining. The initial \"warmup\" phase exhibits rapid representational collapse. This is followed by an \"entropy-seeking\" phase, where the manifold's dimensionality expands substantially, coinciding with peak n-gram memorization. Subsequently, a \"compression-seeking\" phase imposes anisotropic consolidation, selectively preserving variance along dominant eigendirections while contracting others, a transition marked with significant improvement in downstream task performance. We show these phases can emerge from a fundamental interplay of cross-entropy optimization under skewed token frequencies and representational bottlenecks ($d \\ll |V|$). Post-training further transforms geometry: SFT and DPO drive \"entropy-seeking\" dynamics to integrate specific instructional or preferential data, improving in-distribution performance while degrading out-of-distribution robustness. Conversely, RLVR induces \"compression-seeking\", enhancing reward alignment but reducing generation diversity.","short_abstract":"Standard training metrics like loss fail to explain the emergence of complex capabilities in large language models. We take a spectral approach to investigate the geometry of learned representations across pretraining and post-training, measuring effective rank (RankMe) and eigenspectrum decay ($α$-ReQ). With OLMo (1B-...","url_abs":"https://arxiv.org/abs/2509.23024","url_pdf":"https://arxiv.org/pdf/2509.23024v1","authors":"[\"Melody Zixuan Li\",\"Kumar Krishna Agrawal\",\"Arna Ghosh\",\"Komal Kumar Teru\",\"Adam Santoro\",\"Guillaume Lajoie\",\"Blake A. Richards\"]","published":"2025-09-27T00:46:29Z","proceeding":"cs.LG","tasks":"[\"cs.LG\",\"cs.AI\",\"cs.CL\"]","methods":"[\"Language Model\"]","has_code":false}