{"ID":6138105,"CreatedAt":"2026-07-09T01:07:32.349475501Z","UpdatedAt":"2026-07-11T06:21:26.878598377Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2607.07047","arxiv_id":"2607.07047","title":"Riemannian Geometry for Pre-trained Language Model Embeddings","abstract":"Understanding the geometric structure of pre-trained language model embeddings matters for interpretability and safety. We ask whether sentence-level classification signal lives in the Riemannian geometry of contextual token embeddings, and probe it by extracting per-token pullback metrics from a learned encoder's analytical Jacobian and aggregating them with the Fréchet mean on the symmetric positive definite (SPD) manifold; we call this procedure Riemannian Mean Pooling (RMP). Across three datasets with non-trivial linguistic structure (CoLA, CREAK, RTE), RMP outperforms Euclidean mean pooling, while on FEVER-Symmetric, a benchmark constructed to remove annotation-driven lexical artifacts, the method correctly stays at chance. Ablations show that a randomly initialised encoder combined with Fréchet aggregation already beats Euclidean pooling on two of the three signal-bearing datasets, localising the source of the gain to the geometric aggregation rather than to learned manifold structure; the trained encoder contributes additional signal specifically on CREAK, the most knowledge-heavy of the three signal-bearing datasets.","short_abstract":"Understanding the geometric structure of pre-trained language model embeddings matters for interpretability and safety. We ask whether sentence-level classification signal lives in the Riemannian geometry of contextual token embeddings, and probe it by extracting per-token pullback metrics from a learned encoder's anal...","url_abs":"https://arxiv.org/abs/2607.07047","url_pdf":"https://arxiv.org/pdf/2607.07047v1","authors":"[\"Szczepan Konior\",\"Alexandre Quemy\",\"Przemysław Klocek\",\"Grégoire Cattan\",\"Bartłomiej Sobieski\"]","published":"2026-07-08T06:23:46Z","proceeding":"cs.CL","tasks":"[\"cs.CL\",\"cs.AI\"]","methods":"[\"Language Model\"]","has_code":false}