{"ID":6536345,"CreatedAt":"2026-07-14T01:21:01.169441415Z","UpdatedAt":"2026-07-14T05:36:24.914033594Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2607.10134","arxiv_id":"2607.10134","title":"LeRoPE: Learnable RoPE Frequencies Improve Language Modeling","abstract":"Rotary Positional Encodings (RoPE) are currently the most popular positional encodings used in modern language models. RoPE rotates two-dimensional chunks of query and key vectors, operating as a function of their relative positional offset. The position-wise rates of rotation in RoPE typically follow a geometric sequence specified by a fixed base-frequency hyperparameter. Prior work has improved performance by either increasing this parameter to slow rotation or by applying RoPE to only a subset of QK dimensions. In this work we modify RoPE by learning a scalar per frequency, treating frequencies as learnable parameters rather than hyperparameters. We validate Learned RoPE by training a ladder of language models from scratch, ranging from 52M to 2.5B parameters. We observe and analyze the emergence of a high-norm, positional LeRoPE band. LeRoPE consistently outperforms RoPE and partial RoPE across all scales, with RoPE requiring 3.4% more compute (FLOPs) to match LeRoPE at the largest scale.","short_abstract":"Rotary Positional Encodings (RoPE) are currently the most popular positional encodings used in modern language models. RoPE rotates two-dimensional chunks of query and key vectors, operating as a function of their relative positional offset. The position-wise rates of rotation in RoPE typically follow a geometric seque...","url_abs":"https://arxiv.org/abs/2607.10134","url_pdf":"https://arxiv.org/pdf/2607.10134v1","authors":"[\"Petros Karypis\",\"Sean O'Brien\",\"Shreyas Kadekodi\",\"Rui Zhu\",\"Julian McAuley\"]","published":"2026-07-11T05:47:23Z","proceeding":"cs.LG","tasks":"[\"cs.LG\"]","methods":"[\"Language Model\"]","has_code":false}