{"ID":2865292,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2509.22259","arxiv_id":"2509.22259","title":"Rotary Position Encodings for Graphs","abstract":"We study the extent to which rotary position encodings (RoPE), a recent transformer position encoding algorithm broadly adopted in large language models (LLMs) and vision transformers (ViTs), can be applied to graph-structured data. We find that rotating tokens depending on the spectrum of the graph Laplacian efficiently injects structural information into the attention mechanism, boosting performance in synthetic and real-world graph learning tasks. This approach, coined _Wave-Induced Rotary Encodings_ (WIRE), enjoys intriguing theoretical properties: it recovers regular RoPE on grids, and depends asymptotically on the graph effective resistance. Unlike bias-based relative position encodings, WIRE is compatible with linear attention.","short_abstract":"We study the extent to which rotary position encodings (RoPE), a recent transformer position encoding algorithm broadly adopted in large language models (LLMs) and vision transformers (ViTs), can be applied to graph-structured data. We find that rotating tokens depending on the spectrum of the graph Laplacian efficient...","url_abs":"https://arxiv.org/abs/2509.22259","url_pdf":"https://arxiv.org/pdf/2509.22259v3","authors":"[\"Isaac Reid\",\"Arijit Sehanobish\",\"Cederik Höfs\",\"Bruno Mlodozeniec\",\"Leonhard Vulpius\",\"Federico Barbero\",\"Adrian Weller\",\"Krzysztof Choromanski\",\"Richard E. Turner\",\"Petar Veličković\"]","published":"2025-09-26T12:20:18Z","proceeding":"cs.LG","tasks":"[\"cs.LG\",\"cs.AI\"]","methods":"[\"Vision Transformer\",\"Transformer\",\"Large Language Model\",\"Language Model\"]","has_code":false}