{"ID":2829194,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2512.13921","arxiv_id":"2512.13921","title":"Sliding Window Recurrences for Sequence Models","abstract":"Multi-hybrid architectures are poised to take over language modeling due to better quality and performance. We introduce a hierarchical decomposition framework for linear recurrences that allows us to develop algorithms aligned with GPU memory hierarchies, yielding Sliding Window Recurrences. We focus specifically on truncating recurrences to hardware-aligned windows which are naturally jagged, limiting costly inter-warp communication. Using SWR, we develop Phalanx layers that serve as drop-in replacements for windowed attention or linear recurrences. In 1B parameter multi-hybrid models, Phalanx achieves over 10-40% speedup across 4K to 32K context length over optimized Transformers while matching perplexity.","short_abstract":"Multi-hybrid architectures are poised to take over language modeling due to better quality and performance. We introduce a hierarchical decomposition framework for linear recurrences that allows us to develop algorithms aligned with GPU memory hierarchies, yielding Sliding Window Recurrences. We focus specifically on t...","url_abs":"https://arxiv.org/abs/2512.13921","url_pdf":"https://arxiv.org/pdf/2512.13921v1","authors":"[\"Dragos Secrieru\",\"Garyk Brixi\",\"Yoshua Bengio\",\"Taiji Suzuki\",\"Michael Poli\",\"Stefano Massaroli\"]","published":"2025-12-15T21:53:17Z","proceeding":"cs.LG","tasks":"[\"cs.LG\"]","methods":"[\"Transformer\",\"Language Model\"]","has_code":false}