{"ID":2842708,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2511.10696","arxiv_id":"2511.10696","title":"$π$-Attention: Periodic Sparse Transformers for Efficient Long-Context Modeling","abstract":"Transformers have revolutionized natural language processing, but their quadratic complexity with respect to sequence length remains a fundamental bottleneck for long-range modeling. While sparse attention mechanisms like RingAttention reduce computational costs by restricting attention to local neighborhoods, they suffer from limited receptive fields and lack of adaptability. We present \\PiAttention, a periodic sparse Transformer that factorizes attention into ring-local neighborhoods, deterministic $π$-stride skips, and an adaptive fusion gate. The periodic structure provides predictable coverage of distant tokens, while the sparse footprint keeps the per-layer complexity linear in context length. We prove that \\PiAttention achieves $\\mathcal{O}(kL + π\\log L)$ receptive field growth compared to $\\mathcal{O}(kL)$ for RingAttention, where $k$ is the local window size, $π$ is the skip period, and $L$ is the sequence length. Extensive experiments on language modeling, retrieval, and vision-language tasks demonstrate that \\PiAttention matches or surpasses dense attention quality with 8.3\\% lower perplexity than RingAttention while using 50\\% fewer GPUs for the same context length. Our detailed ablations and visualizations reveal the importance of periodic skips, adaptive fusion, and head-level sparsity coordination for efficient long-context modeling.","short_abstract":"Transformers have revolutionized natural language processing, but their quadratic complexity with respect to sequence length remains a fundamental bottleneck for long-range modeling. While sparse attention mechanisms like RingAttention reduce computational costs by restricting attention to local neighborhoods, they suf...","url_abs":"https://arxiv.org/abs/2511.10696","url_pdf":"https://arxiv.org/pdf/2511.10696v2","authors":"[\"Dong Liu\",\"Yanxuan Yu\"]","published":"2025-11-12T09:09:13Z","proceeding":"cs.CL","tasks":"[\"cs.CL\",\"cs.AI\"]","methods":"[\"Transformer\",\"Language Model\"]","has_code":false}