{"ID":2886871,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2508.02308","arxiv_id":"2508.02308","title":"LaMPE: Length-aware Multi-grained Positional Encoding for Adaptive Long-context Scaling Without Training","abstract":"Large language models (LLMs) experience significant performance degradation when the input exceeds the pretraining context window, primarily due to the out-of-distribution (OOD) behavior of Rotary Position Embedding (RoPE). Recent studies mitigate this problem by remapping OOD positions into the in-distribution range with fixed mapping strategies, ignoring the dynamic relationship between input length and the model's effective context window. To this end, we propose Length-aware Multi-grained Positional Encoding (LaMPE), a training-free method that fully utilizes the model's effective context window for adaptive long-context scaling in LLMs. Motivated by the left-skewed frequency distribution of relative positions, LaMPE establishes a dynamic relationship between mapping length and input length through a parametric scaled sigmoid function to adaptively allocate positional capacity across varying input lengths. Meanwhile, LaMPE devises a novel multi-grained attention mechanism that strategically allocates positional resolution across different sequence regions to capture both fine-grained locality and long-range dependencies. Our method can be seamlessly applied to a wide range of RoPE-based LLMs without training. Extensive experiments on three representative LLMs across five mainstream long-context benchmarks demonstrate that LaMPE achieves significant performance improvements compared to existing length extrapolation methods. The code will be released at https://github.com/scar-on/LaMPE.","short_abstract":"Large language models (LLMs) experience significant performance degradation when the input exceeds the pretraining context window, primarily due to the out-of-distribution (OOD) behavior of Rotary Position Embedding (RoPE). Recent studies mitigate this problem by remapping OOD positions into the in-distribution range w...","url_abs":"https://arxiv.org/abs/2508.02308","url_pdf":"https://arxiv.org/pdf/2508.02308v2","authors":"[\"Sikui Zhang\",\"Guangze Gao\",\"Ziyun Gan\",\"Chunfeng Yuan\",\"Zefeng Lin\",\"Houwen Peng\",\"Bing Li\",\"Weiming Hu\"]","published":"2025-08-04T11:22:13Z","proceeding":"cs.CL","tasks":"[\"cs.CL\"]","methods":"[\"Large Language Model\",\"Language Model\"]","has_code":false,"code_links":[{"ID":611375,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_id":2886871,"paper_url":"https://arxiv.org/abs/2508.02308","paper_title":"LaMPE: Length-aware Multi-grained Positional Encoding for Adaptive Long-context Scaling Without Training","repo_url":"https://github.com/scar-on/LaMPE","is_official":false,"mentioned_in_paper":false,"mentioned_in_github":true,"github_stars":0}]}