{"ID":2835589,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2511.23319","arxiv_id":"2511.23319","title":"Every Token Counts: Generalizing 16M Ultra-Long Context in Large Language Models","abstract":"This work explores the challenge of building ``Machines that Can Remember'', framing long-term memory as the problem of efficient ultra-long context modeling. We argue that this requires three key properties: \\textbf{sparsity}, \\textbf{random-access flexibility}, and \\textbf{length generalization}. To address ultra-long-context modeling, we leverage Hierarchical Sparse Attention (HSA), a novel attention mechanism that satisfies all three properties. We integrate HSA into Transformers to build HSA-UltraLong, which is an 8B-parameter MoE model trained on over 8 trillion tokens and is rigorously evaluated on different tasks with in-domain and out-of-domain context lengths to demonstrate its capability in handling ultra-long contexts. Results show that our model performs comparably to full-attention baselines on in-domain lengths while achieving over 90\\% accuracy on most in-context retrieval tasks with contexts up to 16M. This report outlines our experimental insights and open problems, contributing a foundation for future research in ultra-long context modeling.","short_abstract":"This work explores the challenge of building ``Machines that Can Remember'', framing long-term memory as the problem of efficient ultra-long context modeling. We argue that this requires three key properties: \\textbf{sparsity}, \\textbf{random-access flexibility}, and \\textbf{length generalization}. To address ultra-lon...","url_abs":"https://arxiv.org/abs/2511.23319","url_pdf":"https://arxiv.org/pdf/2511.23319v1","authors":"[\"Xiang Hu\",\"Zhanchao Zhou\",\"Ruiqi Liang\",\"Zehuan Li\",\"Wei Wu\",\"Jianguo Li\"]","published":"2025-11-28T16:17:53Z","proceeding":"cs.CL","tasks":"[\"cs.CL\",\"cs.AI\"]","methods":"[\"Transformer\",\"Language Model\"]","has_code":false}