{"ID":2846351,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2511.02818","arxiv_id":"2511.02818","title":"Orion-MSP: Multi-Scale Sparse Attention for Tabular In-Context Learning","abstract":"Tabular data remain the predominant format for real-world applications. Yet, developing effective neural models for tabular data remains challenging due to heterogeneous feature types and complex interactions occurring at multiple scales. Recent advances in tabular in-context learning (ICL), such as TabPFN and TabICL, have achieved state-of-the-art performance comparable to gradient-boosted trees (GBTs) without task-specific fine-tuning. However, current architectures exhibit key limitations: (1) single-scale feature processing that overlooks hierarchical dependencies, (2) dense attention with quadratic scaling in table width, and (3) strictly sequential component processing that prevents iterative representation refinement and cross-component communication. To address these challenges, we introduce Orion-MSP, a tabular ICL architecture featuring three key innovations: (1) multi-scale processing to capture hierarchical feature interactions; (2) block-sparse attention combining windowed, global, and random patterns for scalable efficiency and long-range connectivity; and (3) a Perceiver-style memory enabling safe bidirectional information flow across components. Across diverse benchmarks, Orion-MSP matches or surpasses state-of-the-art performance while scaling effectively to high-dimensional tables, establishing a new standard for efficient tabular in-context learning. The model is publicly available at https://github.com/Lexsi-Labs/Orion-MSP .","short_abstract":"Tabular data remain the predominant format for real-world applications. Yet, developing effective neural models for tabular data remains challenging due to heterogeneous feature types and complex interactions occurring at multiple scales. Recent advances in tabular in-context learning (ICL), such as TabPFN and TabICL,...","url_abs":"https://arxiv.org/abs/2511.02818","url_pdf":"https://arxiv.org/pdf/2511.02818v3","authors":"[\"Mohamed Bouadi\",\"Pratinav Seth\",\"Aditya Tanna\",\"Vinay Kumar Sankarapu\"]","published":"2025-11-04T18:43:44Z","proceeding":"cs.AI","tasks":"[\"cs.AI\",\"cs.LG\"]","methods":"[]","has_code":false,"code_links":[{"ID":607429,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_id":2846351,"paper_url":"https://arxiv.org/abs/2511.02818","paper_title":"Orion-MSP: Multi-Scale Sparse Attention for Tabular In-Context Learning","repo_url":"https://github.com/Lexsi-Labs/Orion-MSP","is_official":false,"mentioned_in_paper":false,"mentioned_in_github":true,"github_stars":0}]}