{"ID":3052330,"CreatedAt":"2026-06-04T04:41:36.695875263Z","UpdatedAt":"2026-06-06T05:44:34.749899951Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2606.04485","arxiv_id":"2606.04485","title":"LimiX-2M: Mitigating Low-Rank Collapse and Attention Bottlenecks in Tabular Foundation Models","abstract":"Tabular foundation models (TFMs) increasingly rival tree ensembles, but their performance is often compute-inefficient: with standard affine scalar tokenization, each feature injects value variation through an essentially one-dimensional channel, and feature IDs/positional signals cannot increase within-feature value degrees of freedom, yielding weak early-layer value sensitivity and redundant hidden states. We present a unified \\emph{tokenize-and-route} framework for strong TFMs: \\textbf{RaBEL} expands each scalar into compact localized RBF features (optionally exponent-gated) to improve conditioning and shallow-layer effective rank, while a reordered bidirectional block \\textbf{S$\\rightarrow$N$\\rightarrow$F} aligns computation with the readout by aggregating cross-sample context before feature mixing and using attention pooling. Together, these changes yield \\textbf{LimiX-2M}, a 2M-parameter model that outperforms larger TabPFN-v2 and TabICL baselines on widely used tabular benchmarks while reducing training and inference costs. These results highlight value-aware tokenization and readout-aligned routing as key levers for improving the accuracy--efficiency trade-off in TFMs. Model checkpoints and inference code are available at https://github.com/limix-ldm-ai/LimiX.","short_abstract":"Tabular foundation models (TFMs) increasingly rival tree ensembles, but their performance is often compute-inefficient: with standard affine scalar tokenization, each feature injects value variation through an essentially one-dimensional channel, and feature IDs/positional signals cannot increase within-feature value d...","url_abs":"https://arxiv.org/abs/2606.04485","url_pdf":"https://arxiv.org/pdf/2606.04485v1","authors":"[\"Yuanrui Wang\",\"Xingxuan Zhang\",\"Han Yu\",\"Mingchao Ming\",\"Gang Ren\",\"Hao Yuan\",\"Li Mao\",\"Yunjia Zhang\",\"Chun Yuan\",\"Peng Cui\"]","published":"2026-06-03T06:07:33Z","proceeding":"cs.LG","tasks":"[\"cs.LG\"]","methods":"[]","has_code":false,"code_links":[{"ID":612794,"CreatedAt":"2026-06-04T04:41:36.695875263Z","UpdatedAt":"2026-06-04T04:41:36.695875263Z","DeletedAt":null,"paper_id":3052330,"paper_url":"https://arxiv.org/abs/2606.04485","paper_title":"LimiX-2M: Mitigating Low-Rank Collapse and Attention Bottlenecks in Tabular Foundation Models","repo_url":"https://github.com/limix-ldm-ai/LimiX","is_official":false,"mentioned_in_paper":false,"mentioned_in_github":true,"github_stars":0}]}