{"ID":6029862,"CreatedAt":"2026-07-08T02:57:47.77373338Z","UpdatedAt":"2026-07-10T16:36:57.938940984Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2607.06482","arxiv_id":"2607.06482","title":"Data Analysis in the Wild: Benchmarking Large Language Models Against Real-World Data Complexities","abstract":"Current benchmarks for evaluating Large Language Models (LLMs) in data analysis often fail to reflect real-world settings. They typically focus on fact retrieval from small tables and overlook the challenges of large multi-tabular datasets, external knowledge integration, and exploratory insight discovery. We introduce DataGovBench, a benchmark derived from governmental open data designed to evaluate LLMs in practical scenarios. The benchmark includes two tasks: Table QA that requires solving complex decomposable questions and producing textual answers or visualizations, and Table Insight that evaluates the ability of models to generate expert-level findings through exploratory data analysis. Comprehensive experiments with state-of-the-art LLMs, both with and without agentic frameworks, reveal significant performance gaps across both tasks. These results suggest that current LLM-based systems remain far from satisfying the demands of real-world data analytics. DataGovBench provides a challenging benchmark for advancing research on LLMs capable of both answering analytical queries and discovering insights from data. Code and sample data are available at https://github.com/SoHasegawa/datagovbench.","short_abstract":"Current benchmarks for evaluating Large Language Models (LLMs) in data analysis often fail to reflect real-world settings. They typically focus on fact retrieval from small tables and overlook the challenges of large multi-tabular datasets, external knowledge integration, and exploratory insight discovery. We introduce...","url_abs":"https://arxiv.org/abs/2607.06482","url_pdf":"https://arxiv.org/pdf/2607.06482v1","authors":"[\"So Hasegawa\",\"Shailaja Keyur Sampat\",\"Lei Liu\",\"Wei-Peng Chen\"]","published":"2026-07-07T16:43:05Z","proceeding":"cs.CL","tasks":"[\"cs.CL\",\"cs.AI\"]","methods":"[\"Large Language Model\",\"Language Model\",\"LoRA\"]","has_code":false,"code_links":[{"ID":614039,"CreatedAt":"2026-07-08T02:57:47.77373338Z","UpdatedAt":"2026-07-08T02:57:47.77373338Z","DeletedAt":null,"paper_id":6029862,"paper_url":"https://arxiv.org/abs/2607.06482","paper_title":"Data Analysis in the Wild: Benchmarking Large Language Models Against Real-World Data Complexities","repo_url":"https://github.com/SoHasegawa/datagovbench","is_official":false,"mentioned_in_paper":false,"mentioned_in_github":true,"github_stars":0}]}