{"ID":2825438,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2512.21373","arxiv_id":"2512.21373","title":"AInsteinBench: Benchmarking Coding Agents on Scientific Repositories","abstract":"We introduce AInsteinBench, a large-scale benchmark for evaluating whether large language model (LLM) agents can operate as scientific computing development agents within real research software ecosystems. Unlike existing scientific reasoning benchmarks which focus on conceptual knowledge, or software engineering benchmarks that emphasize generic feature implementation and issue resolving, AInsteinBench evaluates models in end-to-end scientific development settings grounded in production-grade scientific repositories. The benchmark consists of tasks derived from maintainer-authored pull requests across six widely used scientific codebases, spanning quantum chemistry, quantum computing, molecular dynamics, numerical relativity, fluid dynamics, and cheminformatics. All benchmark tasks are carefully curated through multi-stage filtering and expert review to ensure scientific challenge, adequate test coverage, and well-calibrated difficulty. By leveraging evaluation in executable environments, scientifically meaningful failure modes, and test-driven verification, AInsteinBench measures a model's ability to move beyond surface-level code generation toward the core competencies required for computational scientific research.","short_abstract":"We introduce AInsteinBench, a large-scale benchmark for evaluating whether large language model (LLM) agents can operate as scientific computing development agents within real research software ecosystems. Unlike existing scientific reasoning benchmarks which focus on conceptual knowledge, or software engineering bench...","url_abs":"https://arxiv.org/abs/2512.21373","url_pdf":"https://arxiv.org/pdf/2512.21373v1","authors":"[\"Titouan Duston\",\"Shuo Xin\",\"Yang Sun\",\"Daoguang Zan\",\"Aoyan Li\",\"Shulin Xin\",\"Kai Shen\",\"Yixiao Chen\",\"Qiming Sun\",\"Ge Zhang\",\"Jiashuo Liu\",\"Huan Zhou\",\"Jingkai Liu\",\"Zhichen Pu\",\"Yuanheng Wang\",\"Bo-Xuan Ge\",\"Xin Tong\",\"Fei Ye\",\"Zhi-Chao Zhao\",\"Wen-Biao Han\",\"Zhoujian Cao\",\"Yueran Zhao\",\"Weiluo Ren\",\"Qingshen Long\",\"Yuxiao Liu\",\"Anni Huang\",\"Yidi Du\",\"Yuanyuan Rong\",\"Jiahao Peng\"]","published":"2025-12-24T08:11:11Z","proceeding":"cs.SE","tasks":"[\"cs.SE\",\"cs.AI\",\"cs.PL\"]","methods":"[\"Large Language Model\",\"Language Model\"]","has_code":false}