{"ID":5937758,"CreatedAt":"2026-07-07T03:14:33.014478982Z","UpdatedAt":"2026-07-08T17:45:44.879869709Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2607.04429","arxiv_id":"2607.04429","title":"evalci: A Python Library for Statistically Rigorous Comparison of Language Model Evaluations","abstract":"The dominant practice in language model evaluation is to report a single accuracy number per model and declare the higher one better, without testing whether the gap could plausibly be sampling noise. On benchmarks of a few thousand items, and under temperature sampling where a model can differ from itself run to run by more than the reported gap between models, this practice routinely overstates confidence in headline claims. The statistical machinery to fix this -- confidence intervals, paired significance tests, power analysis, clustered standard errors, multiple-comparison correction -- is well established, but no standard, pip-installable tool packages it in the shape an evaluation actually takes: a per-item results table. We present evalci, a pure-Python library (numpy/scipy/pandas only) that turns a per-item results table into a publication-ready claim -- e.g., \"Model A beats Model B, $Δ=3.1$ pts, 95% CI [1.2, 5.0], paired permutation $p=0.002$, $n=1{,}319$\" -- in one function call, with adapters for lm-evaluation-harness and HELM output. Every routine is validated against an independent reference (statsmodels, or brute-force exact enumeration) rather than only against itself. As a case study, we re-analyze a public comparison of nine language models' MMLU accuracy and find that 3 of the 8 adjacent leaderboard-rank gaps are not statistically significant after correcting for the 36 pairwise comparisons the ranking implies. evalci is available at https://pypi.org/project/evalci/ (source: https://github.com/Shreyaskc/evalci, DOI: https://doi.org/10.5281/zenodo.21201815)","short_abstract":"The dominant practice in language model evaluation is to report a single accuracy number per model and declare the higher one better, without testing whether the gap could plausibly be sampling noise. On benchmarks of a few thousand items, and under temperature sampling where a model can differ from itself run to run b...","url_abs":"https://arxiv.org/abs/2607.04429","url_pdf":"https://arxiv.org/pdf/2607.04429v1","authors":"[\"Shreyas K Chandrahas\"]","published":"2026-07-05T17:48:22Z","proceeding":"cs.CL","tasks":"[\"cs.CL\",\"cs.AI\"]","methods":"[\"Language Model\"]","project_urls":"[\"https://pypi.org/project/evalci/\"]","has_code":false,"code_links":[{"ID":613984,"CreatedAt":"2026-07-07T03:14:33.014478982Z","UpdatedAt":"2026-07-07T03:14:33.014478982Z","DeletedAt":null,"paper_id":5937758,"paper_url":"https://arxiv.org/abs/2607.04429","paper_title":"evalci: A Python Library for Statistically Rigorous Comparison of Language Model Evaluations","repo_url":"https://github.com/Shreyaskc/evalci","is_official":false,"mentioned_in_paper":false,"mentioned_in_github":true,"github_stars":0}]}