{"ID":5484862,"CreatedAt":"2026-07-01T09:50:15.695553695Z","UpdatedAt":"2026-07-01T13:18:23.524231154Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2605.11205","arxiv_id":"2605.11205","title":"The Scaling Law of Evaluation Failure: Why Simple Averaging Collapses Under Data Sparsity and Item Difficulty Gaps, and How Item Response Theory Recovers Ground Truth Across Domains","abstract":"Benchmark evaluation across AI and safety-critical domains overwhelmingly relies on simple averaging. We demonstrate that this practice produces substantially misleading rankings when two conditions co-occur: (1) the evaluation matrix is sparse and (2) items vary substantially in difficulty. Through controlled simulation experiments across four domains -- NLP (GLUE), clinical drug trials, autonomous vehicle safety, and cybersecurity -- we show that Spearman rank correlation $ρ$ between simple-average rankings and ground-truth rankings degrades from $ρ= 1.000$ at 100% coverage to $ρ= 0.809$ at 67% coverage with high difficulty heterogeneity (mean over 20 seeds). A standard two-parameter logistic (2PL) Item Response Theory (IRT) model maintains $ρ\\geq 0.996$ across all conditions. A 150-condition grid sweep over sparsity $S \\in [0, 0.70]$ and difficulty gap $D \\in [0.5, 5.0]$ confirms that ranking error forms a failure surface with a strong $S \\times D$ interaction ($γ_3 = +0.20$, $t = 13.05$), while IRT maintains $ρ\\geq 0.993$ throughout. We discuss implications for Physical AI benchmarking, where evaluation matrices are often incomplete and difficulty gaps are extreme.","short_abstract":"Benchmark evaluation across AI and safety-critical domains overwhelmingly relies on simple averaging. We demonstrate that this practice produces substantially misleading rankings when two conditions co-occur: (1) the evaluation matrix is sparse and (2) items vary substantially in difficulty. Through controlled simulati...","url_abs":"https://arxiv.org/abs/2605.11205","url_pdf":"https://arxiv.org/pdf/2605.11205v1","authors":"[\"Jung Min Kang\"]","published":"2026-05-11T20:17:55Z","proceeding":"cs.LG","tasks":"[\"cs.LG\",\"cs.AI\"]","methods":"[]","has_code":false}