{"ID":5552896,"CreatedAt":"2026-07-02T01:54:51.863792489Z","UpdatedAt":"2026-07-04T00:55:20.191773895Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2607.00304","arxiv_id":"2607.00304","title":"Mapping the Evaluation Frontier: An Empirical Survey of the Bias-Reliability Tradeoff Across Eleven Evaluator-Agent Conditions","abstract":"The bias-reliability tradeoff conjectures that LLM evaluation systems are constrained in (gamma, H, CV) space, where evaluator coupling (gamma), strategy diversity (H), and small-sample measurement reliability (CV(N)) cannot be simultaneously optimized at fixed sample size N. Prior evidence rests on n=5 conditions with complete metrics from a single study. We expand the empirical base to 11 conditions, measuring gamma and H for all 11 (nine with valid weight vectors) and CV(N=5) for seven with sufficient seeds (N \u003e= 5). Five conditions provide the complete (gamma, H, CV) triple. The data confirm the trade-off: conditions with low evaluator coupling (gamma \u003c 0.2) exhibit high measurement noise (CV(N=5) \u003e 1.0), while conditions with strong coupling (gamma \u003e 0.9) achieve low noise (CV(N=5) \u003c 0.16). The correlation r(H, gamma) = -0.989 (n=5, excluding GPT-4o conditions) confirms that evaluator coupling suppresses strategy diversity. Four GPT-4o conditions show gamma=0.000 and H=1.000 across all seeds -- a pattern we attribute to version drift in the June 2026 GPT-4o API. No condition occupies the region {gamma \u003c 0.2, CV(N=5) \u003c 0.3}. We release all per-condition metrics as a standardized benchmark dataset for evaluator comparison.","short_abstract":"The bias-reliability tradeoff conjectures that LLM evaluation systems are constrained in (gamma, H, CV) space, where evaluator coupling (gamma), strategy diversity (H), and small-sample measurement reliability (CV(N)) cannot be simultaneously optimized at fixed sample size N. Prior evidence rests on n=5 conditions with...","url_abs":"https://arxiv.org/abs/2607.00304","url_pdf":"https://arxiv.org/pdf/2607.00304v1","authors":"[\"Zewen Liu\"]","published":"2026-07-01T01:04:33Z","proceeding":"cs.LG","tasks":"[\"cs.LG\",\"cs.AI\",\"cs.CL\"]","methods":"[\"Large Language Model\"]","has_code":false}