{"ID":5937089,"CreatedAt":"2026-07-07T03:14:33.014478982Z","UpdatedAt":"2026-07-09T13:12:41.277846289Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2607.05046","arxiv_id":"2607.05046","title":"CollabEval: Statistically Efficient Collaborative Model Evaluation via Matrix Completion","abstract":"Evaluating generative AI models is a routine, but resource-intensive, process that is conducted over and over again during the course of model development. In this work, we propose Collaborative Evaluation (CollabEval), a simple, effective, and principled method for exploiting dependencies between historical runs of different models on the same tasks to improve statistical efficiency. Specifically, our approach treats model evaluation as a matrix completion problem over an $M \\times N$ matrix of evaluation scores, where $M$ is the total number of models and $N$ is the total number of evaluation prompts. We assume that a subset of these $M$ models are targeted for evaluation. For these target models only a small fraction, $p$, of prompts has been annotated with evaluation scores. Leveraging recent results in prediction-powered inference, we build a low-rank approximation of the score matrix, and use the reconstructed values as control variates in a manner that guarantees unbiased estimates of the true evaluation metric mean, in addition to statistically valid confidence intervals. Empirically, across a wide range of datasets, models, and sparsity levels $p$, we find that CollabEval substantially reduces the mean confidence interval size, and the mean squared error of the point estimate, compared to baseline methods at the same annotation budget.","short_abstract":"Evaluating generative AI models is a routine, but resource-intensive, process that is conducted over and over again during the course of model development. In this work, we propose Collaborative Evaluation (CollabEval), a simple, effective, and principled method for exploiting dependencies between historical runs of di...","url_abs":"https://arxiv.org/abs/2607.05046","url_pdf":"https://arxiv.org/pdf/2607.05046v1","authors":"[\"Adam Fisch\",\"Daniel Deutsch\",\"Joshua Maynez\",\"Alekh Agarwal\",\"Jonathan Berant\",\"William Cohen\",\"Amir Globerson\",\"Jacob Eisenstein\"]","published":"2026-07-06T13:22:04Z","proceeding":"cs.LG","tasks":"[\"cs.LG\"]","methods":"[]","has_code":false}