{"ID":2887152,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2508.02926","arxiv_id":"2508.02926","title":"GrandJury: A Collaborative Machine Learning Model Evaluation Protocol for Dynamic Quality Rubrics","abstract":"Generative Machine Learning models have become central to modern systems, powering applications in creative writing, summarization, multi-hop reasoning, and context-aware dialogue. These models underpin large-scale AI assistants, workflow automation, and autonomous decision-making. In such domains, acceptable response is rarely absolute or static, but plural and highly context-dependent. Yet standard evaluation regimes still rely on static, benchmark-style tests, incentivizing optimization toward leaderboard scores rather than alignment with dynamic user needs or evolving realities. GrandJury introduces a formal evaluation protocol combining time-decayed aggregation, complete traceability, with the support of dynamic, transparent task rubric attribution, and multi-rater human judgment. Together, these elements enable pluralistic, accountable evaluation that captures evolving consensus and surfaces disagreement. We provide an open-source implementation (grandjury PyPI package) and a public collection of Large Language Model (LLM) inference outputs to illustrate the need and method. GrandJury provides a new paradigm for AI practitioners when evaluating machine learning outputs without absolute ground truth.","short_abstract":"Generative Machine Learning models have become central to modern systems, powering applications in creative writing, summarization, multi-hop reasoning, and context-aware dialogue. These models underpin large-scale AI assistants, workflow automation, and autonomous decision-making. In such domains, acceptable response...","url_abs":"https://arxiv.org/abs/2508.02926","url_pdf":"https://arxiv.org/pdf/2508.02926v2","authors":"[\"Arthur Cho\"]","published":"2025-08-04T22:00:44Z","proceeding":"cs.LG","tasks":"[\"cs.LG\",\"cs.AI\",\"cs.HC\"]","methods":"[\"Large Language Model\",\"Language Model\"]","has_code":false}