{"ID":2922210,"CreatedAt":"2026-06-02T02:42:49.606572591Z","UpdatedAt":"2026-06-02T22:51:43.083599133Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2606.00920","arxiv_id":"2606.00920","title":"Accuracy, Stability, and Repeated-Run Reliability of Large Language Models on Deterministic Programming Tasks","abstract":"Run-level pass rate overstates retry-free coverage by up to 17.8 percentage points -- and the gap is largest precisely for mid-performing systems. We investigate this accuracy--stability relationship in large language model (LLM) evaluation for deterministic text-conditioned generation, using programming tasks as a concrete testbed. Standard code-generation benchmarks emphasize single-run accuracy or eventual success under repeated sampling, but many deployment settings also require stability: consistent outcomes across repeated invocations under the same task description. We present a repeated-run evaluation protocol with metrics for run-level accuracy, retry-free coverage, and per-problem variability. On a recency-based benchmark of 100 LeetCode-style problems, we evaluate 16 models from five provider families under two prompt templates with five repeated runs per problem, yielding 16,000 evaluation instances. Although run-level pass rate and perfect stability rate are strongly correlated (r=0.985), pass rate consistently exceeds retry-free coverage -- a gap that reaches 17.8 percentage points and reverses model rankings even among closely matched systems. Prompt effects are model-dependent rather than uniformly beneficial. These results suggest that repeated-run stability analysis is a necessary complement to conventional accuracy reporting for deterministic text-conditioned generation tasks.","short_abstract":"Run-level pass rate overstates retry-free coverage by up to 17.8 percentage points -- and the gap is largest precisely for mid-performing systems. We investigate this accuracy--stability relationship in large language model (LLM) evaluation for deterministic text-conditioned generation, using programming tasks as a con...","url_abs":"https://arxiv.org/abs/2606.00920","url_pdf":"https://arxiv.org/pdf/2606.00920v1","authors":"[\"Yongxi Zhou\",\"Lai Yun Choi\",\"Jiaxi Wen\",\"Wenbo Ye\"]","published":"2026-05-30T23:03:05Z","proceeding":"cs.LG","tasks":"[\"cs.LG\",\"cs.AI\",\"cs.SE\"]","methods":"[\"Large Language Model\",\"Language Model\"]","has_code":false}