{"ID":2840865,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2511.13998","arxiv_id":"2511.13998","title":"LoCoBench-Agent: An Interactive Benchmark for LLM Agents in Long-Context Software Engineering","abstract":"As large language models (LLMs) evolve into sophisticated autonomous agents capable of complex software development tasks, evaluating their real-world capabilities becomes critical. While existing benchmarks like LoCoBench~\\cite{qiu2025locobench} assess long-context code understanding, they focus on single-turn evaluation and cannot capture the multi-turn interactive nature, tool usage patterns, and adaptive reasoning required by real-world coding agents. We introduce \\textbf{LoCoBench-Agent}, a comprehensive evaluation framework specifically designed to assess LLM agents in realistic, long-context software engineering workflows. Our framework extends LoCoBench's 8,000 scenarios into interactive agent environments, enabling systematic evaluation of multi-turn conversations, tool usage efficiency, error recovery, and architectural consistency across extended development sessions. We also introduce an evaluation methodology with 9 metrics across comprehension and efficiency dimensions. Our framework provides agents with 8 specialized tools (file operations, search, code analysis) and evaluates them across context lengths ranging from 10K to 1M tokens, enabling precise assessment of long-context performance. Through systematic evaluation of state-of-the-art models, we reveal several key findings: (1) agents exhibit remarkable long-context robustness; (2) comprehension-efficiency trade-off exists with negative correlation, where thorough exploration increases comprehension but reduces efficiency; and (3) conversation efficiency varies dramatically across models, with strategic tool usage patterns differentiating high-performing agents. As the first long-context LLM agent benchmark for software engineering, LoCoBench-Agent establishes a rigorous foundation for measuring agent capabilities, identifying performance gaps, and advancing autonomous software development at scale.","short_abstract":"As large language models (LLMs) evolve into sophisticated autonomous agents capable of complex software development tasks, evaluating their real-world capabilities becomes critical. While existing benchmarks like LoCoBench~\\cite{qiu2025locobench} assess long-context code understanding, they focus on single-turn evaluat...","url_abs":"https://arxiv.org/abs/2511.13998","url_pdf":"https://arxiv.org/pdf/2511.13998v1","authors":"[\"Jielin Qiu\",\"Zuxin Liu\",\"Zhiwei Liu\",\"Rithesh Murthy\",\"Jianguo Zhang\",\"Haolin Chen\",\"Shiyu Wang\",\"Ming Zhu\",\"Liangwei Yang\",\"Juntao Tan\",\"Roshan Ram\",\"Akshara Prabhakar\",\"Tulika Awalgaonkar\",\"Zixiang Chen\",\"Zhepeng Cen\",\"Cheng Qian\",\"Shelby Heinecke\",\"Weiran Yao\",\"Silvio Savarese\",\"Caiming Xiong\",\"Huan Wang\"]","published":"2025-11-17T23:57:24Z","proceeding":"cs.SE","tasks":"[\"cs.SE\",\"cs.AI\"]","methods":"[\"Large Language Model\",\"Language Model\",\"LoRA\"]","has_code":false}