{"ID":6023507,"CreatedAt":"2026-07-08T01:00:23.257252134Z","UpdatedAt":"2026-07-10T10:25:11.826308806Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2607.06118","arxiv_id":"2607.06118","title":"WebRetriever: A Large-Scale Comprehensive Benchmark for Efficient Web Agent Evaluation","abstract":"As web agents increasingly demonstrate capabilities in automated task execution, the development of robust evaluation frameworks for assessing their navigation and task completion performance has emerged as a critical research priority. However, existing benchmarks exhibit fundamental limitations. First, they suffer from insufficient scale and limited domain diversity, constraining comprehensive evaluation of cross-domain generalization. Second, prevailing LLM-as-Judge evaluation methodologies inadequately capture fine-grained interaction semantics, particularly regarding precise query formulation and filtering operations. Third, current benchmarks predominantly emphasize navigation success metrics while neglecting critical requirements for real-world deployment scenarios. To address these limitations, we introduce WebRetriever, a large-scale benchmark encompassing 800 websites and 1,550 tasks across diverse domains, including consumer, professional, and enterprise sectors, with comprehensive coverage of user intent patterns. We propose NavEval (Navigation Evaluation), a novel LLM-as-Judge framework that leverages rich interaction context beyond visual screenshots, achieving state-of-the-art alignment with human judgment across multiple evaluation datasets. Furthermore, we establish three complementary evaluation protocols that collectively provide holistic assessment of web agent capabilities: navigation proficiency, knowledge-assisted interaction, and end-to-end task completion with information extraction. Extensive experimental analysis reveals substantial performance disparities across evaluation protocols, demonstrating that navigation success alone is an insufficient predictor of real-world application effectiveness. WebRetriever delivers fine-grained diagnostic insights into agent capabilities and establishes a rigorous foundation for advancing web agent research and development.","short_abstract":"As web agents increasingly demonstrate capabilities in automated task execution, the development of robust evaluation frameworks for assessing their navigation and task completion performance has emerged as a critical research priority. However, existing benchmarks exhibit fundamental limitations. First, they suffer fr...","url_abs":"https://arxiv.org/abs/2607.06118","url_pdf":"https://arxiv.org/pdf/2607.06118v1","authors":"[\"Wei Dong\",\"Tianyu Fu\",\"Zhe Yu\",\"Hanning Wang\",\"Anyang Su\",\"Zhizhou Fang\",\"Yuyang Chen\",\"Shuo Wang\",\"Minghui Wu\",\"Ping Jiang\",\"Zhen Lei\",\"Chenxu Zhao\"]","published":"2026-07-07T10:27:31Z","proceeding":"cs.CV","tasks":"[\"cs.CV\",\"cs.MM\"]","methods":"[\"Large Language Model\"]","has_code":false}