{"ID":5675196,"CreatedAt":"2026-07-03T01:40:09.565152011Z","UpdatedAt":"2026-07-05T07:22:02.480239843Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2607.01813","arxiv_id":"2607.01813","title":"MMBench-Live: A Continuously Evolving Benchmark for Multimodal Models","abstract":"Evaluation benchmarks are essential for assessing vision-language models (VLMs), but most multimodal benchmarks are static, making them vulnerable to temporal staleness, data contamination, and costly maintenance. We present MMBench-Live, a continuously evolving multimodal benchmark built by a multi-agent-driven automated pipeline. Our framework treats benchmark evolution as task-guided dataset construction, integrating structured benchmark specification, feedback-controlled real-time data acquisition, and verifiable QA generation with executable reasoning. To maintain cross-version comparability, we introduce a distribution-consistent update strategy that extracts task-related visual patterns from the original benchmark to guide data collection and filtering. Instantiated from MMBench, MMBench-Live contains 5.9K newly generated evaluation instances with a high answer correctness rate, while each update costs about USD 30 and takes 1-2 hours. Extensive evaluations show that MMBench-Live preserves stable model rankings, maintains semantic alignment with the original benchmark, and exhibits weaker contamination-related memorization signals, suggesting a practical and scalable paradigm for sustainable multimodal benchmark evolution. The project is available at https://github.com/PRIS-CV/MMBench-Live.","short_abstract":"Evaluation benchmarks are essential for assessing vision-language models (VLMs), but most multimodal benchmarks are static, making them vulnerable to temporal staleness, data contamination, and costly maintenance. We present MMBench-Live, a continuously evolving multimodal benchmark built by a multi-agent-driven automa...","url_abs":"https://arxiv.org/abs/2607.01813","url_pdf":"https://arxiv.org/pdf/2607.01813v1","authors":"[\"Yuanzhi Liu\",\"Shousheng Zhao\",\"Bo Zhou\",\"Kongming Liang\",\"Zhanyu Ma\"]","published":"2026-07-02T07:27:58Z","proceeding":"cs.CV","tasks":"[\"cs.CV\",\"cs.AI\"]","methods":"[\"Language Model\"]","has_code":false,"code_links":[{"ID":613886,"CreatedAt":"2026-07-03T01:40:09.565152011Z","UpdatedAt":"2026-07-03T01:40:09.565152011Z","DeletedAt":null,"paper_id":5675196,"paper_url":"https://arxiv.org/abs/2607.01813","paper_title":"MMBench-Live: A Continuously Evolving Benchmark for Multimodal Models","repo_url":"https://github.com/PRIS-CV/MMBench-Live","is_official":false,"mentioned_in_paper":false,"mentioned_in_github":true,"github_stars":0}]}