{"ID":2841364,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2511.12264","arxiv_id":"2511.12264","title":"Benchmarking that Matters: Rethinking Benchmarking for Practical Impact","abstract":"Benchmarking has driven scientific progress in Evolutionary Computation, yet current practices fall short of real-world needs. Widely used synthetic suites such as BBOB and CEC isolate algorithmic phenomena but poorly reflect the structure, constraints, and information limitations of continuous and mixed-integer optimization problems in practice. This disconnect leads to the misuse of benchmarking suites for competitions, automated algorithm selection, and industrial decision-making, despite these suites being designed for different purposes. We identify key gaps in current benchmarking practices and tooling, including limited availability of real-world-inspired problems, missing high-level features, and challenges in multi-objective and noisy settings. We propose a vision centered on curated real-world-inspired benchmarks, practitioner-accessible feature spaces and community-maintained performance databases. Real progress requires coordinated effort: A living benchmarking ecosystem that evolves with real-world insights and supports both scientific understanding and industrial use.","short_abstract":"Benchmarking has driven scientific progress in Evolutionary Computation, yet current practices fall short of real-world needs. Widely used synthetic suites such as BBOB and CEC isolate algorithmic phenomena but poorly reflect the structure, constraints, and information limitations of continuous and mixed-integer optimi...","url_abs":"https://arxiv.org/abs/2511.12264","url_pdf":"https://arxiv.org/pdf/2511.12264v1","authors":"[\"Anna V. Kononova\",\"Niki van Stein\",\"Olaf Mersmann\",\"Thomas Bäck\",\"Thomas Bartz-Beielstein\",\"Tobias Glasmachers\",\"Michael Hellwig\",\"Sebastian Krey\",\"Jakub Kůdela\",\"Boris Naujoks\",\"Leonard Papenmeier\",\"Elena Raponi\",\"Quentin Renau\",\"Jeroen Rook\",\"Lennart Schäpermeier\",\"Diederick Vermetten\",\"Daniela Zaharie\"]","published":"2025-11-15T15:42:15Z","proceeding":"cs.NE","tasks":"[\"cs.NE\"]","methods":"[]","has_code":false}