{"ID":2882560,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2508.11117","arxiv_id":"2508.11117","title":"Robot Policy Evaluation for Sim-to-Real Transfer: A Benchmarking Perspective","abstract":"Current vision-based robotics simulation benchmarks have significantly advanced robotic manipulation research. However, robotics is fundamentally a real-world problem, and evaluation for real-world applications has lagged behind in evaluating generalist policies. In this paper, we discuss challenges and desiderata in designing benchmarks for generalist robotic manipulation policies for the goal of sim-to-real policy transfer. We propose 1) utilizing high visual-fidelity simulation for improved sim-to-real transfer, 2) evaluating policies by systematically increasing task complexity and scenario perturbation to assess robustness, and 3) quantifying performance alignment between real-world performance and its simulation counterparts.","short_abstract":"Current vision-based robotics simulation benchmarks have significantly advanced robotic manipulation research. However, robotics is fundamentally a real-world problem, and evaluation for real-world applications has lagged behind in evaluating generalist policies. In this paper, we discuss challenges and desiderata in d...","url_abs":"https://arxiv.org/abs/2508.11117","url_pdf":"https://arxiv.org/pdf/2508.11117v1","authors":"[\"Xuning Yang\",\"Clemens Eppner\",\"Jonathan Tremblay\",\"Dieter Fox\",\"Stan Birchfield\",\"Fabio Ramos\"]","published":"2025-08-14T23:50:41Z","proceeding":"cs.RO","tasks":"[\"cs.RO\"]","methods":"[]","has_code":false}