{"ID":2826350,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2512.19562","arxiv_id":"2512.19562","title":"REALM: A Real-to-Sim Validated Benchmark for Generalization in Robotic Manipulation","abstract":"Vision-Language-Action (VLA) models empower robots to understand and execute tasks described by natural language instructions. However, a key challenge lies in their ability to generalize beyond the specific environments and conditions they were trained on, which is presently difficult and expensive to evaluate in the real-world. To address this gap, we present REALM, a new simulation environment and benchmark designed to evaluate the generalization capabilities of VLA models, with a specific emphasis on establishing a strong correlation between simulated and real-world performance through high-fidelity visuals and aligned robot control. Our environment offers a suite of 15 perturbation factors, 7 manipulation skills, and more than 3,500 objects. Finally, we establish two task sets that form our benchmark and evaluate the π_{0}, π_{0}-FAST, and GR00T N1.5 VLA models, showing that generalization and robustness remain an open challenge. More broadly, we also show that simulation gives us a valuable proxy for the real-world and allows us to systematically probe for and quantify the weaknesses and failure modes of VLAs. Project page: https://martin-sedlacek.com/realm","short_abstract":"Vision-Language-Action (VLA) models empower robots to understand and execute tasks described by natural language instructions. However, a key challenge lies in their ability to generalize beyond the specific environments and conditions they were trained on, which is presently difficult and expensive to evaluate in the...","url_abs":"https://arxiv.org/abs/2512.19562","url_pdf":"https://arxiv.org/pdf/2512.19562v1","authors":"[\"Martin Sedlacek\",\"Pavlo Yefanov\",\"Georgy Ponimatkin\",\"Jai Bardhan\",\"Simon Pilc\",\"Mederic Fourmy\",\"Evangelos Kazakos\",\"Cees G. M. Snoek\",\"Josef Sivic\",\"Vladimir Petrik\"]","published":"2025-12-22T16:44:23Z","proceeding":"cs.RO","tasks":"[\"cs.RO\",\"cs.AI\"]","methods":"[]","project_urls":"[\"https://martin-sedlacek.com/realm\"]","has_code":false,"code_links":[{"ID":605735,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_id":2826350,"paper_url":"https://arxiv.org/abs/2512.19562","paper_title":"REALM: A Real-to-Sim Validated Benchmark for Generalization in Robotic Manipulation","repo_url":"https://github.com/google/nerfies","is_official":false,"mentioned_in_paper":false,"mentioned_in_github":true,"github_stars":0},{"ID":605736,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_id":2826350,"paper_url":"https://arxiv.org/abs/2512.19562","paper_title":"REALM: A Real-to-Sim Validated Benchmark for Generalization in Robotic Manipulation","repo_url":"https://github.com/martin-sedlacek/REALM","is_official":false,"mentioned_in_paper":false,"mentioned_in_github":true,"github_stars":0},{"ID":605737,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_id":2826350,"paper_url":"https://arxiv.org/abs/2512.19562","paper_title":"REALM: A Real-to-Sim Validated Benchmark for Generalization in Robotic Manipulation","repo_url":"https://github.com/nerfies/nerfies.github.io","is_official":false,"mentioned_in_paper":false,"mentioned_in_github":true,"github_stars":0}]}