{"ID":2923633,"CreatedAt":"2026-06-02T04:05:25.881865328Z","UpdatedAt":"2026-06-04T13:12:39.622923895Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2606.02307","arxiv_id":"2606.02307","title":"FATE-VLA:Failue-aware test generation for vision-language-action models","abstract":"Vision-Language-Action (VLA) models are increasingly used as generalist robot policies, yet their evaluation still relies largely on static benchmarks that randomly sample task scenes. In high-dimensional embodied spaces, failures are sparse and clustered, so static benchmarking can underestimate robustness risks. We reframe VLA evaluation as an active failure-discovery problem and propose a failure-aware test-generation approach that combines diversity-driven exploration with surrogate models learned from observed executions. The method steers testing toward high-risk yet diverse scene regions. Across four state-of-the-art VLA models, it uncovers substantially more failures (up to +29.7 % over selected baselines) while revealing more diverse failure modes. This mean that, for instance, in the case of GR00T-N1.6, success rate dropped from 64.4% to 34.7%. More broadly, our findings call for a shift in VLA evaluation: from passive measurement on fixed task suites to adaptive, failure-seeking test generation that exposes the structure of model weaknesses before deployment.","short_abstract":"Vision-Language-Action (VLA) models are increasingly used as generalist robot policies, yet their evaluation still relies largely on static benchmarks that randomly sample task scenes. In high-dimensional embodied spaces, failures are sparse and clustered, so static benchmarking can underestimate robustness risks. We r...","url_abs":"https://arxiv.org/abs/2606.02307","url_pdf":"https://arxiv.org/pdf/2606.02307v1","authors":"[\"Arusa Kanwal\",\"Pablo Valle\",\"Shaukat Ali\",\"Aitor Arrieta\"]","published":"2026-06-01T14:27:13Z","proceeding":"cs.RO","tasks":"[\"cs.RO\"]","methods":"[\"LoRA\"]","has_code":false}