{"ID":5551849,"CreatedAt":"2026-07-02T01:54:51.863792489Z","UpdatedAt":"2026-07-04T07:28:02.592842364Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2607.00547","arxiv_id":"2607.00547","title":"EgoGapBench: Benchmarking Egocentric Action Selection in Multi-Agent Scenes","abstract":"Existing egocentric benchmarks have primarily constructed the egocentric setting from first-person-view data, which makes it difficult to evaluate egocentric perspective itself in isolation. However, understanding first-person-view input and taking an egocentric perspective are separable abilities, especially when first-person body cues are absent or when other agents are present. To isolate egocentric perspective understanding, we introduce EgoGapBench, a diagnostic benchmark for measuring action selection in multi-agent egocentric scenes. We define the ability measured by this benchmark as Egocentric Action Selection (EAS): selecting an appropriate action from the agent's perspective in the presence of other agents. On EgoGapBench, humans answer reliably, whereas both open-source and proprietary MLLMs perform substantially worse and systematically select actions performed by other visible agents. Fine-tuning on existing egocentric data fails to close this gap and can even be detrimental. In contrast, fine-tuning on EgoGapBench training data improves accuracy but does not reach human performance. These results show that EAS is difficult to acquire from first-person-view data alone, and that MLLMs should be evaluated and trained not only for scene understanding but also for egocentric action selection.","short_abstract":"Existing egocentric benchmarks have primarily constructed the egocentric setting from first-person-view data, which makes it difficult to evaluate egocentric perspective itself in isolation. However, understanding first-person-view input and taking an egocentric perspective are separable abilities, especially when firs...","url_abs":"https://arxiv.org/abs/2607.00547","url_pdf":"https://arxiv.org/pdf/2607.00547v1","authors":"[\"Jihyeok Jung\",\"Jeewu Lee\",\"Sanghyeop Kim\",\"Chanhee Han\",\"Seong Joon Oh\"]","published":"2026-07-01T07:39:14Z","proceeding":"cs.CV","tasks":"[\"cs.CV\",\"cs.AI\"]","methods":"[\"Large Language Model\"]","has_code":false}