{"ID":5935900,"CreatedAt":"2026-07-07T01:22:02.77346169Z","UpdatedAt":"2026-07-07T02:10:06.972658124Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2607.02991","arxiv_id":"2607.02991","title":"GuideMe: Multi-Domain Task Guidance and Intervention in Streaming Video","abstract":"While multimodal Large Language Models (MLLMs) excel at offline video understanding, an interesting question of how far they are from serving as a real-time procedural coach remains unknown. Such a role typically requires an MLLM to continuously monitor the execution, detect mistakes, and provide corrective guidance in a closed-loop interaction. In this paper, we construct GuideMe, the first multi-domain benchmark for streaming video that supports training and evaluation of MLLMs for closed-loop interactive task guidance. It comprises 2,458 videos spanning 223.7 hours across diverse domains (\\eg, cooking, object manipulation, daily-life guidance, and fitness), with 47,775 interaction samples covering next-step instructions, completion feedback, error detection, and corrective guidance. To evaluate existing models on GuideMe, we design a three-component assessment framework to measure the capabilities of representative MLLMs, which consists of temporal-semantic bipartite matching for sequence-level alignment, behavioral classification for intervention timing, and LLM-as-a-Judge for content quality. Extensive experiments highlight a critical performance asymmetry: despite excelling at providing instructions, existing MLLMs consistently fail to identify execution errors and respond with corrective feedback. Code and data are released at https://fawnliu.github.io/project/guideme.","short_abstract":"While multimodal Large Language Models (MLLMs) excel at offline video understanding, an interesting question of how far they are from serving as a real-time procedural coach remains unknown. Such a role typically requires an MLLM to continuously monitor the execution, detect mistakes, and provide corrective guidance in...","url_abs":"https://arxiv.org/abs/2607.02991","url_pdf":"https://arxiv.org/pdf/2607.02991v1","authors":"[\"Fang Liu\",\"Jinpeng Chen\",\"Ke Xu\",\"Yuhao Liu\",\"Huankang Guan\",\"Xudong Lu\",\"Bo Yang\",\"Gerhard Hancke\",\"Rui Liu\",\"Rynson W. H. Lau\"]","published":"2026-07-03T06:00:04Z","proceeding":"cs.CV","tasks":"[\"cs.CV\"]","methods":"[\"Large Language Model\",\"Language Model\"]","has_code":false}