{"ID":2838658,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2511.17335","arxiv_id":"2511.17335","title":"Robot Confirmation Generation and Action Planning Using Long-context Q-Former Integrated with Multimodal LLM","abstract":"Human-robot collaboration towards a shared goal requires robots to understand human action and interaction with the surrounding environment. This paper focuses on human-robot interaction (HRI) based on human-robot dialogue that relies on the robot action confirmation and action step generation using multimodal scene understanding. The state-of-the-art approach uses multimodal transformers to generate robot action steps aligned with robot action confirmation from a single clip showing a task composed of multiple micro steps. Although actions towards a long-horizon task depend on each other throughout an entire video, the current approaches mainly focus on clip-level processing and do not leverage long-context information. This paper proposes a long-context Q-former incorporating left and right context dependency in full videos. Furthermore, this paper proposes a text-conditioning approach to feed text embeddings directly into the LLM decoder to mitigate the high abstraction of the information in text by Q-former. Experiments with the YouCook2 corpus show that the accuracy of confirmation generation is a major factor in the performance of action planning. Furthermore, we demonstrate that the long-context Q-former improves the confirmation and action planning by integrating VideoLLaMA3.","short_abstract":"Human-robot collaboration towards a shared goal requires robots to understand human action and interaction with the surrounding environment. This paper focuses on human-robot interaction (HRI) based on human-robot dialogue that relies on the robot action confirmation and action step generation using multimodal scene un...","url_abs":"https://arxiv.org/abs/2511.17335","url_pdf":"https://arxiv.org/pdf/2511.17335v1","authors":"[\"Chiori Hori\",\"Yoshiki Masuyama\",\"Siddarth Jain\",\"Radu Corcodel\",\"Devesh Jha\",\"Diego Romeres\",\"Jonathan Le Roux\"]","published":"2025-11-21T15:55:25Z","proceeding":"cs.RO","tasks":"[\"cs.RO\",\"cs.CL\",\"cs.CV\",\"cs.SD\",\"eess.AS\"]","methods":"[\"Transformer\",\"Large Language Model\"]","has_code":false}