{"ID":5438639,"CreatedAt":"2026-07-01T01:17:58.482524686Z","UpdatedAt":"2026-07-03T04:20:05.427450767Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2606.31158","arxiv_id":"2606.31158","title":"LLM-Powered Interactive Robotic Action Synthesis from Multimodal Speech, Gestures, and Music","abstract":"The quest for intuitive and natural human-robot interaction (HRI) remains a significant challenge in robotics. Traditional methods often rely on rigid, pre-programmed commands that limit the robot's expressiveness and adaptability. This paper introduces a novel framework that leverages the reasoning capabilities of Large Language Models (LLMs) to synthesize complex robotic actions from a rich tapestry of multimodal human inputs: natural speech, hand gestures, and music/sound beats. Our system architecture integrates a speech transcription model, a gesture recognition module, and a signal processing pipeline for beat detection. These processed inputs are contextualized using prompt templates and fed into a LLM. The LLM, informed by a predefined robot action space, reasons over the combined inputs to generate a coherent sequence of actions. This sequence is dispatched to an action queue for execution on a quadruped robot over ROS. The framework has ability to interpret and fuse semantic commands from speech, deictic information from gestures, and rhythmic cues from music. This work represents a step towards creating robots that can interact with humans in a more fluid, creative, and context-aware manner.","short_abstract":"The quest for intuitive and natural human-robot interaction (HRI) remains a significant challenge in robotics. Traditional methods often rely on rigid, pre-programmed commands that limit the robot's expressiveness and adaptability. This paper introduces a novel framework that leverages the reasoning capabilities of Lar...","url_abs":"https://arxiv.org/abs/2606.31158","url_pdf":"https://arxiv.org/pdf/2606.31158v1","authors":"[\"Snehasis Banerjee\",\"Ranjan Dasgupta\"]","published":"2026-06-30T05:35:26Z","proceeding":"cs.RO","tasks":"[\"cs.RO\",\"cs.AI\"]","methods":"[\"Large Language Model\",\"Language Model\"]","has_code":false}