{"ID":2837928,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2511.18319","arxiv_id":"2511.18319","title":"Weakly-supervised Latent Models for Task-specific Visual-Language Control","abstract":"Autonomous inspection in hazardous environments requires AI agents that can interpret high-level goals and execute precise control. A key capability for such agents is spatial grounding, for example when a drone must center a detected object in its camera view to enable reliable inspection. While large language models provide a natural interface for specifying goals, using them directly for visual control achieves only 58\\% success in this task. We envision that equipping agents with a world model as a tool would allow them to roll out candidate actions and perform better in spatially grounded settings, but conventional world models are data and compute intensive. To address this, we propose a task-specific latent dynamics model that learns state-specific action-induced shifts in a shared latent space using only goal-state supervision. The model leverages global action embeddings and complementary training losses to stabilize learning. In experiments, our approach achieves 71\\% success and generalizes to unseen images and instructions, highlighting the potential of compact, domain-specific latent dynamics models for spatial alignment in autonomous inspection.","short_abstract":"Autonomous inspection in hazardous environments requires AI agents that can interpret high-level goals and execute precise control. A key capability for such agents is spatial grounding, for example when a drone must center a detected object in its camera view to enable reliable inspection. While large language models...","url_abs":"https://arxiv.org/abs/2511.18319","url_pdf":"https://arxiv.org/pdf/2511.18319v1","authors":"[\"Xian Yeow Lee\",\"Lasitha Vidyaratne\",\"Gregory Sin\",\"Ahmed Farahat\",\"Chetan Gupta\"]","published":"2025-11-23T07:18:28Z","proceeding":"cs.AI","tasks":"[\"cs.AI\",\"cs.LG\",\"eess.SY\"]","methods":"[\"Language Model\"]","has_code":false}