{"ID":2846221,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2511.02560","arxiv_id":"2511.02560","title":"SigmaCollab: An Application-Driven Dataset for Physically Situated Collaboration","abstract":"We introduce SigmaCollab, a dataset enabling research on physically situated human-AI collaboration. The dataset consists of a set of 85 sessions in which untrained participants were guided by a mixed-reality assistive AI agent in performing procedural tasks in the physical world. SigmaCollab includes a set of rich, multimodal data streams, such as the participant and system audio, egocentric camera views from the head-mounted device, depth maps, head, hand and gaze tracking information, as well as additional annotations performed post-hoc. While the dataset is relatively small in size (~ 14 hours), its application-driven and interactive nature brings to the fore novel research challenges for human-AI collaboration, and provides more realistic testing grounds for various AI models operating in this space. In future work, we plan to use the dataset to construct a set of benchmarks for physically situated collaboration in mixed-reality task assistive scenarios. SigmaCollab is available at https://github.com/microsoft/SigmaCollab.","short_abstract":"We introduce SigmaCollab, a dataset enabling research on physically situated human-AI collaboration. The dataset consists of a set of 85 sessions in which untrained participants were guided by a mixed-reality assistive AI agent in performing procedural tasks in the physical world. SigmaCollab includes a set of rich, mu...","url_abs":"https://arxiv.org/abs/2511.02560","url_pdf":"https://arxiv.org/pdf/2511.02560v1","authors":"[\"Dan Bohus\",\"Sean Andrist\",\"Ann Paradiso\",\"Nick Saw\",\"Tim Schoonbeek\",\"Maia Stiber\"]","published":"2025-11-04T13:30:15Z","proceeding":"cs.HC","tasks":"[\"cs.HC\",\"cs.AI\",\"cs.CV\"]","methods":"[]","has_code":false,"code_links":[{"ID":607419,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_id":2846221,"paper_url":"https://arxiv.org/abs/2511.02560","paper_title":"SigmaCollab: An Application-Driven Dataset for Physically Situated Collaboration","repo_url":"https://github.com/microsoft/SigmaCollab","is_official":false,"mentioned_in_paper":false,"mentioned_in_github":true,"github_stars":0}]}