{"ID":2895302,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2508.00852","arxiv_id":"2508.00852","title":"Visuo-Acoustic Hand Pose and Contact Estimation","abstract":"Accurately estimating hand pose and hand-object contact events is essential for robot data-collection, immersive virtual environments, and biomechanical analysis, yet remains challenging due to visual occlusion, subtle contact cues, limitations in vision-only sensing, and the lack of accessible and flexible tactile sensing. We therefore introduce VibeMesh, a novel wearable system that fuses vision with active acoustic sensing for dense, per-vertex hand contact and pose estimation. VibeMesh integrates a bone-conduction speaker and sparse piezoelectric microphones, distributed on a human hand, emitting structured acoustic signals and capturing their propagation to infer changes induced by contact. To interpret these cross-modal signals, we propose a graph-based attention network that processes synchronized audio spectra and RGB-D-derived hand meshes to predict contact with high spatial resolution. We contribute: (i) a lightweight, non-intrusive visuo-acoustic sensing platform; (ii) a cross-modal graph network for joint pose and contact inference; (iii) a dataset of synchronized RGB-D, acoustic, and ground-truth contact annotations across diverse manipulation scenarios; and (iv) empirical results showing that VibeMesh outperforms vision-only baselines in accuracy and robustness, particularly in occluded or static-contact settings.","short_abstract":"Accurately estimating hand pose and hand-object contact events is essential for robot data-collection, immersive virtual environments, and biomechanical analysis, yet remains challenging due to visual occlusion, subtle contact cues, limitations in vision-only sensing, and the lack of accessible and flexible tactile sen...","url_abs":"https://arxiv.org/abs/2508.00852","url_pdf":"https://arxiv.org/pdf/2508.00852v1","authors":"[\"Yuemin Mao\",\"Uksang Yoo\",\"Yunchao Yao\",\"Shahram Najam Syed\",\"Luca Bondi\",\"Jonathan Francis\",\"Jean Oh\",\"Jeffrey Ichnowski\"]","published":"2025-07-13T20:43:24Z","proceeding":"cs.HC","tasks":"[\"cs.HC\",\"cs.CV\",\"cs.LG\",\"cs.RO\"]","methods":"[]","has_code":false}