SeongJong Yoo*, Snehesh Shrestha*, Irina Muresanu, Cornelia Fermüller
*Equal Contribution
University of Maryland College Park
Abstract
Musicians delicately control their bodies to generate music. Sometimes, their motions are too subtle to be captured by the human eye. To analyze how they move to produce the music, we need to estimate precise 4D human pose (3D pose over time). However, current state-of-the-art (SoTA) visual pose estimation algorithms struggle to produce accurate monocular 4D poses because of occlusions, partial views, and human-object interactions. They are limited by the viewing angle, pixel density, and sampling rate of the cameras and fail to estimate fast and subtle movements, such as in the musical effect of vibrato. We leverage the direct causal relationship between the music produced and the human motions creating them to address these challenges. We propose VioPose: a novel multimodal network that hierarchically estimates dynamics. High-level features are cascaded to low-level features and integrated into Bayesian updates. Our architecture is shown to produce accurate pose sequences, facilitating precise motion analysis, and outperforms SoTA. As part of this work, we collected the largest and the most diverse calibrated violin-playing dataset, including video, sound, and 3D motion capture poses. The code and dataset for this work will be released for the benefit of the research community.
Video
Dataset
To access the dataset, please fill out the request form. The form has two parts: 1) read and accept the license agreement to get access to the VioPose dataset and 2) fill out the request form with information about the requesting party, their organization (explicitly specified and any affiliations at the time of request), the intent of usage, ethical usage of the data, take responsibility to protect the data, and take measures to directly or indirectly misuse the data. Redistribution of the downloaded data is strictly not permitted.
Acknowledgments and Disclosure
We would like to thank peers and faculties from the UMD CS department and the Perception and Robotics Group for their valuable feedback and discussions. Without their contributions and support, this work would not have been possible. Finally, the support of NSF under awards OISE 2020624 and BCS 2318255 is greatly acknowledged.
Bibtex
@inproceedings{yooshrestha2024viopose,
title = {{VioPose}: Violin Performance 4D Pose Estimation by Hierarchical Audiovisual Inference},
author = {SeongJong Yoo*, Snehesh Shrestha*, Irina Muresanu, Cornelia Fermüller},
year = {2024},
}License
VioPose and VioDat is available for non-commercial and research use only and may not be redistributed and should follow requirements under the conditions detailed on the license page. For commercial licensing or if you have any questions, please get in touch with us at snehesh@umd.edu.