Muscle-Driven Piano Playing System Overview
Figure 2 . System Overview. The whole system of our framework is trained in three stages. First, we learn a single-hand tracking policy π track \pi_{\text{track}} that outputs high-frequency muscle activations for direct muscle-driven control (Sec. 4 ). Second, we perform on-policy distillation to obtain a VAE decoder as a low-level servo for muscle control, while taking as input the well-structured latent action 𝐳 t \mathbf{z}_{t} (Sec. 5 ). Lastly, we train a piece-specific high-level controller over the latent to synthesize motions for piano playing (Sec. 6 ). While the first and third stag
论文上下文
Paper title: MUSIC: Learning Muscle-Driven Dexterous Hand Control Abstract: We present a data-driven approach for physics-based, muscle-driven dexterous control that enables musculoskeletal hands to perform precise piano playing for novel pieces of music outside the reference dataset. Our approach combines high-frequency muscle-level control with low-frequency latent-space coordination in a hierarchical architecture. At the low level, general single-hand policies are trained via reinforcement learning to generate dynamic muscle-tendon activations while tracking trajectories from a large reference motion dataset. The resulting tracking policies are then distilled into variational autoencoder (VAE) models, yielding smooth and structured latent spaces that abstract away low-level muscle dynamics. For the high level, we train piece-specific policies to operate in this latent space, coordinating bimanual motions based on specific goals, denoted by note events extracted from given musical scores, to synthesize performances beyond the reference data. In addition, we present an enhanced musculoskeletal hand model that supports fine control of fingers for accurate low-level motion tracking and diverse high-level motion synthesis. We evaluate the control pipeline of our approach on a diverse piano repertoire spanning multiple musical styles and technical demands. Results demonstrate that our approach can synthesize coordinated bimanual motions with accurate key presses, and achieve the state-of-the-art performance of piano playing in physics-based dexterous co Passages referencing this figure: hysics-based control, motion synthesis, hierarchical reinforcement learning † † submissionid: 1773 † † copyright: cc † † journal: TOG † † journalyear: 2026 † † journalvolume: 45 † † journalnumber: 4 † † publicationmonth: 7 † † doi: 10.1145/3811402 † † ccs: Computing methodologies Animation † † ccs: Computing methodologies Physical simulation † † ccs: Computing methodologies Reinforcement learning Figure 1 . Our musculoskeletal hand model (left) and diverse hand poses (right) produced by our muscle-driven motion synthesis models during piano playing. The muscle-tendon units are visualized by blue lines, with the activated ones highlighted in red. The semi-transparent shell indicates the outer skin (collision geometry) of the hand model. 1. Introduction Physically synthesizing human motion h
