VistaBot Architecture for 4D Geometry and Policy Execution

论文上下文

Paper title: VistaBot: View-Robust Robot Manipulation via Spatiotemporal-Aware View Synthesis Abstract: Recently, end-to-end robotic manipulation models have gained significant attention for their generalizability and scalability. However, they often suffer from limited robustness to camera viewpoint changes when training with a fixed camera. In this paper, we propose VistaBot, a novel framework that integrates feed-forward geometric models with video diffusion models to achieve view-robust closed-loop manipulation without requiring camera calibration at test time. Our approach consists of three key components: 4D geometry estimation, view synthesis latent extraction, and latent action learning. VistaBot is integrated into both action-chunking (ACT) and diffusion-based ($π_0$) policies and evaluated across simulation and real-world tasks. We further introduce the View Generalization Score (VGS) as a new metric for comprehensive evaluation of cross-view generalization. Results show that VistaBot improves VGS by 2.79$\times$ and 2.63$\times$ over ACT and $π_0$, respectively, while also achieving high-quality novel view synthesis. Our contributions include a geometry-aware synthesis model, a latent action planner, a new benchmark metric, and extensive validation across diverse environments. The code and models will be made publicly available. Passages referencing this figure: hile VLAs aspire to unify instructions and control across diverse tasks. However, a fundamental bottleneck undermines both paradigms: poor generalization across camera viewpoints. Unlike appearance shifts such as lighting or texture, viewpoint variations disrupt the spatial grounding between perception and action. As a result, even slight changes in perspective can cause dramatic policy failures (Fig. 1 ), often forcing practitioners to re-collect demonstrations or retrain models—an outcome directly at odds with the very scalability that end-to-end frameworks claim to deliver. To address cross-view generalization, prior efforts have primarily fallen into two directions: (1) Reconstruction-based methods [ 47 , 33 ] attempt to recover the underlying 3D geometry from synchronized multi-view s cess of multi-camera calibration further complicates real-world training and deployment. (2) Video generation-based methods [ 22 , 24 ] leverage multi-view-consistent video generative models and derive actions from predicted future frames. However, these typically lack integrated action learning and suffer from low inference efficiency, making them unsuitable for closed-loop robotic manipulation. Figure 1: Our proposed VistaBot demonstrates superior cross-view generalizability compared with SOTA visuomotor policy ( π 0 \pi_{0} and ACT). As shown in the figure, when the camera observation angle undergoes significant changes, VistaBot consistently maintains a high average success rate even under substantial camera viewpoint changes, whereas the success rates of baseline policies drop to near