Wonder3D Single-Image to 3D Diffusion — Poster

论文上下文

Paper title: Wonder3D: Single Image to 3D using Cross-Domain Diffusion Abstract: A CVPR 2024 poster for Wonder3D, demonstrating single-image to 3D generation via cross-domain diffusion, addressing the Janus problem, and comparing results against SyncDreamer and Shape-E with a 2-3 minute inference time. Paper body (method & results): Wonder3D: Single Image to 3D using Cross-Domain Diffusion Xiaoxiao Long1,3,6∗, Yuan-Chen Guo2,3∗, Cheng Lin1†, Yuan Liu1, Zhiyang Dou1 Lingjie Liu4, Yuexin Ma5, Song-Hai Zhang2, Marc Habermann6, Christian Theobalt6, Wenping Wang7† 1 The University of Hong Kong 2 Tsinghua University 3 VAST 4 University of Pennsylvania 5 Shanghai Tech University 6 MPI Informatik 7 Texas A&M University ∗Equal Contributions. https://www.xxlong.site/Wonder3D/ Figure 1. Wonder3D reconstructs highly-detailed textured meshes from a single-view image in only 2 ∼3 minutes. Wonder3D first generates consistent multi-view normal maps with corresponding color images via a cross-domain diffusion model, and then leverages a novel normal fusion method to achieve fast and high-quality reconstruction. † Corresponding authors. 1 arXiv:2310.15008v3 [cs.CV] 8 Nov 2023 Abstract In this work, we introduce Wonder3D, a novel method for efficiently generating high-fidelity textured meshes from single-view images. Recent methods based on Score Dis- tillation Sampling (SDS) have shown the potential to re- cover 3D geometry from 2D diffusion priors, but they typ- ically suffer from time-consuming per-shape optimization and inconsistent geometry. In contrast, certain works di- rectly produce 3D information via fast network inferences, but their results are often of low quality and lack geomet- ric details. To holistically improve the quality, consistency, and efficiency of single-view reconstruction tasks, we pro- pose a cross-domain diffusion model that generates multi- view normal maps and the corresponding color images. To ensure the consistency of generation, we employ a multi- view cross-domain attention mechanism that facilitates in- formation exchange across views and modalities. Lastly, we introduce a geometry-aware normal fusion algorithm that extracts high-quality surfaces from the multi-view 2D rep- resentations. Our extensive evaluations demonstrate that our method achieves high-quality reconstruction results, ro- bust generalization, and good efficiency compared to prior works. 1. Introduction Reconstructing 3D geometry from a single image stands as a fundamental task in computer graphics and 3D computer vision [12, 25, 31, 33, 35, 38, 41, 44], offering a wide range of versatile applications such as virtual reality, video games, 3D content creation, and the precision of robotics grasping. However, this task is notably challenging since it is ill-posed and demands the ability to discern the 3D geometry of both visible and invisible parts. This ability requires extensive knowledge of the 3D world. Recently, the field of 3D generation has experienced rapid and flourishing development with the introduction of diffusion models. A growing body of research [5, 29, 43, 59, 63], such as DreamField [24], DreamFusion [43], and Magic3D [29], resort to distilling prior knowledge of 2D image diffusion models or vision language models to create 3D models from text or images via Score Distillation Sam- pling (SDS) [43]. Despite their compelling results, these methods suffer from two main limitations: efficiency and consistency. The per-shape optimization process typically entails tens of thousands of iterations, involving full-image volume rendering and inferences of the diffusion models. Consequently, it often consumes tens of minutes or even hours on per-shape optimization. Moreover, the 2D prior model operates by considering only a single view at each iteration and strives to make every view resemble the input image. This often results in the generation of 3D shapes exhibiting inconsistencies, thus, often leading to the gen- eration of 3D shapes with inconsistencies such as multiple faces (i.e., the Janus problem [43]). There exists another group of works that endeavor to di- rectly produce 3D geometries like point clouds [37, 41, 71, 75], meshes [16, 34], neural fields [1, 4, 7, 14, 17, 21, 25– 27, 40, 42, 61, 72] via network inference to avoid time- consuming per-shape optimization. Most of them attempt to train 3D generative diffusion models from scratch on 3D assets. However, due to the limited size of publicly avail- able 3D datasets, these methods demonstrate poor general- izability, most of which can only generate shapes on spe- cific categories. More recently, several methods have emerged that di- rectly generate multi-view 2D images, with representative works including SyncDreamer [33] and MVDream [51]. By enhancing the multi-view consistency of image generation, these methods can recover 3D shapes from the generated multi-view images. Following these works, our method also adopts a multi-view generation scheme to favor the flexi- bility and efficiency of 2D representations. However, due to only relying on color images, the fidelity of the gen- erated shapes is not well-maintained, and they struggle to recover geometric details or come with enormous computa- tional costs. To better address the issues of fidelity, consistency, gen- eralizability and efficiency in the aforementioned works, in this paper, we introduce a new approach to the task of single-view 3D reconstruction by generating multi-view consistent normal maps and their corresponding color im- ages with a cross-domain diffusion model. The key idea is to extend the stable diffusion framework to model the joint distribution of two different domains, i.e., normals and colors. We demonstrate that this can be achieved by in- troducing a domain switcher and a cross-domain attention scheme. In particular, the domain switcher allows the dif- fusion model to generate either normal maps or color im- ages, while the cross-domain attention mechanisms assist in the information exchange between the two domains, ulti- mately improving consistency and quality. Finally, in order to stably extract surfaces from the generated views, we pro- pose a geometry-aware normal fusion algorithm that is ro- bust to the inaccuracies and capable of reconstructing clean and high-quality geometries (see Figure 1). We conduct extensive experiments on the Google Scanned Object dataset [13] and various 2D images with different styles. The experiments validate that Wonder3D is capable of producing high-quality geometry with high ef- ficiency in comparison with baseline methods. Wonder3D possesses several distinctive properties and accordingly has the following contributions: • Wonder3D holistically considers the issues of generation quality, efficiency, generalizability, and consistency for 2 single-view 3D reconstruction. It has achieved a lead- ing level of geometric details with reasonably good effi- ciency among current zero-shot single-view reconstruc- tion methods. • We propose a new multi-view cross-domain 2D diffusion model to predict normal maps and color images. This representation not only adapts to the original data distri- bution of Stable Diffusion model but also effectively cap- tures the rich surface details of the target shape. • We propose a cross-domain attention mechanism to pro- duce multi-view normal maps and color images that are consistently aligned. This mechanism facilitates infor- mation perception across different domains, enabling our method to recover high-fidelity geometry. • We introduce a novel geometry-aware normal fusion al- gorithm that can robustly extract surfaces from the gener- ated normal maps and color images. 2. Related Works 2.1. 2D Diffusion Models for 3D Generation Recent compelling successes in 2D diffusion models [8, 22, 47] and large vision language models (e.g., CLIP model [45]) provide new possibilities for generating 3D as- sets using the strong priors of 2D diffusion models. Pi- oneering works DreamFusion [43] and SJC [59] propose to distill a 2D text-to-image generation model to generate 3D shapes from texts, and many follow-up works follow such per-shape optimization scheme. For the task of text- to-3D [2, 5, 6, 23, 29, 48, 49, 57, 63, 65, 69, 77] or image- to-3D synthesis [38, 44, 46, 50, 54, 67], these methods typ- ically optimize a 3D representation (i.e., NeRF, mesh, or SDF), and then leverage neural rendering to generate 2D images from various viewpoints. The images are then fed into the 2D diffusion models or CLIP model for calculating SDS [43] losses, which can guide the 3D shape optimiza- tion. However, most of these methods always suffer from low efficiency and multi-face problem, where a per-shape opti- mization consumes tens of minutes and the optimized ge- ometry tends to produce multiple faces due to the lack of explicit 3D supervision. A recent work one-2-3-45 [15] proposes to leverage a generalizable neural reconstruction method SparseNeuS [36] to directly produce 3D geometry from the generated images from zero123 [31]. Although the method achieves high efficiency, its results are of low- quality and lack geometric details. 2.2. 3D Generative Models Instead of performing a time-consuming per-shape opti- mization guided by 2D diffusion models, some works at- tempt to directly train 3D diffusion models based on vari- ous 3D representations, like point clouds [37, 41, 71, 75], meshes [16, 34], neural fields [1, 4, 7, 14, 17, 21, 25– 27, 40, 42, 61, 72] However, due to the limited size of pub- lic available 3D assets dataset, most of the works have only been validated on limited categories of shapes, and how to scale up on large datasets is still an open problem. On the contrary, our method adopts 2D representations and, thus, can be built upon the 2D diffusion models [47] whose pre- trained priors significantly facilitate zero-shot generaliza- tion ability. 2.3. Multi-view Diffusion Models To generate consistent multi-view images, some efforts [3, 10, 18, 28, 32, 53, 55, 56, 58, 64, 66, 68, 70, 76] are made to extend 2D diffusion models from single-view images to multi-view images. However, most of these methods fo- cus on image generation and are not designed for 3D re- construction. The works [66, 73] first warp estimated depth maps to produce incomplete novel view images to then per- form inpainting on them, but their result quality signifi- cantly degrades when the depth maps estimated by exter- nal depth estimation models are inaccurate. The recent works Viewset Diffusion [53], SyncDreamer [33], and MV- Dream [51] share a similar idea to produce consistent multi- view color images via attention layers. However, unlike that normal maps explicitly encode geometric information, re- construction from color images always suffers from texture ambiguity, and, thus, they either struggle to recover geo- metric details or require huge computational costs. Sync- Dreamer [33] requires dense views for 3D reconstruction, but still suffers from low-quality geometry and blurring tex- tures. MVDream [51] still resorts to a time-consuming optimization using SDS loss for 3D reconstruction, and its multi-view distillation scheme requires 1.5 hours. In contrast, our method can reconstruct high-quality textured meshes in just 2 minutes. 3. Problem Fo