LucidDreamer Text-to-3D with ISM — Poster

Paper context

Paper title: LucidDreamer: Towards High-Fidelity Text-to-3D Generation via Interval Score Matching Abstract: A CVPR 2024 poster presenting LucidDreamer, a text-to-3D generation method using Interval Score Matching (ISM) to fix SDS over-smoothing, featuring methodology diagrams, ablation studies on Lego castles, and results like Hatsune Miku. Paper body (method & results): LucidDreamer: Towards High-Fidelity Text-to-3D Generation via Interval Score Matching Yixun Liang*1 Xin Yang*1,2 Jiantao Lin1 Haodong Li1 Xiaogang Xu3,4 Yingcong Chen∗∗1,2 1 HKUST (GZ) 2 HKUST 3 Zhejiang Lab 4 Zhejiang University yliang982@connect.hkust-gz.edu.cn xin.yang@connect.ust.hk jlin695@hkust-gz.edu.cn hli736@connect.hkust-gz.edu.cn xgxu@zhejianglab.com yingcongchen@ust.hk “A warrior with red cape riding a horse” “A beautiful cyborg with brown hair” “A forbidden castle high up in the mountains” “A highly-detailed sandcastle” “A portrait of Hatsune Miku, robot” “An armored green-skin orc warrior riding a vicious hog” “A DSLR photo of a football helmet.” “ Viking axe, fantasy, weapon, blender, 8k, HD. ” “Gandalf smiling, white hair, head, HDR, photorealistic,8K.” “A portrait of IRONMAN, white hair , head, HDR, photorealistic,8K.” “A DSLR photo of a bagel filled with cream cheese and lox.” “ A blue motorcycle.” “A Supercar made out of toy bricks.” “ A delicious hamburger.” “'Zombie JOKER, head, HDR, photorealistic,8K.” “Saber from Fate stay Night, 3D, girl, anime” “A pyramid shaped burrito with a slice cut out of it.” “ A DSLR photo of a LV handbag” Figure 1. Examples of text-to-3D content creations with our framework. We present a text-to-3D generation framework, named the LucidDreamer, to distill high-fidelity textures and shapes from pretrained 2D diffusion models (detailed shows on Sec. 4) with a novel Interval Score Matching objective and an Advanced 3D distillation pipeline. Together, we achieve superior 3D generation results with photorealistic quality in a short training time. Please zoom in for details. ** Corresponding author. *The first two authors contributed equally to this work. * Conceptualization: Yixun Liang: 60%, Xin Yang: 40%, * Methodology: Xin Yang: 60%, Yixun Liang: 40%. 1 arXiv:2311.11284v3 [cs.CV] 2 Dec 2023 Abstract The recent advancements in text-to-3D generation mark a significant milestone in generative models, unlocking new possibilities for creating imaginative 3D assets across var- ious real-world scenarios. While recent advancements in text-to-3D generation have shown promise, they often fall short in rendering detailed and high-quality 3D models. This problem is especially prevalent as many methods base them- selves on Score Distillation Sampling (SDS). This paper identifies a notable deficiency in SDS, that it brings incon- sistent and low-quality updating direction for the 3D model, causing the over-smoothing effect. To address this, we pro- pose a novel approach called Interval Score Matching (ISM). ISM employs deterministic diffusing trajectories and utilizes interval-based score matching to counteract over-smoothing. Furthermore, we incorporate 3D Gaussian Splatting into our text-to-3D generation pipeline. Extensive experiments show that our model largely outperforms the state-of-the-art in quality and training efficiency. Our code will be available at: EnVision-Research/LucidDreamer 1. Introduction Digital 3D asserts have become indispensable in our digital age, enabling the visualization, comprehension, and interac- tion with complex objects and environments that mirror our real-life experiences. Their impact spans a wide range of do- mains including architecture, animation, gaming, virtual and augmented reality, and is widely used in retail, online confer- encing, education, etc. The extensive use of 3D technologies brings a significant challenge, i.e., generating high-quality 3D content is a process that needs a lot of time, effort, and skilled expertise. This stimulates the rapid developments of 3D content gen- eration approaches [5, 14, 16, 21–24, 29, 31, 34, 35, 41, 47]. Among them, text-to-3D generation [5, 14, 21, 29, 31, 34, 47, 52] stands out for its ability to create imaginative 3D models from mere text descriptions. This is achieved by utilizing a pretrained text-to-image diffusion model as a strong im- age prior to supervise the training of a neural parameterized 3D model, enabling for rendering 3D consistent images in alignment with the text. This remarkable capability is funda- mentally grounded in the use of Score Distillation Sampling (SDS). SDS acts as the core mechanism that lifts 2D results from diffusion models to the 3D world, enabling the training of 3D models without images [4, 5, 16, 21, 29, 34, 50]. Despite its popularity, empirical observations have shown that SDS often encounters issues such as over-smoothing, which significantly hampers the practical application of high- fidelity 3D generation. In this paper, we thoroughly investi- gate the underlying cause of this problem. Specifically, we reveal that the mechanism behind SDS is to match the images 𝑥! = 𝑔(𝜃, 𝑐) pseudo-ground-truth 𝑥)! " 𝑥" denoise + noise SDS update direction average 𝜖! 𝜖" 𝜖# 𝜖$ noises “A Lego Porsche car” “A Tudor style house” “An icecream” Figure 2. Examples of SDS [34]. Let t = 500, we simulate the SDS distillation process by sampling xt with same x0 but different noises {ϵ1, ..., ϵ4}. We discover that the SDS distillation process produces overly-smoothed pseudo-ground-truth (i.e., ˆxt 0) for x0. First, the random noise and timestep sampling strategy of SDS drives x0 towards the averaged ˆxt 0 and eventually leads to the “feature-averaging” result. Second, SDS exploits the diffusion model for ˆxt 0 estimation in one step, which results in low-quality guidance at large timesteps. Please refer to Sec. 3.1 for more analysis. rendered by the 3D model with the pseudo-Ground-Truth (pseudo-GT) generated by the diffusion model. However, as shown in Fig. 2, the generated pseudo-GTs are usually inconsistent and have low visual quality. Consequently, all update directions provided by these pseudo-GTs are subse- quently applied to the same 3D model. Due to the average effect, the final results tend to be over-smooth and lack of details. This paper aims to overcome the aforementioned limita- tions. We show that the unsatisfactory pseudo-GTs origi- nated from two aspects. Firstly, these pseudo-GTs are one- step reconstruction results from the diffusion models, which have high reconstruction errors. Besides, the intrinsic ran- domness in the diffusion trajectory makes these pseudo-GTs semantically variant, which causes an averaging effect and eventually leads to over-smoothing results. To address these issues, we propose a novel approach called Interval Score Matching (ISM). ISM improves SDS with two effective mechanisms. Firstly, by employing DDIM inversion, ISM produces an invertible diffusion trajectory and mitigates the averaging effect caused by pseudo-GT inconsistency. Sec- ondly, rather than matching the pseudo-GTs with images rendered by the 3D model, ISM conducts matching between two interval steps in the diffusion trajectory, which avoids one-step reconstruction that yields high reconstruction error. We show that our ISM loss consistently outperforms SDS by a large margin with highly realistic and detailed results. Finally, we also show that our ISM is not only compatible with the original 3D model introduced in [34], by utilizing a more advanced model – 3D Gaussian Splatting [20], our 2 model achieves superior results compared to the state-of- the-art approaches, including Magic3D [21], Fantasia3D [5], and ProlificDreamer [47]. Notably, these competitors re- quire multi-stage training, which is not needed in our model. This not only reduces our training cost but also maintains a simple training pipeline. Overall, our contributions can be summarized as follows. • We provide an in-depth analysis of Score Distillation Sam- pling (SDS), the fundamental component in text-to-3D generation, and identify its key limitations for providing inconsistent and low-quality pseudo-GTs. This provides an explanation of the over-smoothing effect that exists in many approaches. • In response to SDS’s limitations, we propose the Inter- val Score Matching (ISM). With invertible diffusion tra- jectories and interval-based matching, ISM significantly outperforms SDS with highly realistic and detailed results. • By integrating with 3D Gaussian Splatting, our model achieves state-of-the-art performance, surpassing existing methods with less training costs. 2. Related Works Text-to-3D Generation. One work can be categorized as text-to-3D generation [2, 5–7, 12, 17, 21, 30, 34, 38, 39, 41, 45, 48]. As a pioneer, DreamField [17] firstly train NeRF [32] with CLIP [37] guidance to achieve text-to-3D distillation. However, the results is unsatisfactory due to the weak supervision from CLIP loss. With the advance of diffu- sion model, Dreamfusion [34] introduces Score Distillation Sampling (SDS) to distill 3D assets from pre-trained 2D text- to-image diffusion models. SDS facilitates 3D distillation by seeking specific modes in a text-guide diffusion model, allowing for training a 3D model based on the 2D knowledge of diffusion models. This quickly motivates a great number of following works [5, 16, 21, 30, 34, 36, 50] and becomes a critical integration of them. These works improve the perfor- mance of text-to-3D in various ways. For example, some of them [5, 12, 21, 30, 45, 48] improve the visual quality of text- to-3D distillation via modifying NeRF or introducing other advanced 3D representations. The other some [2, 6, 41] fo- cus on addressing the Janus problems, e.g., MVDream [41] propose to fine-tune the pre-trained diffusion models to make it 3D aware, and GSGEN [6] proposes a novel approach by introducing a 3D diffusion model for joint optimization. However, all these methods heavily rely on the Score Dis- tillation Sampling. Albeit promising, SDS has shown over- smoothing effects in a lot of literatures [21, 31, 34, 50]. Besides, it need coupling with a large conditional guidance scale [12], leading to over-saturation results. There are also some very recent works [18, 47, 49, 52] target at improving SDS. ProlificDreamer [47] proposes VSD to model 3D rep- resentation as a distribution. HiFA [52] propose a iterative to estimate a better sampling direction. Although signifi- cant improve has been made, these works require a much longer training stage. CSD [49] and NFSD [18] are two concurrent works that analyze the components in the SDS to obtain empirical solutions to improve the original SDS. Our work is intrinsically different in the sense that it provides a systematic analysis on the the inconsistency and low-quality pseudo-ground-truths in SDS. And by introducing the In- terval Score Matching, it achieves superior results without increasing the computational burden. Differentiable 3D Representations. Differentiable 3D rep- resentation is a crucial integration of text-guided 3D genera- tion. Given a 3D representation with trainable parameter θ, a differentiable rendering equation g(θ, c) is used to render an image in camera pose c of that 3D representation. As pro- cess is differentiable, we could train the 3D representation to fit our condition with backpropag