HumanGaussian Text-to-3D Human Generation — Poster

Paper context

Paper title: HumanGaussian: Text-Driven 3D Human Generation with Gaussian Splatting Abstract: A conference poster presenting HumanGaussian, a framework for generating high-quality 3D humans from text prompts using Gaussian Splatting, featuring structure-aware SDS and annealed negative guidance. Paper body (method & results): HumanGaussian: Text-Driven 3D Human Generation with Gaussian Splatting Xian Liu1, Xiaohang Zhan2, Jiaxiang Tang3, Ying Shan2, Gang Zeng3, Dahua Lin1, Xihui Liu4, Ziwei Liu5 1CUHK 2Tencent AI Lab 3PKU 4HKU 5NTU Project Page: https://alvinliu0.github.io/projects/HumanGaussian A boy with a beanie wearing a hoodie and joggers. A body builder wearing a tanktop. A Viking. A Black woman wearing sunglasses, a white t-shirt and jeans. A Texas ranger. Figure 1. We propose HumanGaussian, an efficient yet effective framework that generates high-quality 3D humans with fine-grained geometry and realistic appearance. Our method adapts 3D Gaussian Splatting into text-driven 3D human generation with novel designs. arXiv:2311.17061v2 [cs.CV] 14 Mar 2024 Abstract Realistic 3D human generation from text prompts is a de- sirable yet challenging task. Existing methods optimize 3D representations like mesh or neural fields via score distil- lation sampling (SDS), which suffers from inadequate fine details or excessive training time. In this paper, we pro- pose an efficient yet effective framework, HumanGaussian, that generates high-quality 3D humans with fine-grained geometry and realistic appearance. Our key insight is that 3D Gaussian Splatting is an efficient renderer with peri- odic Gaussian shrinkage or growing, where such adaptive density control can be naturally guided by intrinsic human structures. Specifically, 1) we first propose a Structure- Aware SDS that simultaneously optimizes human appear- ance and geometry. The multi-modal score function from both RGB and depth space is leveraged to distill the Gaus- sian densification and pruning process. 2) Moreover, we devise an Annealed Negative Prompt Guidance by decom- posing SDS into a noisier generative score and a cleaner classifier score, which well addresses the over-saturation issue. The floating artifacts are further eliminated based on Gaussian size in a prune-only phase to enhance generation smoothness. Extensive experiments demonstrate the supe- rior efficiency and competitive quality of our framework, rendering vivid 3D humans under diverse scenarios. 1. Introduction Creating high-quality 3D humans from user condition is of great importance to a wide variety of applications, ranging from virtual try-on [29, 69, 70, 78] to immersive telepres- ence [27, 28, 39, 42, 67, 98]. To this end, researchers ex- plore the task of text-driven 3D human generation, which synthesizes the character’s appearance and geometry based on text prompts. Traditional methods resort to a hand- crafted pipeline, where 3D models are first regressed from multi-view human captures, and then undergo a series of manual processes like rigging and skinning [3, 26, 35, 38]. To ease human labor for 3D asset creation of diverse lay- outs, the exemplar work DreamFusion [59] proposes score distillation sampling (SDS) to harness rich 2D text-to-image prior (e.g., Stable Diffusion [65], Imagen [66]) by optimiz- ing 3D scenes to render samples that reside on the manifold of higher likelihood. Though accomplishing reasonable re- sults on single objects [6, 50, 64, 83], it is hard for them to model detailed human bodies with complex articulations. To incorporate structural guidance, recent text-driven 3D human studies combine SDS with body shape models such as SMPL [46] and imGHUM [1]. In particular, a common paradigm is to integrate human priors into representations like mesh and neural radiance field (NeRF), either by taking the body shape as mesh/density initialization [20, 36, 93], or by learning a deformation field based on linear blend skinning (LBS) [4, 85, 92]. However, they mostly compro- mise to trade-off between efficiency and quality: the mesh- based methods [25, 41, 90] struggle to model fine topolo- gies like accessories and wrinkles; while the NeRF-based methods [22, 23, 94] are time/memory-consuming to render high-resolution results. How to achieve fine-grained gener- ation efficiently remains an unsolved problem. Recently, the explicit neural representation of 3D Gaus- sian Splatting (3DGS) [33] provides a new perspective for real-time scene reconstruction. It enables multi-scale mod- eling across multiple granularities, which is suitable for 3D human generation. Nevertheless, it is non-trivial to ex- ploit such representation in this task with two challenges: 1) 3DGS characterizes a tile-based rasterization by sorting and α-blending anisotropic splats within each view frustum, which only back-propagates a small set of high-confidence Gaussians. However, as verified in the 3D surface-/volume- rendering studies [17, 31, 52, 76], sparse gradient could hinder network optimization of geometry and appearance. Therefore, structural guidance is required in 3DGS, es- pecially for the human domain that demands hierarchi- cal structure modeling and generation controllability. 2) The naive SDS necessitates a large classifier-free guidance (CFG) [18] scale for image-text alignment (e.g., 100 as used in [59]). But it sacrifices visual quality with over-saturated patterns, making realistic human generation difficult. Be- sides, due to the stochasticity of SDS loss, the original gradient-based density control in 3DGS is unstable, which incurs blurry results with floating artifacts. In this paper, we propose an efficient yet effective frame- work, HumanGaussian, that generates high-quality 3D hu- mans with fine-grained geometry and realistic appearance. Our intuition lies in that 3D Gaussian Splatting is an effi- cient renderer with periodic Gaussian shrinkage or growing, where such adaptive density control can be naturally guided by intrinsic human structures. The key is to incorporate explicit structural guidance and gradient regularization to facilitate Gaussian optimization. Specifically, we first pro- pose a Structure-Aware SDS that jointly learns human ap- pearance and geometry. Unlike previous studies [7, 33, 89] that adopt the generic priors like Structure-from-Motion (SfM) points or Point-E [54], we instead anchor the Gaus- sian initial positions on the SMPL-X mesh. The subsequent densification and pruning processes thus focus on regions around the body surface, effectively capturing geometric deformations like accessories and wrinkles. Additionally, we extend the pre-trained Stable Diffusion [65] to simul- taneously denoise the image RGB and depth as our SDS source model. Such dual-branch design distills the joint distribution of two spatially-aligned targets (i.e., RGB and depth), which boosts the Gaussian convergence with both structural guidance and textural realism. To further improve renderings with natural appearance, we devise an Annealed Negative Prompt Guidance. In particular, we decompose SDS into a noisier generative score and a cleaner classi- fier score, where equipping the latter term with a decreas- ing negative prompt guidance enables realistic generation under nominal CFG scales (e.g., 7.5), as also proven in con- current text-to-3D studies [32, 91]. In this way, we man- age to avoid over-saturated patterns with appropriate CFG scales that well balance sample quality and diversity. More- over, due to the high variance of SDS loss, directly relying on gradient information to control densities results in blurry geometry [33]. In contrast, we propose to eliminate floating artifacts based on Gaussian size in a prune-only phase. To summarize, our main contributions are three-fold: 1) We propose an efficient yet effective framework Hu- manGaussian for high-quality 3D human generation with fine-grained geometry and realistic appearance. As one of the earliest attempts in taming Gaussian Splatting for text- driven 3D human domain, we hope to pave the way for fu- ture research. 2) We propose the Structure-Aware SDS to jointly learn human appearance and geometry with explicit structural guidance. 3) We devise the Annealed Negative Prompt Guidance to guarantee realistic results and elimi- nate floating artifacts. Extensive experiments demonstrate the superior efficiency and competitive quality of our frame- work, rendering vivid 3D humans under diverse scenarios. 2. Related Work 3D Neural Representations. Diverse 3D scene representa- tions are proposed for spatial geometry and texture model- ing, such as voxel, point cloud, mesh, and neural field. With the trade-off among training time, memory efficiency, ren- dering capability, and network compatibility, different rep- resentations are chosen based on problem setting: 1) Voxel, a Euclidean representation that stores scene information in a grid manner [9, 49, 87], can be easily adapted for CNNs, but is limited in render resolution due to the cubic com- putational cost. 2) Point cloud, a discrete point set sam- pled from 3D surface, is efficient to render [40, 60, 61]. However, it fails to capture the fine-grained details due to its discontinuous nature. 3) Mesh, a compact represen- tation expressing the connectivity among vertices, edges, and faces, inherits time efficiency from the well-rounded graphic pipelines [16, 81, 84], but struggles to create accu- rate topology. 4) Neural field, an implicit function of each 3D position’s attributes, is capable of modeling complex structures in arbitrary resolution [44, 52, 56, 86, 88], yet the optimization and inference are slow. Recently, 3D Gaussian Splatting (3DGS) [33, 48] has shown impressive results in the 3D reconstruction, surpassing previous representations with better quality and faster convergence. In this work, we try to unlock the potential of 3D Gaussian splatting on the challenging task of text-driven 3D human generation. Text-to-3D Generation. Recent diffusion-based text-to- 3D works can be grouped into two types: 1) 3D native pipelines, which directly capture the distribution of 3D data [30, 47, 54] or the reconstructed intermediate fea- tures [5, 8, 15, 55] on specific domains. Although some recent works [2, 21] extend the model’s capacity by train- ing on large-scale 3D datasets like Objaverse [10], they are still confined to single objects. 2) Optimization-based 2D lifting pipelines, which optimize 3D scene representa- tions in a per-prompt manner by distilling from rich prior learned in the 2D domain. For example, some early at- tempts use CLIP guidance [62] to boost multi-view image- text alignment [24, 51, 53, 79], while recent methods re- sort to score distillation sampling (SDS) to inherit unprece- dented rendering quality from exemplar text-to-image mod- els [6, 50, 59, 80, 83]. Notably, the heavy computation burden of NeRF incurs long training time, which motivates concurrent works to adapt the representation of Gaussian splatting for text-to-3D generation [7, 77, 89]. In this work, we choose 3D Gaussian due to its efficiency and efficacy, but focus on the text-driven 3D human domain that demands both fine detail capturing and realistic texture generation. Text-Driven 3D Human Generation. By incorporating human prior like SMPL [46] and imGHUM [1]