LumiX Text-to-Intrinsic Map Generation Overview

Paper context

Paper title: LumiX: Structured and Coherent Text-to-Intrinsic Generation Abstract: We present LumiX, a structured diffusion framework for coherent text-to-intrinsic generation. Conditioned on text prompts, LumiX jointly generates a comprehensive set of intrinsic maps (e.g., albedo, irradiance, normal, depth, and final color), providing a structured and physically consistent description of an underlying scene. This is enabled by two key contributions: 1) Query-Broadcast Attention, a mechanism that ensures structural consistency by sharing queries across all maps in each self-attention block. 2) Tensor LoRA, a tensor-based adaptation that parameter-efficiently models cross-map relations for efficient joint training. Together, these designs enable stable joint diffusion training and unified generation of multiple intrinsic properties. Experiments show that LumiX produces coherent and physically meaningful results, achieving 23% higher alignment and a better preference score (0.19 vs. -0.41) compared to the state of the art, and it can also perform image-conditioned intrinsic decomposition within the same framework. Paper body excerpt: LumiX: Structured and Coherent Text-to-Intrinsic Generation LumiX : Structured and Coherent Text-to-Intrinsic Generation Xu Han 1,2⋄ Biao Zhang 2 Xiangjun Tang 2 Xianzhi Li 1 † Peter Wonka 2 † 1 HUST 2 KAUST Abstract We present LumiX , a structured diffusion framework for coherent text-to-intrinsic generation. Conditioned on text prompts, LumiX jointly generates a comprehensive set of intrinsic maps ( e.g ., albedo, irradiance, normal, depth, and final color), providing a structured and physically consistent description of an underlying scene. This is enabled by two key contributions: 1) Query-Broadcast Attention, a mechanism that ensures structural consistency by sharing queries across all maps in each self-attention block. 2) Tensor LoRA, a tensor-based adaptation that parameter-efficiently models cross-map relations for efficient joint training. Together, these designs enable stable joint diffusion training and unified generation of multiple intrinsic properties. Experiments show that LumiX produces coherent and physically meaningful results, achieving 23% higher alignment and a better preference score (0.19 vs. -0.41) compared to the state of the art, and it can also perform image-conditioned intrinsic decomposition within the same framework. † † footnotetext: † Corresponding authors. † † footnotetext: ⋄ This work was done during Xu Han’s internship at KAUST. Figure 2 : Overview of LumiX . Our goal is to generate a coherent set of intrinsic maps from text. Left: Training. Multiple intrinsic images are encoded into the latent space and concatenated along the batch dimension. We introduce Query-Broadcast Attention ( Sec. 4.1 ) to ensure pixel alignment across properties, and Tensor LoRA ( Sec. 4.2 ) to efficiently finetune the K ​ V KV projections for each property. Different timesteps are assigned to different properties for flexible conditioning. Right: Inference. Given a text or image input, LumiX jointly outputs all intrinsic maps in a single forward pass, supporting both text-to-intrinsic generation and intrinsic decomposition. 1 Introduction Recent advances in text-to-image diffusion models [ rombach2022high ] have made it possible to generate realistic and detailed images from natural language. However, these models still produce a single RGB image and do not reveal the underlying structure of the scene, such as geometry, lighting, or material properties. For many vision and graphics tasks, a single image is not enough. We often need a structured