A conference poster presenting a method for dynamic scene representation in urban areas using 3D Gaussians and neural fields, addressing limitations of static 3D Gaussian Splatting.
Paper title: Dynamic 3D Gaussian Fields for Urban Areas Abstract: A conference poster presenting a method for dynamic scene representation in urban areas using 3D Gaussians and neural fields, addressing limitations of static 3D Gaussian Splatting. Paper body (method & results): <!DOCTYPE html> <html lang="en"> <head> <meta content="text/html; charset=utf-8" http-equiv="content-type"/> <title>Dynamic 3D Gaussian Fields for Urban Areas</title> <!--Generated on Wed Jun 5 10:21:42 2024 by LaTeXML (version 0.8.8) http://dlmf.nist.gov/LaTeXML/.--> <meta content="width=device-width, initial-scale=1, shrink-to-fit=no" name="viewport"/> <link href="https://cdn.jsdelivr.net/npm/bootstrap@5.3.0/dist/css/bootstrap.min.css" rel="stylesheet" type="text/css"/> <link href="/static/browse/0.3.4/css/ar5iv.0.7.9.min.css" rel="stylesheet" type="text/css"/> <link href="/static/browse/0.3.4/css/ar5iv-fonts.0.7.9.min.css" rel="stylesheet" type="text/css"/> <link href="/static/browse/0.3.4/css/latexml_styles.css" rel="stylesheet" type="text/css"/> <script src="https://cdn.jsdelivr.net/npm/bootstrap@5.3.0/dist/js/bootstrap.bundle.min.js"></script> <script src="https://cdnjs.cloudflare.com/ajax/libs/html2canvas/1.3.3/html2canvas.min.js"></script> <script src="/static/browse/0.3.4/js/addons_new.js"></script> <script src="/static/browse/0.3.4/js/feedbackOverlay.js"></script> <base href="/html/2406.03175v1/"/></head> <body> <nav class="ltx_page_navbar"> <nav class="ltx_TOC"> <ol class="ltx_toclist"> <li class="ltx_tocentry ltx_tocentry_section"><a class="ltx_ref" href="https://arxiv.org/html/2406.03175v1#S1" title="In Dynamic 3D Gaussian Fields for Urban Areas"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">1 </span>Introduction</span></a></li> <li class="ltx_tocentry ltx_tocentry_section"><a class="ltx_ref" href="https://arxiv.org/html/2406.03175v1#S2" title="In Dynamic 3D Gaussian Fields for Urban Areas"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">2 </span>Related Work</span></a></li> <li class="ltx_tocentry ltx_tocentry_section"> <a class="ltx_ref" href="https://arxiv.org/html/2406.03175v1#S3" title="In Dynamic 3D Gaussian Fields for Urban Areas"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">3 </span>Method</span></a> <ol class="ltx_toclist ltx_toclist_section"> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2406.03175v1#S3.SS1" title="In 3 Method ‣ Dynamic 3D Gaussian Fields for Urban Areas"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">3.1 </span>Problem setup</span></a></li> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2406.03175v1#S3.SS2" title="In 3 Method ‣ Dynamic 3D Gaussian Fields for Urban Areas"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">3.2 </span>Representation</span></a></li> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2406.03175v1#S3.SS3" title="In 3 Method ‣ Dynamic 3D Gaussian Fields for Urban Areas"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">3.3 </span>Composition and Rendering</span></a></li> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2406.03175v1#S3.SS4" title="In 3 Method ‣ Dynamic 3D Gaussian Fields for Urban Areas"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">3.4 </span>Optimization</span></a></li> </ol> </li> <li class="ltx_tocentry ltx_tocentry_section"> <a class="ltx_ref" href="https://arxiv.org/html/2406.03175v1#S4" title="In Dynamic 3D Gaussian Fields for Urban Areas"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">4 </span>Experiments</span></a> <ol class="ltx_toclist ltx_toclist_section"> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2406.03175v1#S4.SS1" title="In 4 Experiments ‣ Dynamic 3D Gaussian Fields for Urban Areas"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">4.1 </span>Comparison to State-of-the-Art</span></a></li> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2406.03175v1#S4.SS2" title="In 4 Experiments ‣ Dynamic 3D Gaussian Fields for Urban Areas"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">4.2 </span>Ablation Studies</span></a></li> </ol> </li> <li class="ltx_tocentry ltx_tocentry_section"><a class="ltx_ref" href="https://arxiv.org/html/2406.03175v1#S5" title="In Dynamic 3D Gaussian Fields for Urban Areas"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">5 </span>Conclusion</span></a></li> <li class="ltx_tocentry ltx_tocentry_section"><a class="ltx_ref" href="https://arxiv.org/html/2406.03175v1#S6" title="In Dynamic 3D Gaussian Fields for Urban Areas"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">6 </span>Acknowledgements</span></a></li> <li class="ltx_tocentry ltx_tocentry_appendix"> <a class="ltx_ref" href="https://arxiv.org/html/2406.03175v1#A1" title="In Dynamic 3D Gaussian Fields for Urban Areas"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">A </span>Appendix</span></a> <ol class="ltx_toclist ltx_toclist_appendix"> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2406.03175v1#A1.SS1" title="In Appendix A Appendix ‣ Dynamic 3D Gaussian Fields for Urban Areas"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">A.1 </span>Demonstration Video</span></a></li> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2406.03175v1#A1.SS2" title="In Appendix A Appendix ‣ Dynamic 3D Gaussian Fields for Urban Areas"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">A.2 </span>Method</span></a></li> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2406.03175v1#A1.SS3" title="In Appendix A Appendix ‣ Dynamic 3D Gaussian Fields for Urban Areas"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">A.3 </span>Experiments</span></a></li> </ol> </li> </ol></nav> </nav> <div class="ltx_page_main"> <div class="ltx_page_content"> <article class="ltx_document ltx_authors_1line" lang="en"> <h1 class="ltx_title ltx_title_document">Dynamic 3D Gaussian Fields for Urban Areas</h1> <div class="ltx_authors"> <span class="ltx_creator ltx_role_author"> <span class="ltx_personname">Tobias Fischer<sup class="ltx_sup" id="id11.10.id1"><span class="ltx_text ltx_font_italic" id="id11.10.id1.1">1</span></sup> Jonas Kulhanek<sup class="ltx_sup" id="id12.11.id2"><span class="ltx_text ltx_font_italic" id="id12.11.id2.1">1,3</span></sup> Samuel Rota Bulò<sup class="ltx_sup" id="id13.12.id3"><span class="ltx_text ltx_font_italic" id="id13.12.id3.1">2</span></sup> Lorenzo Porzi<sup class="ltx_sup" id="id14.13.id4"><span class="ltx_text ltx_font_italic" id="id14.13.id4.1">2</span></sup> <br class="ltx_break"/><span class="ltx_text ltx_font_bold" id="id5.5.1">Marc Pollefeys<sup class="ltx_sup" id="id5.5.1.1"><span class="ltx_text ltx_font_medium ltx_font_italic" id="id5.5.1.1.1">1</span></sup></span> <span class="ltx_text ltx_font_bold" id="id6.6.2">Peter Kontschieder<sup class="ltx_sup" id="id6.6.2.1"><span class="ltx_text ltx_font_medium ltx_font_italic" id="id6.6.2.1.1">2</span></sup></span> <br class="ltx_break"/> <sup class="ltx_sup" id="id15.14.id5"><span class="ltx_text ltx_font_italic" id="id15.14.id5.1">1</span></sup> ETH Zürich <sup class="ltx_sup" id="id16.15.id6"><span class="ltx_text ltx_font_italic" id="id16.15.id6.1">2</span></sup> Meta Reality Labs <sup class="ltx_sup" id="id17.16.id7"><span class="ltx_text ltx_font_italic" id="id17.16.id7.1">3</span></sup> CTU Prague <br class="ltx_break"/><a class="ltx_ref ltx_url ltx_font_typewriter" href="https://tobiasfshr.github.io/pub/4dgf/" style="font-size:90%;" title="">https://tobiasfshr.github.io/pub/4dgf/</a><span class="ltx_text" id="id18.17.id8" style="font-size:90%;"> </span> </span></span> </div> <div class="ltx_abstract"> <h6 class="ltx_title ltx_title_abstract">Abstract</h6> <p class="ltx_p" id="id10.1"><span class="ltx_text" id="id10.1.1">We present an efficient neural 3D scene representation for novel-view synthesis (NVS) in large-scale, dynamic urban areas. Existing works are not well suited for applications like mixed-reality or closed-loop simulation due to their limited visual quality and non-interactive rendering speeds. Recently, rasterization-based approaches have achieved high-quality NVS at impressive speeds. However, these methods are limited to small-scale, <em class="ltx_emph ltx_font_italic" id="id10.1.1.1">homogeneous</em> data, <em class="ltx_emph ltx_font_italic" id="id10.1.1.2">i.e</em>.<span class="ltx_text" id="id10.1.1.3"></span> they cannot handle severe appearance and geometry variations due to weather, season, and lighting and do not scale to larger, dynamic areas with thousands of images. We propose 4DGF, a neural scene representation that scales to large-scale <em class="ltx_emph ltx_font_italic" id="id10.1.1.4">dynamic</em> urban areas, handles <em class="ltx_emph ltx_font_italic" id="id10.1.1.5">heterogeneous</em> input data, and substantially improves rendering speeds. We use 3D Gaussians as an efficient geometry scaffold while relying on neural fields as a compact and flexible appearance model. We integrate scene dynamics via a scene graph at global scale while modeling articulated motions on a local level via deformations. This decomposed approach enables flexible scene composition suitable for real-world applications. In experiments, we surpass the state-of-the-art by over 3 dB in PSNR and more than <math alttext="200\times" class="ltx_math_unparsed" display="inline" id="id10.1.1.m1.1"><semantics id="id10.1.1.m1.1a"><mrow id="id10.1.1.m1.1b"><mn id="id10.1.1.m1.1.1">200</mn><mo id="id10.1.1.m1.1.2" lspace="0.222em">×</mo></mrow><annotation encoding="application/x-tex" id="id10.1.1.m1.1c">200\times</annotation><annotation encoding="application/x-llamapun" id="id10.1.1.m1.1d">200 ×</annotation></semantics></math> in rendering speed.</span></p> </div> <figure class="ltx_figure" id="S0.F1"><img alt="Refer to caption" class="ltx_graphics ltx_centering ltx_img_landscape" height="308" id="S0.F1.g1" src="x1.png" width="830"/> <figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span class="ltx_text" id="S0.F1.9.1.1" style="font-size:90%;">Figure 1</span>: </span><span class="ltx_text ltx_font_bold" id="S0.F1.10.2" style="font-size:90%;">Summary.<span class="ltx_text ltx_font_medium" id="S0.F1.10.2.1"> Given a set of <em class="ltx_emph ltx_font_italic" id="S0.F1.10.2.1.1">heterogeneous</em> input sequences that capture a common geographic area in varying environmental conditions (<em class="ltx_emph ltx_font_italic" id="S0.F1.10.2.1.2">e.g</em>.<span class="ltx_text" id="S0.F1.10.2.1.3"></span> weather, season, and lighting) with distinct dynamic objects (<em class="ltx_emph ltx_font_italic" id="S0.F1.10.2.1.4">e.g</em>.<span cl