A conference poster presenting a method to animate single-subject sketches using text prompts and text-to-video priors, detailing the representation, pipeline, network architecture, and comparisons.
Paper title: Breathing Life Into Sketches Using Text-to-Video Priors Abstract: A conference poster presenting a method to animate single-subject sketches using text prompts and text-to-video priors, detailing the representation, pipeline, network architecture, and comparisons. Paper body (method & results): <!DOCTYPE html> <html lang="en"> <head> <meta content="text/html; charset=utf-8" http-equiv="content-type"/> <title>Breathing Life Into Sketches Using Text-to-Video Priors</title> <!--Generated on Tue Nov 21 15:14:09 2023 by LaTeXML (version 0.8.7) http://dlmf.nist.gov/LaTeXML/.--> <meta content="width=device-width, initial-scale=1, shrink-to-fit=no" name="viewport"/> <link href="https://cdn.jsdelivr.net/npm/bootstrap@5.3.0/dist/css/bootstrap.min.css" rel="stylesheet" type="text/css"/> <link href="/static/browse/0.3.4/css/ar5iv_0.7.4.min.css" rel="stylesheet" type="text/css"/> <link href="/static/browse/0.3.4/css/latexml_styles.css" rel="stylesheet" type="text/css"/> <script src="https://cdn.jsdelivr.net/npm/bootstrap@5.3.0/dist/js/bootstrap.bundle.min.js"></script> <script src="https://cdnjs.cloudflare.com/ajax/libs/html2canvas/1.3.3/html2canvas.min.js"></script> <script src="/static/browse/0.3.4/js/addons.js"></script> <script src="/static/browse/0.3.4/js/feedbackOverlay.js"></script> <base href="/html/2311.13608v1/"/></head> <body> <nav class="ltx_page_navbar"> <nav class="ltx_TOC"> <ol class="ltx_toclist"> <li class="ltx_tocentry ltx_tocentry_section"><a class="ltx_ref" href="#S1" title="1 Introduction ‣ Breathing Life Into Sketches Using Text-to-Video Priors"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">1 </span>Introduction</span></a></li> <li class="ltx_tocentry ltx_tocentry_section"> <a class="ltx_ref" href="#S2" title="2 Previous Work ‣ Breathing Life Into Sketches Using Text-to-Video Priors"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">2 </span>Previous Work</span></a> <ol class="ltx_toclist ltx_toclist_section"> <li class="ltx_tocentry ltx_tocentry_paragraph"><a class="ltx_ref" href="#S2.SS0.SSS0.Px1" title="Sketches ‣ 2 Previous Work ‣ Breathing Life Into Sketches Using Text-to-Video Priors"><span class="ltx_text ltx_ref_title">Sketches</span></a></li> <li class="ltx_tocentry ltx_tocentry_paragraph"><a class="ltx_ref" href="#S2.SS0.SSS0.Px2" title="Sketch-based animation ‣ 2 Previous Work ‣ Breathing Life Into Sketches Using Text-to-Video Priors"><span class="ltx_text ltx_ref_title">Sketch-based animation</span></a></li> <li class="ltx_tocentry ltx_tocentry_paragraph"><a class="ltx_ref" href="#S2.SS0.SSS0.Px3" title="Text-to-video generation ‣ 2 Previous Work ‣ Breathing Life Into Sketches Using Text-to-Video Priors"><span class="ltx_text ltx_ref_title">Text-to-video generation</span></a></li> <li class="ltx_tocentry ltx_tocentry_paragraph"><a class="ltx_ref" href="#S2.SS0.SSS0.Px4" title="Image-to-video generation ‣ 2 Previous Work ‣ Breathing Life Into Sketches Using Text-to-Video Priors"><span class="ltx_text ltx_ref_title">Image-to-video generation</span></a></li> </ol> </li> <li class="ltx_tocentry ltx_tocentry_section"> <a class="ltx_ref" href="#S3" title="3 Preliminaries ‣ Breathing Life Into Sketches Using Text-to-Video Priors"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">3 </span>Preliminaries</span></a> <ol class="ltx_toclist ltx_toclist_section"> <li class="ltx_tocentry ltx_tocentry_paragraph"><a class="ltx_ref" href="#S3.SS0.SSS0.Px1" title="Vector representation ‣ 3 Preliminaries ‣ Breathing Life Into Sketches Using Text-to-Video Priors"><span class="ltx_text ltx_ref_title">Vector representation</span></a></li> <li class="ltx_tocentry ltx_tocentry_paragraph"><a class="ltx_ref" href="#S3.SS0.SSS0.Px2" title="Score-Distillation Sampling ‣ 3 Preliminaries ‣ Breathing Life Into Sketches Using Text-to-Video Priors"><span class="ltx_text ltx_ref_title">Score-Distillation Sampling</span></a></li> </ol> </li> <li class="ltx_tocentry ltx_tocentry_section"> <a class="ltx_ref" href="#S4" title="4 Method ‣ Breathing Life Into Sketches Using Text-to-Video Priors"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">4 </span>Method</span></a> <ol class="ltx_toclist ltx_toclist_section"> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="#S4.SS1" title="4.1 Representation ‣ 4 Method ‣ Breathing Life Into Sketches Using Text-to-Video Priors"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">4.1 </span>Representation</span></a></li> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="#S4.SS2" title="4.2 Text-Driven Optimization ‣ 4 Method ‣ Breathing Life Into Sketches Using Text-to-Video Priors"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">4.2 </span>Text-Driven Optimization</span></a></li> <li class="ltx_tocentry ltx_tocentry_subsection"> <a class="ltx_ref" href="#S4.SS3" title="4.3 Neural Displacement Field ‣ 4 Method ‣ Breathing Life Into Sketches Using Text-to-Video Priors"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">4.3 </span>Neural Displacement Field</span></a> <ol class="ltx_toclist ltx_toclist_subsection"> <li class="ltx_tocentry ltx_tocentry_paragraph"><a class="ltx_ref" href="#S4.SS3.SSS0.Px1" title="Shared backbone ‣ 4.3 Neural Displacement Field ‣ 4 Method ‣ Breathing Life Into Sketches Using Text-to-Video Priors"><span class="ltx_text ltx_ref_title">Shared backbone</span></a></li> <li class="ltx_tocentry ltx_tocentry_paragraph"><a class="ltx_ref" href="#S4.SS3.SSS0.Px2" title="Local path ‣ 4.3 Neural Displacement Field ‣ 4 Method ‣ Breathing Life Into Sketches Using Text-to-Video Priors"><span class="ltx_text ltx_ref_title">Local path</span></a></li> <li class="ltx_tocentry ltx_tocentry_paragraph"><a class="ltx_ref" href="#S4.SS3.SSS0.Px3" title="Global path ‣ 4.3 Neural Displacement Field ‣ 4 Method ‣ Breathing Life Into Sketches Using Text-to-Video Priors"><span class="ltx_text ltx_ref_title">Global path</span></a></li> </ol> </li> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="#S4.SS4" title="4.4 Training Details ‣ 4 Method ‣ Breathing Life Into Sketches Using Text-to-Video Priors"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">4.4 </span>Training Details</span></a></li> </ol> </li> <li class="ltx_tocentry ltx_tocentry_section"> <a class="ltx_ref" href="#S5" title="5 Results ‣ Breathing Life Into Sketches Using Text-to-Video Priors"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">5 </span>Results</span></a> <ol class="ltx_toclist ltx_toclist_section"> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="#S5.SS1" title="5.1 Comparisons ‣ 5 Results ‣ Breathing Life Into Sketches Using Text-to-Video Priors"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">5.1 </span>Comparisons</span></a></li> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="#S5.SS2" title="5.2 Ablation Study ‣ 5 Results ‣ Breathing Life Into Sketches Using Text-to-Video Priors"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">5.2 </span>Ablation Study</span></a></li> </ol> </li> <li class="ltx_tocentry ltx_tocentry_section"><a class="ltx_ref" href="#S6" title="6 Limitations ‣ Breathing Life Into Sketches Using Text-to-Video Priors"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">6 </span>Limitations</span></a></li> <li class="ltx_tocentry ltx_tocentry_section"><a class="ltx_ref" href="#S7" title="7 Conclusions ‣ Breathing Life Into Sketches Using Text-to-Video Priors"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">7 </span>Conclusions</span></a></li> <li class="ltx_tocentry ltx_tocentry_section"><a class="ltx_ref" href="#S8" title="8 Acknowledgements ‣ Breathing Life Into Sketches Using Text-to-Video Priors"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">8 </span>Acknowledgements</span></a></li> <li class="ltx_tocentry ltx_tocentry_part"> <a class="ltx_ref" href="#Pt1" title="Breathing Life Into Sketches Using Text-to-Video Priors"><span class="ltx_text ltx_ref_title"></span> ‣ <span class="ltx_text ltx_ref_title">Breathing Life Into Sketches Using Text-to-Video Priors</span></a> <ol class="ltx_toclist ltx_toclist_part"> <li class="ltx_tocentry ltx_tocentry_appendix"><a class="ltx_ref" href="#A1" title="Appendix A Additional results and videos ‣ Breathing Life Into Sketches Using Text-to-Video Priors"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref"><span class="ltx_text" style="font-size:144%;">A</span> </span><span class="ltx_text" style="font-size:144%;">Additional results and videos</span></span></a></li> <li class="ltx_tocentry ltx_tocentry_appendix"> <a class="ltx_ref" href="#A2" title="Appendix B Analysis and ablation ‣ Breathing Life Into Sketches Using Text-to-Video Priors"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref"><span class="ltx_text" style="font-size:144%;">B</span> </span><span class="ltx_text" style="font-size:144%;">Analysis and ablation</span></span></a> <ol class="ltx_toclist ltx_toclist_appendix"> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="#A2.SS1" title="B.1 Text prompt effect ‣ Appendix B Analysis and ablation ‣ Breathing Life Into Sketches Using Text-to-Video Priors"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">B.1 </span>Text prompt effect</span></a></li> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="#A2.SS2" title="B.2 Different levels of abstraction ‣ Appendix B Analysis and ablation ‣ Breathing Life Into Sketches Using Text-to-Video Priors"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">B.2 </span>Different levels of abstraction</span></a></li> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="#A2.SS3" title="B.3 Sketch representation ‣ Appendix B Analysis and ablation ‣ Breathing Life Into Sketches Using Text-to-Video Priors"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">B.3 </span>Sketch representation</span></a></li> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="#A2.SS4" title="B.4 Learning rate scaling and tradeoffs ‣ Appendix B Analysis and ablation ‣ Breathing Life Into Sketches Using Text-to-Video Priors"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">B.4 </span>Learning rate scaling and tradeoffs</span></a></li> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="#A2.SS5" title="B.5 Hyperparameter effects ‣ Appendix B Analysis and ablation ‣ Breathing Life Into Sketches Using Text-to-Video Priors"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">B.5 </span>Hyperparameter effects</span></a></li> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="#A2.SS6" title="B.6 Other text-to-video backbones ‣ Appendix B Analysis and ablation ‣ Breathing Life Into Sketches Using Text-to-Video Priors"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">B.6 </span>Other text-to-video backbones</span>