LORS Method for Efficient Network Stacking Poster
A conference poster detailing the LORS method for efficient network stacking, featuring framework diagrams, stacking paradigms, experimental results on DeiT encoders, pseudo-code implementations, and feature visualizations.
论文上下文
Paper title: LORS: Low-rank Residual Structure for Parameter-Efficient Network Stacking Abstract: A conference poster detailing the LORS method for efficient network stacking, featuring framework diagrams, stacking paradigms, experimental results on DeiT encoders, pseudo-code implementations, and feature visualizations. Paper body (method & results): <!DOCTYPE html> <html lang="en"> <head> <meta content="text/html; charset=utf-8" http-equiv="content-type"/> <title>LORS: Low-rank Residual Structure for Parameter-Efficient Network Stacking</title> <!--Generated on Thu Mar 7 07:58:13 2024 by LaTeXML (version 0.8.7) http://dlmf.nist.gov/LaTeXML/.--> <meta content="width=device-width, initial-scale=1, shrink-to-fit=no" name="viewport"/> <link href="https://cdn.jsdelivr.net/npm/bootstrap@5.3.0/dist/css/bootstrap.min.css" rel="stylesheet" type="text/css"/> <link href="/static/browse/0.3.4/css/ar5iv_0.7.4.min.css" rel="stylesheet" type="text/css"/> <link href="/static/browse/0.3.4/css/latexml_styles.css" rel="stylesheet" type="text/css"/> <script src="https://cdn.jsdelivr.net/npm/bootstrap@5.3.0/dist/js/bootstrap.bundle.min.js"></script> <script src="https://cdnjs.cloudflare.com/ajax/libs/html2canvas/1.3.3/html2canvas.min.js"></script> <script src="/static/browse/0.3.4/js/addons.js"></script> <script src="/static/browse/0.3.4/js/feedbackOverlay.js"></script> <base href="/html/2403.04303v1/"/></head> <body> <nav class="ltx_page_navbar"> <nav class="ltx_TOC"> <ol class="ltx_toclist"> <li class="ltx_tocentry ltx_tocentry_section"><a class="ltx_ref" href="https://arxiv.org/html/2403.04303v1#S1" title="1 Introduction ‣ LORS: Low-rank Residual Structure for Parameter-Efficient Network Stacking"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">1 </span>Introduction</span></a></li> <li class="ltx_tocentry ltx_tocentry_section"><a class="ltx_ref" href="https://arxiv.org/html/2403.04303v1#S2" title="2 Related Work ‣ LORS: Low-rank Residual Structure for Parameter-Efficient Network Stacking"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">2 </span>Related Work</span></a></li> <li class="ltx_tocentry ltx_tocentry_section"> <a class="ltx_ref" href="https://arxiv.org/html/2403.04303v1#S3" title="3 Approach ‣ LORS: Low-rank Residual Structure for Parameter-Efficient Network Stacking"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">3 </span>Approach</span></a> <ol class="ltx_toclist ltx_toclist_section"> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2403.04303v1#S3.SS1" title="3.1 Preliminary ‣ 3 Approach ‣ LORS: Low-rank Residual Structure for Parameter-Efficient Network Stacking"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">3.1 </span>Preliminary</span></a></li> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2403.04303v1#S3.SS2" title="3.2 Formulation of Our Method ‣ 3 Approach ‣ LORS: Low-rank Residual Structure for Parameter-Efficient Network Stacking"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">3.2 </span>Formulation of Our Method</span></a></li> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2403.04303v1#S3.SS3" title="3.3 Applying LORS to AdaMixer’s Decoders ‣ 3 Approach ‣ LORS: Low-rank Residual Structure for Parameter-Efficient Network Stacking"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">3.3 </span>Applying LORS to AdaMixer’s Decoders</span></a></li> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2403.04303v1#S3.SS4" title="3.4 Analysis on Parameter Reduction ‣ 3 Approach ‣ LORS: Low-rank Residual Structure for Parameter-Efficient Network Stacking"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">3.4 </span>Analysis on Parameter Reduction</span></a></li> </ol> </li> <li class="ltx_tocentry ltx_tocentry_section"> <a class="ltx_ref" href="https://arxiv.org/html/2403.04303v1#S4" title="4 Experiments ‣ LORS: Low-rank Residual Structure for Parameter-Efficient Network Stacking"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">4 </span>Experiments</span></a> <ol class="ltx_toclist ltx_toclist_section"> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2403.04303v1#S4.SS1" title="4.1 Implementation Details ‣ 4 Experiments ‣ LORS: Low-rank Residual Structure for Parameter-Efficient Network Stacking"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">4.1 </span>Implementation Details</span></a></li> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2403.04303v1#S4.SS2" title="4.2 Main Results ‣ 4 Experiments ‣ LORS: Low-rank Residual Structure for Parameter-Efficient Network Stacking"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">4.2 </span>Main Results</span></a></li> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2403.04303v1#S4.SS3" title="4.3 Ablation Study ‣ 4 Experiments ‣ LORS: Low-rank Residual Structure for Parameter-Efficient Network Stacking"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">4.3 </span>Ablation Study</span></a></li> </ol> </li> <li class="ltx_tocentry ltx_tocentry_section"><a class="ltx_ref" href="https://arxiv.org/html/2403.04303v1#S5" title="5 Conclusion ‣ LORS: Low-rank Residual Structure for Parameter-Efficient Network Stacking"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">5 </span>Conclusion</span></a></li> </ol></nav> </nav> <div class="ltx_page_main"> <div class="ltx_page_content"> <div aria-label="Conversion errors have been found" class="package-alerts ltx_document" role="status"> <button aria-label="Dismiss alert" onclick="closePopup()"> <span aria-hidden="true"><svg aria-hidden="true" focusable="false" height="20" role="presentation" viewbox="0 0 44 44" width="20"> <path d="M0.549989 4.44999L4.44999 0.549988L43.45 39.55L39.55 43.45L0.549989 4.44999Z"></path> <path d="M39.55 0.549988L43.45 4.44999L4.44999 43.45L0.549988 39.55L39.55 0.549988Z"></path> </svg></span> </button> <p>HTML conversions <a href="https://info.dev.arxiv.org/about/accessibility_html_error_messages.html" target="_blank">sometimes display errors</a> due to content that did not convert correctly from the source. This paper uses the following packages that are not yet supported by the HTML conversion tool. Feedback on these issues are not necessary; they are known and are being worked on.</p> <ul arial-label="Unsupported packages used in this paper"> <li>failed: epic</li> </ul> <p>Authors: achieve the best HTML results from your LaTeX submissions by following these <a href="https://info.arxiv.org/help/submit_latex_best_practices.html" target="_blank">best practices</a>.</p> </div><div class="section" id="target-section"><div id="license-tr">License: arXiv.org perpetual non-exclusive license</div><div id="watermark-tr">arXiv:2403.04303v1 [cs.CV] 07 Mar 2024</div></div> <script> function closePopup() { document.querySelector('.package-alerts').style.display = 'none'; } </script> <article class="ltx_document ltx_authors_1line"> <h1 class="ltx_title ltx_title_document">LORS: Low-rank Residual Structure for Parameter-Efficient Network Stacking</h1> <div class="ltx_authors"> <span class="ltx_creator ltx_role_author"> <span class="ltx_personname">Jialin Li, Qiang Nie, Weifu Fu, Yuhuan Lin, Guangpin Tao, Yong Liu, Chengjie Wang <br class="ltx_break"/>Youtu Lab, Tencent <br class="ltx_break"/><math alttext="\{" class="ltx_Math" display="inline" id="id1.1.m1.1"><semantics id="id1.1.m1.1a"><mo id="id1.1.m1.1.1" maxsize="90%" minsize="90%" xref="id1.1.m1.1.1.cmml">{</mo><annotation-xml encoding="MathML-Content" id="id1.1.m1.1b"><ci id="id1.1.m1.1.1.cmml" xref="id1.1.m1.1.1">{</ci></annotation-xml><annotation encoding="application/x-tex" id="id1.1.m1.1c">\{</annotation><annotation encoding="application/x-llamapun" id="id1.1.m1.1d">{</annotation></semantics></math><span class="ltx_text ltx_font_typewriter" id="id2.2.1" style="font-size:90%;">jarenli, ryanwfu, gleelin, guangpintao, choasliu, jasoncjwang<math alttext="\}" class="ltx_Math" display="inline" id="id2.2.1.m1.1"><semantics id="id2.2.1.m1.1a"><mo id="id2.2.1.m1.1.1" mathvariant="normal" stretchy="false" xref="id2.2.1.m1.1.1.cmml">}</mo><annotation-xml encoding="MathML-Content" id="id2.2.1.m1.1b"><ci id="id2.2.1.m1.1.1.cmml" xref="id2.2.1.m1.1.1">normal-}</ci></annotation-xml><annotation encoding="application/x-tex" id="id2.2.1.m1.1c">\}</annotation><annotation encoding="application/x-llamapun" id="id2.2.1.m1.1d">}</annotation></semantics></math>@tencent.com, qnie.cuhk@gmail.com</span> </span></span> </div> <div class="ltx_abstract"> <h6 class="ltx_title ltx_title_abstract">Abstract</h6> <p class="ltx_p" id="id3.id1">Deep learning models, particularly those based on transformers, often employ numerous stacked structures, which possess identical architectures and perform similar functions. While effective, this stacking paradigm leads to a substantial increase in the number of parameters, posing challenges for practical applications. In today’s landscape of increasingly large models, stacking depth can even reach dozens, further exacerbating this issue. To mitigate this problem, we introduce <span class="ltx_text ltx_font_bold" id="id3.id1.1">LORS</span> (<span class="ltx_text ltx_font_bold" id="id3.id1.2">LO</span>w-rank <span class="ltx_text ltx_font_bold" id="id3.id1.3">R</span>esidual <span class="ltx_text ltx_font_bold" id="id3.id1.4">S</span>tructure). LORS allows stacked modules to share the majority of parameters, requiring a much smaller number of unique ones per module to match or even surpass the performance of using entirely distinct ones, thereby significantly reducing parameter usage. We validate our method by applying it to the stacked decoders of a query-based object detector, and conduct extensive experiments on the widely used MS COCO dataset. Experimental results demonstrate the effectiveness of our method, as even with a 70% reduction in the parameters of the decoder, our method still enables the model to achieve comparable or even better performance than its original. </p> </div> <section class="ltx_section" id="S1"> <h2 class="ltx_title ltx_title_section"> <span class="ltx_tag ltx_tag_section">1 </span>Introduction</h2> <figure class="ltx_figure" id="S1.F1"><img alt="Refer to caption" class="ltx_graphics ltx_centering ltx_img_portrait" height="615" id="S1.F1.g1" src="extracted/5454683/images/Figure1.png" width="479"/> <figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span class="ltx_text" id="S1.F1.2.1.1" style="font-size:90%;">Figure 1</span>: </span><span class="ltx_text" id="S1.F1.3.2" style="font-size:90%;">The LORS calculation process, which could be adaptive or static, depending on whether an adaptively generated kernel is used in the matrix manipulation for private parameters. </span></figcaption> </figure> <div class="ltx_para" id="S1.p1"> <p class="ltx_p" id="S1.p1.1">In
