A conference poster detailing methods to tune frequency bias in State Space Models (SSMs) through initialization and training dynamics, including theoretical analysis and experimental results on long-range benchmarks.
Paper title: Tuning Frequency Bias of State Space Models Abstract: A conference poster detailing methods to tune frequency bias in State Space Models (SSMs) through initialization and training dynamics, including theoretical analysis and experimental results on long-range benchmarks. Paper body (method & results): <!DOCTYPE html> <html lang="en"> <head> <meta content="text/html; charset=utf-8" http-equiv="content-type"/> <title>Tuning Frequency Bias of State Space Models</title> <!--Generated on Wed Oct 2 21:02:43 2024 by LaTeXML (version 0.8.8) http://dlmf.nist.gov/LaTeXML/.--> <meta content="width=device-width, initial-scale=1, shrink-to-fit=no" name="viewport"/> <link href="https://cdn.jsdelivr.net/npm/bootstrap@5.3.0/dist/css/bootstrap.min.css" rel="stylesheet" type="text/css"/> <link href="/static/browse/0.3.4/css/ar5iv.0.7.9.min.css" rel="stylesheet" type="text/css"/> <link href="/static/browse/0.3.4/css/ar5iv-fonts.0.7.9.min.css" rel="stylesheet" type="text/css"/> <link href="/static/browse/0.3.4/css/latexml_styles.css" rel="stylesheet" type="text/css"/> <script src="https://cdn.jsdelivr.net/npm/bootstrap@5.3.0/dist/js/bootstrap.bundle.min.js"></script> <script src="https://cdnjs.cloudflare.com/ajax/libs/html2canvas/1.3.3/html2canvas.min.js"></script> <script src="/static/browse/0.3.4/js/addons_new.js"></script> <script src="/static/browse/0.3.4/js/feedbackOverlay.js"></script> <base href="/html/2410.02035v1/"/></head> <body> <nav class="ltx_page_navbar"> <nav class="ltx_TOC"> <ol class="ltx_toclist"> <li class="ltx_tocentry ltx_tocentry_section"><a class="ltx_ref" href="https://arxiv.org/html/2410.02035v1#S1" title="In Tuning Frequency Bias of State Space Models"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">1 </span>Introduction</span></a></li> <li class="ltx_tocentry ltx_tocentry_section"><a class="ltx_ref" href="https://arxiv.org/html/2410.02035v1#S2" title="In Tuning Frequency Bias of State Space Models"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">2 </span>What is the Frequency Bias of an SSM?</span></a></li> <li class="ltx_tocentry ltx_tocentry_section"><a class="ltx_ref" href="https://arxiv.org/html/2410.02035v1#S3" title="In Tuning Frequency Bias of State Space Models"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">3 </span>Frequency Bias of an SSM at Initialization</span></a></li> <li class="ltx_tocentry ltx_tocentry_section"><a class="ltx_ref" href="https://arxiv.org/html/2410.02035v1#S4" title="In Tuning Frequency Bias of State Space Models"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">4 </span>Frequency Bias of an SSM during Training</span></a></li> <li class="ltx_tocentry ltx_tocentry_section"> <a class="ltx_ref" href="https://arxiv.org/html/2410.02035v1#S5" title="In Tuning Frequency Bias of State Space Models"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">5 </span>Tuning the Frequency Bias of an SSM</span></a> <ol class="ltx_toclist ltx_toclist_section"> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2410.02035v1#S5.SS1" title="In 5 Tuning the Frequency Bias of an SSM ‣ Tuning Frequency Bias of State Space Models"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">5.1 </span>Tuning Frequency Bias by Scaling the Initialization</span></a></li> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2410.02035v1#S5.SS2" title="In 5 Tuning the Frequency Bias of an SSM ‣ Tuning Frequency Bias of State Space Models"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">5.2 </span>Tuning Frequency Bias by a Sobolev Filter</span></a></li> </ol> </li> <li class="ltx_tocentry ltx_tocentry_section"><a class="ltx_ref" href="https://arxiv.org/html/2410.02035v1#S6" title="In Tuning Frequency Bias of State Space Models"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">6 </span>Experiments and Discussions</span></a></li> <li class="ltx_tocentry ltx_tocentry_section"><a class="ltx_ref" href="https://arxiv.org/html/2410.02035v1#S7" title="In Tuning Frequency Bias of State Space Models"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">7 </span>Conclusion</span></a></li> <li class="ltx_tocentry ltx_tocentry_appendix"><a class="ltx_ref" href="https://arxiv.org/html/2410.02035v1#A1" title="In Tuning Frequency Bias of State Space Models"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">A </span>Proofs</span></a></li> <li class="ltx_tocentry ltx_tocentry_appendix"><a class="ltx_ref" href="https://arxiv.org/html/2410.02035v1#A2" title="In Tuning Frequency Bias of State Space Models"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">B </span>Functional Derivatives</span></a></li> <li class="ltx_tocentry ltx_tocentry_appendix"><a class="ltx_ref" href="https://arxiv.org/html/2410.02035v1#A3" title="In Tuning Frequency Bias of State Space Models"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">C </span>Scaling Laws of the Initialization</span></a></li> <li class="ltx_tocentry ltx_tocentry_appendix"><a class="ltx_ref" href="https://arxiv.org/html/2410.02035v1#A4" title="In Tuning Frequency Bias of State Space Models"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">D </span>More Numerical Experiments on the Illustrative Example</span></a></li> <li class="ltx_tocentry ltx_tocentry_appendix"> <a class="ltx_ref" href="https://arxiv.org/html/2410.02035v1#A5" title="In Tuning Frequency Bias of State Space Models"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">E </span>Details of the Experiments</span></a> <ol class="ltx_toclist ltx_toclist_appendix"> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2410.02035v1#A5.SS1" title="In Appendix E Details of the Experiments ‣ Tuning Frequency Bias of State Space Models"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">E.1 </span>Denoising Sequential Autoencoder</span></a></li> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2410.02035v1#A5.SS2" title="In Appendix E Details of the Experiments ‣ Tuning Frequency Bias of State Space Models"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">E.2 </span>Long-Range Arena</span></a></li> </ol> </li> <li class="ltx_tocentry ltx_tocentry_appendix"> <a class="ltx_ref" href="https://arxiv.org/html/2410.02035v1#A6" title="In Tuning Frequency Bias of State Space Models"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">F </span>Supplementary Experiments</span></a> <ol class="ltx_toclist ltx_toclist_appendix"> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2410.02035v1#A6.SS1" title="In Appendix F Supplementary Experiments ‣ Tuning Frequency Bias of State Space Models"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">F.1 </span>Predict the Magnitudes of Waves</span></a></li> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2410.02035v1#A6.SS2" title="In Appendix F Supplementary Experiments ‣ Tuning Frequency Bias of State Space Models"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">F.2 </span>Tuning Frequency Bias in Moving MNIST Video Prediction</span></a></li> </ol> </li> </ol></nav> </nav> <div class="ltx_page_main"> <div class="ltx_page_content"> <article class="ltx_document ltx_authors_1line"> <h1 class="ltx_title ltx_title_document">Tuning Frequency Bias of State Space Models</h1> <div class="ltx_authors"> <span class="ltx_creator ltx_role_author"> <span class="ltx_personname"> Annan Yu </span><span class="ltx_author_notes"> <span class="ltx_contact ltx_role_affiliation"> </span></span></span> <span class="ltx_creator ltx_role_author"> <span class="ltx_personname"><sup class="ltx_sup" id="id14.2.id1"><span class="ltx_text ltx_font_italic" id="id14.2.id1.1">1</span></sup> Dongwei Lyu </span><span class="ltx_author_notes">Corresponding author: <a class="ltx_ref ltx_url ltx_font_typewriter" href="ay262@cornell.edu" title="">ay262@cornell.edu</a>. <span class="ltx_contact ltx_role_affiliation"> </span></span></span> <span class="ltx_creator ltx_role_author"> <span class="ltx_personname"><sup class="ltx_sup" id="id15.2.id1"><span class="ltx_text ltx_font_italic" id="id15.2.id1.1">2</span></sup> Soon Hoe Lim </span><span class="ltx_author_notes"> <span class="ltx_contact ltx_role_affiliation"> </span></span></span> <span class="ltx_creator ltx_role_author"> <span class="ltx_personname"><sup class="ltx_sup" id="id16.2.id1"><span class="ltx_text ltx_font_italic" id="id16.2.id1.1">3,4</span></sup> <br class="ltx_break"/>Michael W. Mahoney </span><span class="ltx_author_notes"> <span class="ltx_contact ltx_role_affiliation"> </span></span></span> <span class="ltx_creator ltx_role_author"> <span class="ltx_personname"><sup class="ltx_sup" id="id17.4.id1"><span class="ltx_text ltx_font_italic" id="id17.4.id1.1">5,6,7</span></sup> N. Benjamin Erichson<sup class="ltx_sup" id="id18.5.id2"><span class="ltx_text ltx_font_italic" id="id18.5.id2.1">5,6</span></sup> <br class="ltx_break"/><sup class="ltx_sup" id="id19.6.id3">1</sup> Center for Applied Mathematics </span><span class="ltx_author_notes"> <span class="ltx_contact ltx_role_affiliation"> </span></span></span> <span class="ltx_creator ltx_role_author"> <span class="ltx_personname"> Cornell University <br class="ltx_break"/><sup class="ltx_sup" id="id20.2.id1">2</sup> Data Science Institute </span><span class="ltx_author_notes"> <span class="ltx_contact ltx_role_affiliation"> </span></span></span> <span class="ltx_creator ltx_role_author"> <span class="ltx_personname"> University of Chicago <br class="ltx_break"/><sup class="ltx_sup" id="id21.2.id1">3</sup> Department of Mathematics </span><span class="ltx_author_notes"> <span class="ltx_contact ltx_role_affiliation"> </span></span></span> <span class="ltx_creator ltx_role_author"> <span class="ltx_personname"> KTH Royal Institute of Technology <br class="ltx_break"/><sup class="ltx_sup" id="id22.2.id1">4</sup> Nordita </span><span class="ltx_author_notes"> <span class="ltx_contact ltx_role_affiliation"> </span></span></span> <span class="ltx_creator ltx_role_author"> <span class="ltx_personname"> KTH Royal Institute of Technology and Stockholm University <br class="ltx_break"/><sup class="ltx_sup" id="id23.4.id1">5</sup> Lawrence Berkeley National Laboratory <br class="ltx_break"/><sup class="ltx_sup" id="id24.5.id2">6</sup> International Computer Science Institute <br class="ltx_break"/><sup class="ltx_sup" id="id25.6.id3">7</sup> Department of Statistics </span><span class="ltx_author_notes"> <span class="ltx_contact ltx_role_affiliation"> </span></span></span> <span class="ltx_creator ltx_role_author"> <span class="ltx_personname"> University of California at Berkeley </span><span class="ltx_author_notes"> <span class="ltx_contact ltx_role_affiliation"> </span></span></span> </div> <div class="ltx_abstract"> <h6 cl