An ICLR conference poster detailing DEPT, a method that decouples embeddings from transformer bodies. It features method diagrams for GLOB, TRIM, and SPEC variants, alongside tables and charts demonstrating reduced memory costs and improved generalization on The Pile and downstream tasks.
Paper title: DEPT: Decoupled Embeddings for Pre-training Language Models Abstract: An ICLR conference poster detailing DEPT, a method that decouples embeddings from transformer bodies. It features method diagrams for GLOB, TRIM, and SPEC variants, alongside tables and charts demonstrating reduced memory costs and improved generalization on The Pile and downstream tasks. Paper body (method & results): <!DOCTYPE html> <html lang="en"> <head> <meta content="text/html; charset=utf-8" http-equiv="content-type"/> <title>4 Performance metrics across the subsets of the pile when starting from a randomly-initialized embedding matrix for our 24-block model. DEPT models outperform the baselines.</title> <!--Generated on Wed Oct 22 10:53:33 2025 by LaTeXML (version 0.8.8) http://dlmf.nist.gov/LaTeXML/.--> <meta content="width=device-width, initial-scale=1, shrink-to-fit=no" name="viewport"/> <link href="/static/browse/0.3.4/css/arxiv-html-papers-20250916.css" rel="stylesheet" type="text/css"/> <script src="https://cdn.jsdelivr.net/npm/bootstrap@5.3.0/dist/js/bootstrap.bundle.min.js"></script> <script src="https://cdnjs.cloudflare.com/ajax/libs/html2canvas/1.3.3/html2canvas.min.js"></script> <script src="/static/browse/0.3.4/js/addons_new.js"></script> <script src="/static/browse/0.3.4/js/feedbackOverlay.js"></script> <base href="/html/2410.05021v1/"/></head> <body> <nav class="ltx_page_navbar"> </nav> <div class="ltx_page_main"> <div class="ltx_page_content"> <article class="ltx_document ltx_authors_1line"> <div class="ltx_para" id="p1"> <p class="ltx_p"><span class="ltx_text ltx_inline-block" style="width:345.0pt;"> <span class="ltx_inline-block ltx_transformed_outer" style="width:345.0pt;height:14.6pt;vertical-align:-6.3pt;"><span class="ltx_transformed_inner" style="transform:translate(-255.9pt,10.9pt) scale(0.402674128993276,0.402674128993276) ;"> <span class="ltx_tabular ltx_guessed_headers ltx_align_middle"> <span class="ltx_thead"> <span class="ltx_tr" style="--ltx-bg-color:#FFFFFF;"> <span class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row"><span class="ltx_text ltx_font_bold" style="--ltx-bg-color:#FFFFFF;">Name</span></span> <span class="ltx_td ltx_align_left ltx_th ltx_th_column"><span class="ltx_text ltx_font_bold" style="--ltx-bg-color:#FFFFFF;">StackExchange</span></span> <span class="ltx_td ltx_align_left ltx_th ltx_th_column"><span class="ltx_text ltx_font_bold" style="--ltx-bg-color:#FFFFFF;">Ubuntu IRC</span></span> <span class="ltx_td ltx_align_left ltx_th ltx_th_column"><span class="ltx_text ltx_font_bold" style="--ltx-bg-color:#FFFFFF;">PubMed Central</span></span> <span class="ltx_td ltx_align_left ltx_th ltx_th_column"><span class="ltx_text ltx_font_bold" style="--ltx-bg-color:#FFFFFF;">PhilPapers</span></span> <span class="ltx_td ltx_align_left ltx_th ltx_th_column"><span class="ltx_text ltx_font_bold" style="--ltx-bg-color:#FFFFFF;">DM Mathematics</span></span> <span class="ltx_td ltx_align_left ltx_th ltx_th_column"><span class="ltx_text ltx_font_bold" style="--ltx-bg-color:#FFFFFF;">Arxiv</span></span> <span class="ltx_td ltx_align_left ltx_th ltx_th_column"><span class="ltx_text ltx_font_bold" style="--ltx-bg-color:#FFFFFF;">Gutenberg (PG-19)</span></span> <span class="ltx_td ltx_align_left ltx_th ltx_th_column"><span class="ltx_text ltx_font_bold" style="--ltx-bg-color:#FFFFFF;">HackerNews</span></span></span> </span> <span class="ltx_tbody"> <span class="ltx_tr" style="--ltx-bg-color:#FFFFFF;"> <span class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_tt"><span class="ltx_ERROR undefined">\csvreader</span><span class="ltx_text" style="--ltx-bg-color:#FFFFFF;">[ head to column names, late after line=</span></span> <span class="ltx_td ltx_border_tt"></span> <span class="ltx_td ltx_border_tt"></span> <span class="ltx_td ltx_border_tt"></span> <span class="ltx_td ltx_border_tt"></span> <span class="ltx_td ltx_border_tt"></span> <span class="ltx_td ltx_border_tt"></span> <span class="ltx_td ltx_border_tt"></span> <span class="ltx_td ltx_border_tt"></span></span> <span class="ltx_tr" style="--ltx-bg-color:#E6E6E6;"> <span class="ltx_td ltx_nopad_r ltx_align_left ltx_th ltx_th_row"><span class="ltx_text" style="--ltx-bg-color:#E6E6E6;">]tables/the_pile_350M-r2.csv<span class="ltx_ERROR undefined">\csvcoli</span></span></span> <span class="ltx_td ltx_align_left"><math alttext="\csvcolii" class="ltx_Math" display="inline" id="p1.m1" intent=":literal" style="--ltx-bg-color:#E6E6E6;"><semantics><merror class="ltx_ERROR undefined undefined"><mtext>\csvcolii</mtext></merror><annotation encoding="application/x-tex">\csvcolii</annotation></semantics></math></span> <span class="ltx_td ltx_align_left"><math alttext="\csvcoliii" class="ltx_Math" display="inline" id="p1.m2" intent=":literal" style="--ltx-bg-color:#E6E6E6;"><semantics><merror class="ltx_ERROR undefined undefined"><mtext>\csvcoliii</mtext></merror><annotation encoding="application/x-tex">\csvcoliii</annotation></semantics></math></span> <span class="ltx_td ltx_align_left"><math alttext="\csvcoliv" class="ltx_Math" display="inline" id="p1.m3" intent=":literal" style="--ltx-bg-color:#E6E6E6;"><semantics><merror class="ltx_ERROR undefined undefined"><mtext>\csvcoliv</mtext></merror><annotation encoding="application/x-tex">\csvcoliv</annotation></semantics></math></span> <span class="ltx_td ltx_align_left"><math alttext="\csvcolv" class="ltx_Math" display="inline" id="p1.m4" intent=":literal" style="--ltx-bg-color:#E6E6E6;"><semantics><merror class="ltx_ERROR undefined undefined"><mtext>\csvcolv</mtext></merror><annotation encoding="application/x-tex">\csvcolv</annotation></semantics></math></span> <span class="ltx_td ltx_align_left"><math alttext="\csvcolvi" class="ltx_Math" display="inline" id="p1.m5" intent=":literal" style="--ltx-bg-color:#E6E6E6;"><semantics><merror class="ltx_ERROR undefined undefined"><mtext>\csvcolvi</mtext></merror><annotation encoding="application/x-tex">\csvcolvi</annotation></semantics></math></span> <span class="ltx_td ltx_align_left"><math alttext="\csvcolvii" class="ltx_Math" display="inline" id="p1.m6" intent=":literal" style="--ltx-bg-color:#E6E6E6;"><semantics><merror class="ltx_ERROR undefined undefined"><mtext>\csvcolvii</mtext></merror><annotation encoding="application/x-tex">\csvcolvii</annotation></semantics></math></span> <span class="ltx_td ltx_align_left"><math alttext="\csvcolviii" class="ltx_Math" display="inline" id="p1.m7" intent=":literal" style="--ltx-bg-color:#E6E6E6;"><semantics><merror class="ltx_ERROR undefined undefined"><mtext>\csvcolviii</mtext></merror><annotation encoding="application/x-tex">\csvcolviii</annotation></semantics></math></span> <span class="ltx_td ltx_align_left"><math alttext="\csvcolix" class="ltx_Math" display="inline" id="p1.m8" intent=":literal" style="--ltx-bg-color:#E6E6E6;"><semantics><merror class="ltx_ERROR undefined undefined"><mtext>\csvcolix</mtext></merror><annotation encoding="application/x-tex">\csvcolix</annotation></semantics></math></span></span> </span> </span> </span></span></span> <span class="ltx_caption"><span class="ltx_tag ltx_tag_p"><span class="ltx_text" style="font-size:90%;">Table 4</span>: </span><span class="ltx_text" style="font-size:90%;">Performance metrics across the subsets of the pile when starting from a randomly-initialized embedding matrix for our <math alttext="24" class="ltx_Math" display="inline" id="p1.m10" intent=":literal"><semantics><mn>24</mn><annotation encoding="application/x-tex">24</annotation></semantics></math>-block model. <span class="ltx_text ltx_font_typewriter">DEPT</span> models outperform the baselines.</span></span> </p> </div> </article> </div> <footer class="ltx_page_footer"> <div class="ltx_page_logo">Generated on Wed Oct 22 10:53:33 2025 by <a class="ltx_LaTeXML_logo" href="http://dlmf.nist.gov/LaTeXML/"><span style="letter-spacing:-0.2em; margin-right:0.1em;">L<span class="ltx_font_smallcaps" style="position:relative; bottom:2.2pt;">a</span>T<span class="ltx_font_smallcaps" style="font-size:120%;position:relative; bottom:-0.2ex;">e</span></span><span style="font-size:90%; position:relative; bottom:-0.2ex;">XML</span><img alt="Mascot Sammy" src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAsAAAAOCAYAAAD5YeaVAAAAAXNSR0IArs4c6QAAAAZiS0dEAP8A/wD/oL2nkwAAAAlwSFlzAAALEwAACxMBAJqcGAAAAAd0SU1FB9wKExQZLWTEaOUAAAAddEVYdENvbW1lbnQAQ3JlYXRlZCB3aXRoIFRoZSBHSU1Q72QlbgAAAdpJREFUKM9tkL+L2nAARz9fPZNCKFapUn8kyI0e4iRHSR1Kb8ng0lJw6FYHFwv2LwhOpcWxTjeUunYqOmqd6hEoRDhtDWdA8ApRYsSUCDHNt5ul13vz4w0vWCgUnnEc975arX6ORqN3VqtVZbfbTQC4uEHANM3jSqXymFI6yWazP2KxWAXAL9zCUa1Wy2tXVxheKA9YNoR8Pt+aTqe4FVVVvz05O6MBhqUIBGk8Hn8HAOVy+T+XLJfLS4ZhTiRJgqIoVBRFIoric47jPnmeB1mW/9rr9ZpSSn3Lsmir1fJZlqWlUonKsvwWwD8ymc/nXwVBeLjf7xEKhdBut9Hr9WgmkyGEkJwsy5eHG5vN5g0AKIoCAEgkEkin0wQAfN9/cXPdheu6P33fBwB4ngcAcByHJpPJl+fn54mD3Gg0NrquXxeLRQAAwzAYj8cwTZPwPH9/sVg8PXweDAauqqr2cDjEer1GJBLBZDJBs9mE4zjwfZ85lAGg2+06hmGgXq+j3+/DsixYlgVN03a9Xu8jgCNCyIegIAgx13Vfd7vdu+FweG8YRkjXdWy329+dTgeSJD3ieZ7RNO0VAXAPwDEAO5VKndi2fWrb9jWl9Esul6PZbDY9Go1OZ7PZ9z/lyuD3OozU2wAAAABJRU5ErkJggg=="/></a> </div></footer> </div> </body> </html>