This poster introduces Hammer, a family of function-calling models using function masking to improve robustness. It details the method, shows performance gains on benchmarks like BFCL, and analyzes masking ratios.
Paper title: Robust Function-Calling for On-Device Language Models via Function Masking Abstract: This poster introduces Hammer, a family of function-calling models using function masking to improve robustness. It details the method, shows performance gains on benchmarks like BFCL, and analyzes masking ratios. Paper body (method & results): <!DOCTYPE html> <html lang="en"> <head> <meta content="text/html; charset=utf-8" http-equiv="content-type"/> <title>Hammer: Robust Function-Calling for On-Device Language Models via Function Masking</title> <!--Generated on Sun Oct 6 18:43:28 2024 by LaTeXML (version 0.8.8) http://dlmf.nist.gov/LaTeXML/.--> <meta content="width=device-width, initial-scale=1, shrink-to-fit=no" name="viewport"/> <link href="https://cdn.jsdelivr.net/npm/bootstrap@5.3.0/dist/css/bootstrap.min.css" rel="stylesheet" type="text/css"/> <link href="/static/browse/0.3.4/css/ar5iv.0.7.9.min.css" rel="stylesheet" type="text/css"/> <link href="/static/browse/0.3.4/css/ar5iv-fonts.0.7.9.min.css" rel="stylesheet" type="text/css"/> <link href="/static/browse/0.3.4/css/latexml_styles.css" rel="stylesheet" type="text/css"/> <script src="https://cdn.jsdelivr.net/npm/bootstrap@5.3.0/dist/js/bootstrap.bundle.min.js"></script> <script src="https://cdnjs.cloudflare.com/ajax/libs/html2canvas/1.3.3/html2canvas.min.js"></script> <script src="/static/browse/0.3.4/js/addons_new.js"></script> <script src="/static/browse/0.3.4/js/feedbackOverlay.js"></script> <base href="/html/2410.04587v1/"/></head> <body> <nav class="ltx_page_navbar"> <nav class="ltx_TOC"> <ol class="ltx_toclist"> <li class="ltx_tocentry ltx_tocentry_section"><a class="ltx_ref" href="https://arxiv.org/html/2410.04587v1#S1" title="In Hammer: Robust Function-Calling for On-Device Language Models via Function Masking"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">1 </span>Introduction</span></a></li> <li class="ltx_tocentry ltx_tocentry_section"><a class="ltx_ref" href="https://arxiv.org/html/2410.04587v1#S2" title="In Hammer: Robust Function-Calling for On-Device Language Models via Function Masking"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">2 </span>Related Works</span></a></li> <li class="ltx_tocentry ltx_tocentry_section"> <a class="ltx_ref" href="https://arxiv.org/html/2410.04587v1#S3" title="In Hammer: Robust Function-Calling for On-Device Language Models via Function Masking"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">3 </span>Problem Statement and Analysis</span></a> <ol class="ltx_toclist ltx_toclist_section"> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2410.04587v1#S3.SS1" title="In 3 Problem Statement and Analysis ‣ Hammer: Robust Function-Calling for On-Device Language Models via Function Masking"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">3.1 </span>Misleadingness by Function Name and Parameter Name</span></a></li> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2410.04587v1#S3.SS2" title="In 3 Problem Statement and Analysis ‣ Hammer: Robust Function-Calling for On-Device Language Models via Function Masking"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">3.2 </span>The Impact of Excessive Focus on the Naming</span></a></li> </ol> </li> <li class="ltx_tocentry ltx_tocentry_section"> <a class="ltx_ref" href="https://arxiv.org/html/2410.04587v1#S4" title="In Hammer: Robust Function-Calling for On-Device Language Models via Function Masking"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">4 </span>Methodology</span></a> <ol class="ltx_toclist ltx_toclist_section"> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2410.04587v1#S4.SS1" title="In 4 Methodology ‣ Hammer: Robust Function-Calling for On-Device Language Models via Function Masking"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">4.1 </span>Function Masking</span></a></li> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2410.04587v1#S4.SS2" title="In 4 Methodology ‣ Hammer: Robust Function-Calling for On-Device Language Models via Function Masking"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">4.2 </span>Irrelevance-Augmented Dataset</span></a></li> </ol> </li> <li class="ltx_tocentry ltx_tocentry_section"> <a class="ltx_ref" href="https://arxiv.org/html/2410.04587v1#S5" title="In Hammer: Robust Function-Calling for On-Device Language Models via Function Masking"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">5 </span>Evaluation</span></a> <ol class="ltx_toclist ltx_toclist_section"> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2410.04587v1#S5.SS1" title="In 5 Evaluation ‣ Hammer: Robust Function-Calling for On-Device Language Models via Function Masking"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">5.1 </span>Experimental Setup</span></a></li> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2410.04587v1#S5.SS2" title="In 5 Evaluation ‣ Hammer: Robust Function-Calling for On-Device Language Models via Function Masking"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">5.2 </span>Overall Performance on Various Benchmarks</span></a></li> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2410.04587v1#S5.SS3" title="In 5 Evaluation ‣ Hammer: Robust Function-Calling for On-Device Language Models via Function Masking"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">5.3 </span>Detailed Performance on Different Types of Function Calling</span></a></li> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2410.04587v1#S5.SS4" title="In 5 Evaluation ‣ Hammer: Robust Function-Calling for On-Device Language Models via Function Masking"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">5.4 </span>Ablation on Different Base Models</span></a></li> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2410.04587v1#S5.SS5" title="In 5 Evaluation ‣ Hammer: Robust Function-Calling for On-Device Language Models via Function Masking"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">5.5 </span>Ablation on Different Masking Ratio</span></a></li> <li class="ltx_tocentry ltx_tocentry_subsection"><a class="ltx_ref" href="https://arxiv.org/html/2410.04587v1#S5.SS6" title="In 5 Evaluation ‣ Hammer: Robust Function-Calling for On-Device Language Models via Function Masking"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">5.6 </span>Ablation on Different Proportions of Irrelevance-Augmented Data</span></a></li> </ol> </li> <li class="ltx_tocentry ltx_tocentry_section"><a class="ltx_ref" href="https://arxiv.org/html/2410.04587v1#S6" title="In Hammer: Robust Function-Calling for On-Device Language Models via Function Masking"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">6 </span>Conclusion</span></a></li> <li class="ltx_tocentry ltx_tocentry_appendix"><a class="ltx_ref" href="https://arxiv.org/html/2410.04587v1#A1" title="In Hammer: Robust Function-Calling for On-Device Language Models via Function Masking"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">A </span>Extra Experimental Results</span></a></li> <li class="ltx_tocentry ltx_tocentry_appendix"><a class="ltx_ref" href="https://arxiv.org/html/2410.04587v1#A2" title="In Hammer: Robust Function-Calling for On-Device Language Models via Function Masking"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">B </span>Different Types of Function-calling Tasks</span></a></li> <li class="ltx_tocentry ltx_tocentry_appendix"><a class="ltx_ref" href="https://arxiv.org/html/2410.04587v1#A3" title="In Hammer: Robust Function-Calling for On-Device Language Models via Function Masking"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">C </span>Example Input to Models with Function Masking</span></a></li> </ol></nav> </nav> <div class="ltx_page_main"> <div class="ltx_page_content"> <article class="ltx_document ltx_authors_1line"> <h1 class="ltx_title ltx_title_document"> Hammer: Robust Function-Calling for On-Device <br class="ltx_break"/>Language Models via Function Masking </h1> <div class="ltx_authors"> <span class="ltx_creator ltx_role_author"> <span class="ltx_personname">Qiqiang Lin<sup class="ltx_sup" id="id6.6.id1">∗</sup> <br class="ltx_break"/><span class="ltx_text ltx_font_typewriter" id="id7.7.id2">linqiqiang1@oppo.com</span> Muning Wen<sup class="ltx_sup" id="id8.8.id3">∗</sup> <br class="ltx_break"/><span class="ltx_text ltx_font_typewriter" id="id9.9.id4">muningwen@sjtu.edu.cn</span> Qiuying Peng<sup class="ltx_sup" id="id10.10.id5"><span class="ltx_text ltx_font_italic" id="id10.10.id5.1">∗,+</span></sup> <br class="ltx_break"/><span class="ltx_text ltx_font_typewriter" id="id11.11.id6">pengqiuying@oppo.com</span> Guanyu Nie <br class="ltx_break"/><span class="ltx_text ltx_font_typewriter" id="id12.12.id7">nieg@iastate.edu</span> Junwei Liao <br class="ltx_break"/><span class="ltx_text ltx_font_typewriter" id="id13.13.id8">jwliao.ai@gmail.com</span> Jun Wang <br class="ltx_break"/><span class="ltx_text ltx_font_typewriter" id="id14.14.id9">wangjun15@oppo.com</span> Xiaoyun Mo <br class="ltx_break"/><span class="ltx_text ltx_font_typewriter" id="id15.15.id10">moxiaoyun@oppo.com</span> Jiamu Zhou <br class="ltx_break"/><span class="ltx_text ltx_font_typewriter" id="id16.16.id11">zhoujiamu@oppo.com</span> Cheng Cheng <br class="ltx_break"/><span class="ltx_text ltx_font_typewriter" id="id17.17.id12">chengcheng2@oppo.com</span> Yin Zhao <br class="ltx_break"/><span class="ltx_text ltx_font_typewriter" id="id18.18.id13">zhaoyin@oppo.com</span> Jun Wang<sup class="ltx_sup" id="id19.19.id14">+</sup> <br class="ltx_break"/><span class="ltx_text ltx_font_typewriter" id="id20.20.id15">wangjun7@oppo.com</span> Weinan Zhang<sup class="ltx_sup" id="id21.21.id16">+</sup> <br class="ltx_break"/><span class="ltx_text ltx_font_typewriter" id="id22.22.id17">wnzhang@sjtu.edu.cn</span> </span></span> </div> <div class="ltx_abstract"> <h6 class="ltx_title ltx_title_abstract">Abstract</h6> <p class="ltx_p" id="id23.id1">Large language models have demonstrated impressive value in performing as autonomous agents when equipped with external tools and API calls. Nonetheless, effectively harnessing their potential for executing complex tasks crucially relies on enhancements in their function-calling capabilities. This paper identifies a critical gap in existing function-calling models, where performance varies significantly across benchmarks, often due to being misled by specific naming conventions. To address such an issue, we introduce Hammer, a novel family of foundation models specifically engineered for on-device function calling. Hammer employs an augmented dataset that enhances models’ sensitivity to irrelevant functions and