RaTA-Tool Multimodal Tool Retrieval Pipeline
Figure 3: Overview of our pipeline. RaTA-Tool supports multimodal user queries by jointly processing the input prompt and associated modalities. A fine-tuned LLM encodes the combined inputs to generate a structured task description of the user request, further aligned via Direct Preference Optimization (DPO). This description is embedded and used to retrieve the most relevant tool from an external tool collection.
Paper context
Paper title: RaTA-Tool: Retrieval-based Tool Selection with Multimodal Large Language Models Abstract: Tool learning with foundation models aims to endow AI systems with the ability to invoke external resources -- such as APIs, computational utilities, and specialized models -- to solve complex tasks beyond the reach of standalone language generation. While recent advances in Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs) have expanded their reasoning and perception capabilities, existing tool-use methods are predominantly limited to text-only inputs and closed-world settings. Consequently, they struggle to interpret multimodal user instructions and cannot generalize to tools unseen during training. In this work, we introduce RaTA-Tool, a novel framework for open-world multimodal tool selection. Rather than learning direct mappings from user queries to fixed tool identifiers, our approach enables an MLLM to convert a multimodal query into a structured task description and subsequently retrieve the most appropriate tool by matching this representation against semantically rich, machine-readable tool descriptions. This retrieval-based formulation naturally supports extensibility to new tools without retraining. To further improve alignment between task descriptions and tool selection, we incorporate a preference-based optimization stage using Direct Preference Optimization (DPO). To support research in this setting, we also introduce the first dataset for open-world multimodal tool use, featuring standardized tool descriptions derived from Hugging Face Passages referencing this figure: ls may not be observed during training. Rather than predicting tool names or IDs, we transform multimodal user queries into structured task descriptions and retrieve suitable tools via semantic matching against rich, machine-readable tool representations. This retrieval-based formulation decouples tool selection from fixed inventories and enables scalable generalization to newly introduced tools. Figure 1: Pipeline of dataset creation. We first perform a metadata acquisition step by scraping model cards from Hugging Face. The collected data then undergoes a cleaning stage. Finally, given a user query and its associated prompt, we feed this information into an LLM to generate a structured tool description in a standardized JSON format. 3 Proposed Method 3.1 Dataset Creation Preliminaries. A he expected input format and modalities supported by the tool; 2 output , the type and structure of the generated output; and 3 process , a concise functional summary of the tool capabilities. This structured representation enables consistent and machine-readable tool descriptions, facilitating downstream tool selection and reasoning tasks. An overview of the dataset creation pipeline is shown in Fig. 1 , while some qualitative examples from our dataset are shown in Fig. 2 . Figure 3: Overview of our pipeline. RaTA-Tool supports multimodal user queries by jointly processing the input prompt and associated modalities. A fine-tuned LLM encodes the combined inputs to generate a structured task description of the user request, further aligned via Direct Preference Optimization (DPO). This desc
