LVLM vs LLM Instruction-Following Ability Gap
Figure 1: LVLMs (left) show lower instruction-following ability than LLMs (right). We examine this gap quantitatively and explore the factors that contribute to reductions in instruction-following ability.
Paper context
Paper title: Instruction-Following Evaluation of Large Vision-Language Models Abstract: Following the initial flourishing of large language models (LLMs), there has been a surge in proposed large vision-language models (LVLMs) that integrate LLMs with vision capabilities. However, it has been observed that LVLMs, after tuning to visual instruction using commonly used training datasets, often fail to exhibit the instruction-following ability that was present in the LLM before integration, leading to results in which they do not follow task instructions as expected. This study quantitatively demonstrates that LVLMs' instruction-following ability declines after fine-tuning and analyzes its underlying causes. In particular, we constructed new training datasets highlighting whether the output format is specified. Then, we investigated how explicitly indicating the output format during fine-tuning affects LVLMs' instruction-following ability. Our quantitative evaluation confirmed that LVLMs' instruction-following ability declines after fine-tuning with commonly used datasets. Furthermore, we found that LVLMs trained with datasets, including instructions on output format, tend to follow instructions more accurately than models that do not. These findings suggest that including samples with instructions on output format during (visual) instruction tuning may help mitigate the decline in instruction-following abilities. Passages referencing this figure: Figure 1: LVLMs (left) show lower instruction-following ability than LLMs (right). However, as shown in Figure 1 , while the LLM, before it is integrated into the LVLM, can follow instructions and generate correct responses, it has been qualitatively confirmed by Fu et al. We also specify factors behind the diminished ability of LVLM to follow instructions (Figure 1 ). 1 Influence of insufficient instructions on the output format In Figure 1 , we present an example in which the LVLM does not properly account for instructions for the output format.
