JudgeLM Fine-Tuning LLMs as Scalable Judges — Poster

Paper context

Paper title: JudgeLM: Fine-tuned Large Language Models are Scalable Judges Abstract: A conference poster presenting JudgeLM, a framework for fine-tuning LLMs to act as scalable judges for open-ended scenarios, featuring data generation pipelines, bias mitigation strategies, and performance comparisons against baselines like GPT-4 and PandaLM. Paper body (method & results): Published as a conference paper at ICLR 2025 JUDGELM: FINE-TUNED LARGE LANGUAGE MODELS ARE SCALABLE JUDGES Lianghui Zhu1,2 ∗ Xinggang Wang1† Xinlong Wang2† 1 School of EIC, Huazhong University of Science & Technology 2 Beijing Academy of Artificial Intelligence Code & Models: https://github.com/baaivision/JudgeLM ABSTRACT Evaluating Large Language Models (LLMs) in open-ended scenarios is challeng- ing because existing benchmarks and metrics can not measure them comprehen- sively. To address this problem, we propose to fine-tune LLMs as scalable judges (JudgeLM) to evaluate LLMs efficiently and effectively in open-ended bench- marks. We first propose a comprehensive, large-scale, high-quality dataset con- taining task seeds, LLMs-generated answers, and GPT-4-generated judgments for fine-tuning high-performance judges, as well as a new benchmark for evaluating the judges. We train JudgeLM at different scales from 7B, 13B, to 33B parame- ters, and conduct a systematic analysis of its capabilities and behaviors. We then analyze the key biases in fine-tuning LLM as a judge and consider them as position bias, knowledge bias, and format bias. To address these issues, JudgeLM intro- duces a bag of techniques including swap augmentation, reference support, and reference drop, which clearly enhance the judge’s performance. JudgeLM obtains the state-of-the-art judge performance on both the existing PandaLM benchmark and our proposed new benchmark. Our JudgeLM is efficient and the JudgeLM-7B only needs 3 minutes to judge 5K samples with 8 A100 GPUs. JudgeLM obtains high agreement with the teacher judge, achieving an agreement exceeding 90% that even surpasses human-to-human agreement1. JudgeLM also demonstrates extended capabilities in being judges of the single answer, multimodal models, multiple answers, multi-turn chat, etc. 1 INTRODUCTION Recent advancements in large language models (LLMs) have fostered significant interest due to their remarkable performance in following instructions and their broad capabilities in dealing with open-ended scenarios. Based on the open-source LLMs, including OPT (Zhang et al., 2022), Flan- T5 (Chung et al., 2022), LLaMA (Touvron et al., 2023a), and Pythia (Biderman et al., 2023), re- searchers propose numerous methods to align these models with human preferences through instruc- tion fine-tuning. These aligned LLMs demonstrate enhanced abilities in comprehending human in- structions and generating more coherent responses. Nonetheless, existing benchmarks (Hendrycks et al., 2020; Liang et al., 2022) and traditional metrics (Lin, 2004; Papineni et al., 2002; Zhang et al., 2019; Sellam et al., 2020; Yuan et al., 2021) do not adequately estimate the capabilities of LLMs in open-ended scenarios. Therefore, a new benchmark method that could evaluate LLMs comprehensively in open-ended tasks is needed. Concurrent works are making efforts to explore various methods for evaluating the performance of LLM. The arena-format (Zheng et al., 2023) methods leverage crowdsourced platforms to extract anonymous LLM competition results. While evaluations by humans are trustworthy, they are also time-consuming and financially demanding. Some approaches (Chiang et al., 2023) utilize GPT-4 as ∗This work was done when Lianghui Zhu was an intern at Beijing Academy of Artificial Intelligence. †Corresponding authors: xgwang@hust.edu.cn and wangxinlong@baai.ac.cn. 1As a reference, the max agreement among humans in MT-bench (Zheng et al., 2023) is 82%. 1 arXiv:2310.17631v2 [cs.CL] 1 Mar 2025 Published as a conference paper at ICLR 2025 a judge. Nevertheless, these methods grapple with challenges of potential data exposure and volatile API model transitions, potentially compromising the judge’s reproducibility. PandaLM (Wang et al., 2023) attempts to fine-tune open-source LLMs for evaluating answers. However, limitations stem- ming from the training data quality, and inherent LLM biases, undermine the effectiveness of such fine-tuned models in the role of a judge. GPT-4 Answer1: The modifiers used in the sentence are: \"green,\" …… Answer2: She drove her green car quickly. Answer Pair Score: 10, 1 Reason: Assistant2 didn’t provide any answer to the question, hence the low score … GPT-4 Judgments Instruction: Identify the words that are used as modifiers. Input: She drove her green car quickly. Task Vicuna LLaMA ChatGLM Alpaca Koala MPT Open Assistant ... (a) Data generation pipeline of our JudgeLM. We first collect 105K seed tasks as questions. Then, we extract answers from 11 LLMs and randomly sample a pair of answers from the answer set. Last, we input the tasks, the sampled answer pairs, and optionally reference answers to GPT-4, which generates scores and detailed reasons as a judge teacher. JudgeLM 7B 13B 33B ref. sup. swap aug. ref. drop random() > 𝛼? random() > 𝛽? Yes Yes No No LLMs Task Answer Pair Judgment Judge Sample Reference optional input 𝛼: swap answer threshold (0.5) 𝛽: reference drop threshold (0.5) random(): random numbers between [0,1) Vicuna LLaMA Alpaca Koala ... (b) An illustration of the JudgeLM’s fine-tuning and various functions. We use generated judge samples to fine-tune LLMs as scalable judges. When fine-tuning LLMs as judges, we also propose swap augmentation, reference support, and reference drop to address the position bias, knowledge bias, and format bias, respectively. Judge Answer Pairs LLM1 LLM2 Grade Single Answers Score: 9 LLM1 Judge Multiple Answers LLM1 LLM2 LLM3 Explain Judgments WHY LLM2 LLM1 Multi-turn Chat about Judgments LLM2 LLM1 Judge Multimodal Answers LLM2 LLM1 (c) An illustration of various functions of our JudgeLM. Figure 1: An overview of our scalable JudgeLM including data generation, fine-tuning, and various functions. In this paper, we propose to evaluate LLMs through fine-tuned open-source LLMs, which serve as scalable judges (JudgeLM) achieving satisfactory agreement with the teacher judge. Our method- ology incorporates scalable judges as evaluators in open-ended tasks, coupled with a high-quality dataset conducive to both training and evaluating the judge models. Within our framework, we adapt open-source LLMs to serve as judges and analyze their scaling ability in relation to model size (ranging from 7B to 33B) and volume of training data (extending from 3.5K to 100K). Our curated dataset comprises 105K seed questions, LLM answer pairs, and judgments from the teacher judge, GPT-4, as shown in Fig. 1a. Note that we generated two judgments for each seed task with and without reference answers. This dataset is partitioned, with 100K seed questions allocated for training (2 × larger than PandaLM) and the remainder for validation (29 × larger than PandaLM). Utilizing LLMs as judges inevitably introduces biases such as position bias (favoring answers in spe- cific positions), knowledge bias (over-reliance on pre-trained knowledge), and format bias (optimal performance only under specific prompt formats) as shown in Fig. 8, 10, 12, 13. When fine-tuning is not possible, GPT-4-API-based judge (Zheng et al., 2023) tries to alleviate this by well-designed prompt methods, i.e., Chain-of-thought, few-shot judge, and judging multiple times with different positions. JudgeLM presents a new way that can address these biases in the fine-tuning stage, skip- ping the complicated prompt methods and multi-turn API calling. Moreover, our JudgeLM system presents extended capabilities as shown in Fig. 1b, including grading single answers, judging multi- ple answers, judging multimodal models, multi-turn chat, etc. In contrast to arena-format methods, our approach is rapid and has a low cost. For instance, JudgeLM-7B requires only 8 A100 GPUs and can evaluate 5000 response pairs in just 3 minutes. In comparison to closed-source LLM judges, JudgeLM ensures reproducibility and protects user pri- vacy. When compared to concurrent open-source LLM judges, our system explores both the scaling ability and biases in LLM fine-tuning. Furthermore, JudgeLM dataset stands as the most diverse and high-quality one, significantly benefitting subsequent research in judge model investigations. 2 Published as a conference paper at ICLR 2025 Our main contributions can be summarized as follows: • We introduce a high-quality, large-scale dataset for judge models, enriched with diverse seed tasks, LLMs-generated answers, and detailed judgments from GPT-4, laying the foun- dation for future LLMs evaluating research. • We propose JudgeLM, a scalable language model judge, designed for evaluating LLMs in open-ended scenarios. It achieves an agreement exceeding 90% that surpasses the human- to-human agreement. Our JudgeLM can also generalize to many extended tasks. • We analyze the biases inherent to LLM judge fine-tuning and introduce a series of meth- ods to address them. Our methods significantly improve the consistency of the model in different cases, making the JudgeLM more reliable and flexible. 2 RELATED WORK 2.1 INSTRUCTION FINE-TUNING OF LARGE LANGUAGE MODELS With the development of large language models (LLMs), researchers find that fine-tuning pre-trained LLMs such as GPT-3 (Brown et al., 2020), T5 (Raffel et al., 2020), OPT (Zhang et al., 2022), and PaLM (Chowdhery et al., 2022) enable LLMs to follow human instructions and help with open- ended tasks. The instruction fine-tuned LLMs such as InstructGPT (Ouyang et al., 2022), Chat- GPT (OpenAI, 2022), FLAN-T5 (Chung et al., 2022), FLAN-PaLM (Chung et al., 2022), OPT- IML (Iyer et al., 2022), and GPT-4 (OpenAI, 2023) exhibit stronger ability in zero-shot or few-shot tasks than their base models. After Meta released the powerful open-source LLM LLaMA (Tou- vron et al., 2023a) and LLaMA2 (Touvron et al., 2023b), lots of instruction fine-tuning works based on LLaMA or LLaMA2 were proposed in the natural language generation or multimodal genera- tion domain, such as Alpaca, Vicuna (Chiang et al., 2023), OpenFlamingo (Awadalla et al., 2023), LLaMA-Adapter (Zhang et al., 2023), and Emu (Sun et al., 2023). Our JudgeLM also belongs to the LLaMA family and takes the Vicuna series as base models. Our JudgeLM follows the instruction fine-tuning manner to create LLM judges and proposes to model the judgment-generation task as “grading, judging, and reasoning”. We further collect a high-quality, large-scale dataset for research in judging the performance of LLMs. 2.2 EVALUATION OF LARGE LANGUAGE MODELS As many open-source large language models (LLMs) and their fine-tuned variants are proposed and present remarkable performance on various tasks, evaluating the capabilities of LLMs becomes a popular and challenging task. To address this problem, Chatbot Arena (Zheng et al., 2023) aims to build a crowdsourced platform that ranks the LLMs through pairwise comparison and Elo rating. The crowdsourced way to evaluate LLMs has more reliable results but faces high costs and low effi- ciency. Vicuna (Chiang et al., 2023) uses GPT-4 as a