SAFE for Long-Form LLM Factuality Evaluation

论文上下文

Paper title: Extrinsic Hallucinations in LLMs | Lil'Log Abstract: The figure illustrates the SAFE pipeline for evaluating factuality in LLM outputs, detailing a four-step process and comparing automated vs. human evaluation results. Source body: Table of Contents What Causes Hallucinations? Pre-training Data Issues Fine-tuning New Knowledge Hallucination Detection Retrieval-Augmented Evaluation Sampling-Based Detection Calibration of Unknown Knowledge Indirect Query Anti-Hallucination Methods RAG → Edits and Attribution Chain of Actions Sampling Methods Fine-tuning for Factuality Fine-tuning for Attribution Appendix: Evaluation Benchmarks Citation References Hallucination in large language models usually refers to the model generating unfaithful, fabricated, inconsistent, or nonsensical content. As a term, hallucination has been somewhat generalized to cases when the model makes mistakes. Here, I would like to narrow down the problem of hallucination to cases where the model output is fabricated and not grounded by either the provided context or world knowledge. There are two types of hallucination: In-context hallucination: The model output should be consistent with the source content in context. Extrinsic hallucination: The model output should be grounded by the pre-training dataset. However, given the size of the pre-training dataset, it is too expensive to retrieve and identify conflicts per generation. If we consider the pre-training data corpus as a proxy for world knowledge, we essentially try to ensure the model output is factual and verifiable by external world knowledge. Equally importantly, when the model does not know about a fact, it should say so. This post focuses on extrinsic hallucination. To avoid hallucination, LLMs need to be (1) factual and (2) acknowledge not knowing the answer when applicable. What Causes Hallucinations? # Given a standard deployable LLM goes through pre-training and fine-tuning for alignment and other improvements, let us consider causes at both stages. Pre-training Data Issues # The volume of the pre-training data corpus is enormous, as it is supposed to represent world knowledge in all available written forms. Data crawled from the public Internet is the most common choice and thus out-of-date, missing, or incorrect information is expected. As the model may incorrectly memorize this information by simply maximizing the log-likelihood, we would expect the model to make mistakes. Fine-tuning New Knowledge # Fine-tuning a pre-trained LLM via supervised fine-tuning and RLHF is a common technique for improving certain capabilities of the model like instruction following. Introducing new knowledge at the fine-tuning stage is hard to avoid. Fine-tuning usually consumes much less compute, making it debatable whether the model can reliably learn new knowledge via small-scale fine-tuning. Gekhman et al. 2024 studied the research question of whether fine-tuning LLMs on new knowledge encourages hallucinations. They found that (1) LLMs learn fine-tuning examples with new knowledge slower than other examples with knowledge consistent with the pre-existing knowledge of the model; (2) Once the examples with new knowledge are eventually learned, they increase the model’s tendency to hallucinate. Given a closed-book QA dataset (i.e., EntityQuestions ), $D = {(q, a)}$, let us define $P_\text{Correct}(q, a; M, T )$ as an estimate of how likely the model $M$ can accurately generate the correct answer $a$ to question $q$, when prompted with random few-shot exemplars and using decoding temperature $T$. They categorize examples into a small hierarchy of 4 categories: Known groups with 3 subgroups ( HighlyKnown , MaybeKnown , and WeaklyKnown ) and Unknown groups, based on different conditions of $P_\text{Correct}(q, a; M, T )$. Knowledge categorization of close-book QA examples based on how likely the model outputs correct answers. (Image source: Gekhman et al. 2024 ) Some interesting observations of the experiments, where dev set accuracy is considered a proxy for hallucinations. Unknown examples are fitted substantially slower than Known . The best dev performance is obtained when the LLM fits the majority of the Known training examples but only a few of the Unknown ones. The model starts to hallucinate when it learns most of the Unknown examples. Among Known examples, MaybeKnown cases result in better overall performance, more essential than HighlyKnown ones. Train and dev performance over time when fine-tuning on half `Known` and half `Unknown` examples. `Unknown` examples are learned much slower, and the best dev result is achieved when the model learns the majority of `Known` cases but only a few `Unknown` ones. (Image source: Gekhman et al. 2024 ) These empirical results from Gekhman et al. (2024) point out the risk of using supervised fine-tuning for updating LLMs’ knowledge. Hallucination Detection # Retrieval-Augmented Evaluation # To quantify model hallucinations, Lee et al. (2022) introduced a new benchmark dataset, FactualityPrompt , consisting of both factual and nonfactual prompts. This dataset uses Wikipedia documents or sentences as the knowledge base for factuality grounding. The Wikipedia documents are known ground-truth from the FEVER dataset, and the sentences are selected based on tf-idf or sentence embedding-based similarity. The evaluation framework for the FactualityPrompt benchmark. (Image source: Lee, et al. 2022 ) Given the model continuation and paired Wikipedia text, two evaluation metrics for hallucination are considered: Hallucination NE (Named Entity) errors : Using a pretrained entity detection model and document-level grounding, this metric measures the fraction of detected named entities that do not appear in the ground truth document. Entailment ratios : Using a RoBERTa model fine-tuned on MNLI and sentence-level knowledge grounding, this metric calculates the fraction of generated sentences that are marked as relevant to the paired Wikipedia sentence by the entailment model. Lower NE errors and higher entailment ratios indicate higher factuality, and both metrics are found to be correlated with human annotations. Larger models are found to perform better on this