Human Expertise in Algorithmic Prediction — Poster

Paper context

Paper title: Human Expertise in Algorithmic Prediction Abstract: A conference poster titled 'Human Expertise in Algorithmic Prediction' by researchers from MIT and Yale. It explores how human experts provide signal in algorithmically indistinguishable subsets, using Emergency Room triage and radiology as case studies. Paper body (method & results): Human Expertise in Algorithmic Prediction Rohan Alur EECS, LIDS MIT ralur@mit.edu Manish Raghavan EECS, LIDS, Sloan MIT mragh@mit.edu Devavrat Shah EECS, IDSS, LIDS, SDSC MIT devavrat@mit.edu Abstract We introduce a novel framework for incorporating human expertise into algorithmic predictions. Our approach leverages human judgment to distinguish inputs which are algorithmically indistinguishable, or “look the same" to predictive algorithms. We argue that this framing clarifies the problem of human-AI collaboration in prediction tasks, as experts often form judgments by drawing on information which is not encoded in an algorithm’s training data. Algorithmic indistinguishability yields a natural test for assessing whether experts incorporate this kind of “side information", and further provides a simple but principled method for selectively incorporating human feedback into algorithmic predictions. We show that this method provably improves the performance of any feasible algorithmic predictor and precisely quantify this improvement. We find empirically that although algo- rithms often outperform their human counterparts on average, human judgment can improve algorithmic predictions on specific instances (which can be identified ex-ante). In an X-ray classification task, we find that this subset constitutes nearly 30% of the patient population. Our approach provides a natural way of uncovering this heterogeneity and thus enabling effective human-AI collaboration. 1 Introduction Despite remarkable advances in machine learning, human judgment continues to play a critical role in many high-stakes prediction tasks. For example, consider the problem of triage in the emergency room, where healthcare providers assess and prioritize patients for immediate care. On one hand, prognostic algorithms offer significant promise for improving triage decisions; indeed, algorithmic predictions are often more accurate than even expert human decision makers [1, 2, 3, 4, 5, 6, 7, 8]. On the other hand, predictive algorithms may fail to fully capture the relevant context for each individual. For example, an algorithmic risk score may only have access to tabular electronic health records or other structured data (e.g., medical imaging), while a physician has access to many additional modalities—not least of which is the ability to directly examine the patient! These two observations—that algorithms often outperform humans, but humans often have access to a richer information set—are not in conflict with each other. Indeed, [9] find exactly this phenomenon in an analysis of emergency room triage decisions. This suggests that, even in settings where algorithms outperform humans, algorithms might still benefit from some form of human input. Ideally this collaboration will yield human-AI complementarity [10, 11], in which a joint system outperforms either a human or algorithm working alone. Our work thus begins with the following question: When (and how) can human judgment improve the predictions of any learning algorithm? Example: X-ray classification. Consider the problem of diagnosing atelectasis (a partially or fully collapsed lung; we study this task in detail in Section 5). Today’s state-of-the-art deep learning models can perform well on these kinds of classification tasks using only a patient’s chest X-ray as Preprint. Under review. arXiv:2402.00793v3 [cs.LG] 30 Oct 2024 input [12, 13, 14]. We are interested in whether we can further improve these algorithmic predictions by incorporating a “second opinion” from a physician, particularly because the physician may have access to information (e.g., by directly observing the patient) which is not present in the X-ray. A first heuristic, without making any assumptions about the available predictive models, is to ask whether a physician can distinguish patients whose imaging data are identical. For example, if a physician can correctly indicate that one patient is suffering from atelectasis while another is not—despite the patients having identical chest X-rays—the physician must have information that the X-ray does not capture. In principle, this could form the basis for a statistical test: we could ask whether the physician performs better than random in distinguishing a large number of such patients. If so, even a predictive algorithm which outperforms the physician might benefit from human input. Of course, we are unlikely to find identical observations in continuous-valued and/or high-dimensional data (like X-rays). A natural relaxation is to instead consider observations which are sufficiently “similar”, as suggested by [9]. In this work we propose a more general notion of algorithmic indistinguishability, or coarser subsets of inputs in which no algorithm (in some rich, user-defined class) has significant predictive power. We show that these subsets can be discovered via a novel connection to multicalibration [15], and formally demonstrate that using human feedback to predict outcomes within these subsets can outperform any algorithmic predictor (in the same user-defined class). In addition to being tractable, this framework is relevant from a decision-theoretic perspective: although we’ve focused thus far on algorithms’ fundamental informational constraints, it is also natural to ask whether an expert provides signal which is merely difficult for an algorithm to learn directly (due to e.g., limited training data or computational constraints). Our approach naturally interpolates between these contexts by defining indistinguishability with respect to whichever class of models is practically relevant for a given prediction task. We elaborate on these contributions below. Contributions. We propose a novel framework for human-AI collaboration in prediction tasks. Our approach uses human feedback to refine predictions within sets of inputs which are algorithmically indistinguishable, or “look the same" to predictive algorithms. In Section 4 we present a simple method to incorporate this feedback only when it improves on the best feasible predictive model (and precisely quantify this improvement). This extends the “omnipredictors” result of [16] in the special case of squared error, which may be of independent interest.1 In Section 5 we present experiments demonstrating that although humans fail to outperform algorithmic predictors on average, there exist specific (algorithmically indistinguishable) instances on which humans are more accurate than the best available predictor (and these instances are identifiable ex ante).2 In Section 6 we consider the complementary setting in which an algorithm provides recommendations to many downstream users, who independently choose when to comply. We provide conditions under which a predictor is robust to these compliance patterns, and thus be simultaneously optimal for all downstream users. 2 Related work The relative strengths of humans and algorithms. Our work is motivated by large body of literature which studies the relative strengths of human judgment and algorithmic decision making [1, 2, 3, 5] or identifies behavioral biases in decision making [17, 18, 19, 20]. More recent work also studies whether predictive algorithms can improve expert decision making [4, 8, 21, 14]. Recommendations, deferral and complementarity. One popular approach for incorporating human judgment into algorithmic predictions is by deferring some instances to a human decision maker [22, 23, 24, 25, 26, 27]. Other work studies contexts where human decision makers are free to override algorithmic recommendations [28, 29, 30, 31, 14], which may suggest alternative design criteria for these algorithms [32, 33, 34]. More generally, systems which achieve human-AI complementarity (as defined in Section 1) have been previously studied in [35, 32, 36, 37, 38, 39, 40]. [11] develop a comprehensive taxonomy of this area, which generally takes the predictor as given, or learns a predictor which is optimized to complement a particular model of human decision making. In contrast, we give stronger results which demonstrate when human judgment can improve the performance of any model in a rich class of possible predictors (Section 4), or when a single algorithm can complement many heterogeneous users (Section 6). 1We elaborate on connections to [16] in Appendix D. 2Code to replicate our experiments is available at https://github.com/ralur/heap-repl. 2 Performative prediction. A recent line of work studies performative prediction [41], or settings in which predictions influence future outcomes. For example, predicting the risk of adverse health outcomes may directly inform treatment decisions, which in turn affects future health outcomes. This can complicate the design and evaluation of predictive algorithms, and there is a growing literature which seeks to address these challenges [42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54]. Performativity is also closely related to the selective labels problem, in which some historical outcomes are unobserved as a consequence of past human decisions [55]. Though these issues arise in many canonical human-AI collaboration tasks, we focus on standard supervised learning problems in which predictions do not causally affect the outcome of interest. These include e.g., weather prediction, stock price forecasting and many medical diagnosis tasks, including the X-ray diagnosis task we study in Section 5. In particular, although a physician’s diagnosis may inform subsequent treatment decisions, it does not affect the contemporaneous presence or absence of a disease. More generally, our work can be applied to any “prediction policy problem”, where accurate predictions can be translated into policy gains without explicitly modeling causality [56]. Algorithmic monoculture. Our results can be viewed as one approach to mitigating algorithmic monoculture, in which different algorithms make similar decisions and thus similar mistakes [57, 58]. This could occur because these systems are trained on similar datasets, or because they share similar inductive biases. We argue that these are precisely the settings in which a “diversifying” human opinion may be especially valuable. We find empirical evidence for this in Section 5: on instances where multiple models agree on a prediction, human judgment adds substantial predictive value. Multicalibration, omnipredictors and boosting. Our results make use of tools from theoretical computer science, particularly work on omnipredictors [16] and its connections to multicalibration. [59] show that multicalibration is tightly connected to a cryptographic notion of indistinguishability, which serves as conceptual inspiration for our work. Finally, [60] provide an elegant boosting algorithm for learning multicalibrated partitions that we make use of in our experiments, and [61] provide results which reveal tight connections between a related notion of “swap agnostic learning”, multi-group fairnes