Bias Bias@HFES04 - IHMC Internal

Biased About Biases: The "Theory of the Handicapped Mind" in the Psychology of Intelligence Analysis Robert R. Hoffman IHMC Presented at the 2004 Meeting of the Human Factors and Ergonomics Society, New Orleans, LA, September, 2004 Robert R. Hoffman, Ph.D., Research Scientist Institute for Human & Machine Cognition 40 South Alcaniz St. Pensacola, FL 32502-6008 [email protected] IHMC tel (850) 202-4462 IHMC fax (850) 202-4440 RRH office (850) 202-4418 RRH cell (850) 748-0182 www.ihmc.us Abstract The notion of "reasoning bias" pervades much theory and research in the fields of cognitive psychology and judgment and decision making. A great deal of research in the tradition of laboratory experimental psychology, especially classic researches by Tversky and Kahneman, have led to the widespread belief that human reasoning is biased in a number of ways. Research claims to have demonstrated a number of biases in reasoning about frequency or likelihood, and a number of biases in logical reasoning. This belief dovetails with the "bounded rationality" thesis, and has readily fed over into computer science and applied cognitive science. Many efforts on creating performance and decision aid technologies are premised on a goal, sometimes the single goal, of compensating for human biases. An example is the landmark work, "The psychology of intelligence analysis" by Richards Heuer, which is based upon articles he wrote in 1978-1986, which clearly shows the impact of the research on bias in the recurrent theme of human limitations and weaknesses. Although it can be argued that the machine, in contrast to the human headbone, is unable to cope with either meaning or context, and therefore any artificial compensatory mechanisms or approaches are bound themselves to be limited, this is not the argument that I pursue in this report. One can ask, "What, specifically, does it mean for a reasoning sequence to be biased?" The fact that a reasoning sequence leads to wrong answers is by itself not enough (i.e., the "bias" is not inherently a property of the final decision, but is attributable instead to the process that led to the

decision). A breakdown of the defining features is very revealing. When each entry in the pantheon of biases is compared to the defining features, none of them stands up well under scrutiny. This analysis suggests that many of the so-called biases are perhaps properly not considered biases at all. This calls into question the notion that bias is an aspect of human reasoning for which one should craft compensatory mechanisms, and that such compensatory mechanisms must be provided by machines. This is not to say that there are not phenomena, both laboratory and real-world, that might at first glance be reasonably attributed to "bias." Rather, the argument is that purported reasoning biases are either: (1) artifacts of the artificial laboratory that do not generalize, (2) inevitable consequences of how human learn and comprehend, or (3) rational analyses. Two examples might be helpful. A number of studies have demonstrated the gambler's fallacy, that if a low-probability event has occurred with unusually high frequency over some period of time, that it will likely not occur again. This has been demonstrated using a variety of simplistic laboratory tasks. But change the context: The basketball player has hit five free-throw shots in a row. In this case people are likely to assume the player is "hot" and will continue to make the shots. A number of studies have demonstrated the hindsight bias. The phrase "I knew it all along" seems to be used only in the cognitive psychology textbooks that explain biases, but be that as it may, hindsight analysis is rational: After an event you get additional information. Anyone taken back in time, and given the same information as was initially available, would have made the same judgment. The deconstruction of the notion of bias serves to re-orient efforts at aiding technologies, including aiding technologies for the IC. The assumption that the purpose of aiding technologies is to compensate for biases leads to the creation of models of bias to inform the creation of technologies that compensate for human limitations. An alternative view called "Amplified Intelligence," is that technologies should be based on models of expert reasoning, perceptual, and collaborative capabilities, so that these might be amplified and extended. Aids created from the two differing viewpoints would have different features and uses. I would hypothesize that aids made from the perspective of AI would be more useful.