Noise: A Flaw in Human Judgment

Noise: A Flaw in Human Judgment is a nonfiction book by professors Daniel Kahneman, Olivier Sibony and Cass Sunstein. It was first published on May 18, 2021. The book concerns 'noise' in human judgment and decision-making. The authors define noise in human judgment as "undesirable variability in judgments of the same problem" and focus on the statistical properties and psychological perspectives of the issue. Examples they give include their own finding at an insurance company that the median premiums set by underwriters independently for the same five fictive customers varied by 55%, five times as much as expected by most underwriters and their executives. Another example is that two psychiatrists who independently diagnosed 426 state hospital patients agreed on which mental illness the patient suffered from only in half of the cases and a finding that French court judges were more lenient if it happened to be the defendant's birthday. Kahneman, Sibony and Sunstein argue that noise in human judgment is a thoroughly prevalent and insufficiently addressed problem in matters of judgment. They write that noise arises because of factors such as cognitive biases, mood, group dynamics and emotional reactions. While contrasting statistical bias to noise, they describe cognitive bias as a significant factor giving rise to both statistical bias and noise. The authors write that noise can lead to gross injustices, unacceptable health hazards, and loss of time and wealth. They argue that organizations should be more committed to reducing noise and promote noise audits and decision hygiene as strategies to detect, measure, and prevent noise. Noise: A Flaw in Human Judgment became a The New York Times Bestseller and received generally positive reviews among critics. Common critiques against efforts to reduce noise are that such efforts dehumanize those affected by the judgments and that it can lead to discrimination. Some commentators also questioned the authors' claims about the novelty of the noise concept.

Content

Noise in human judgment Noise: A Flaw in Human Judgment was authored by psychologist and Nobel Prize in Economics laureate Daniel Kahneman, management consultant and professor Olivier Sibony, and law professor and Holberg Prize laureate Cass Sunstein. They write that 'noise' in human judgment presents itself in several forms: disagreement between judges, disagreement within judges, and even in judgments made only once by a person or group, since such a judgment can be viewed as only one possible outcome in a cloud of possible judgments that the judge in question could have arrived at. (Note that the term "judge" in the book denotes any person making an assessment of some kind.) The reasons given by the authors for why noise arises include cognitive biases, differences in skill, differences in 'taste' (preferences) and emotional reactions, mood in the moment, level of fatigue, and group dynamics. The authors consider noise in predictions and evaluations but not in thought processes such as habits and unconscious decisions. They write that whereas it is good to have noise for certain types of judgments, such as matters of taste when it comes to cultural entities (for example film reviews), society should do much to decrease noise in matters of judgment. This is because noise leads to sizeable consequences for example for fairness, health, safety and costs in terms of time and money.

Noise, statistical bias, and cognitive bias Kahneman, Sibony and Sunstein use a shooting range as an analogy to illustrate noise and statistical bias, and how cognitive bias affect them both. Fig. 1 is an adaptation of the same illustration in the book, comparing how noise and bias affect the accuracies of judgments made by a team of judges. (The original illustration comes from an academic article on noise by Kahneman and his colleagues.) Target a): Since all shots here are in or close to the bullseye, the collective judgment is accurate. Target b): Here there is less accuracy, because of statistical bias: the shots are systematically off in one direction. Put differently, on average there is an error. Target c): Here there is no error on average and thus no bias, because the errors from each shot cancel each other out. However, there is much noise, because the shots differ much from each other. Target d): This error is the largest, since it has both bias and noise. Reducing error is most difficult here.

Examples of noise In the book, Kahneman, Sibony and Sunstein provide many examples of noise in human judgment. Beyond the below areas where noise exists, the book also looks at for example performance evaluation and business strategy. However, a recurring statement in the book is that "wherever there is judgment, there is noise, and more of it than you think", so they argue that there is noise in most areas of human decision making.

Criminal law A study on 208 criminal judges showed that their independently given punitive recommendations on 16 fictive cases varied greatly in harshness. For example, the judges only unanimously recommended imprisonment in three of the cases, and while the recommended number of prison years in one case was 1.1 years on average, one recommendation was as high as 15 years.

Education A study looked at 682 real decisions by college admissions officers and found that the officers awarded the academic strengths of applicants more importance on cloudy days and, conversely, favored nonacademic strengths on sunny days.

Medicine One study showed that whereas some radiologists never produced false negatives (missed real breast cancer) when examining mammograms, other radiologists did so half the time. For false positives, the range was 1–64 %.

Recruitment A meta-analysis showed that a quarter of the time, two separate recruitment interviewers disagreed on which job candidate was the best fit for the job. This was despite the interviewers sitting on the same panel, thus having seen the candidates in the exact same circumstances.

Types of noise

… excerpt ends here. Continue reading the full article.