Posted by: structureofnews | July 6, 2021

The Fault In Ourselves

Whose fault is it when something bad happens? Who do we hold to account when we find injustice?

It’s a natural – and good – impulse journalists have, to look for wrongdoers when we see wrongdoing, to identify bad actors and uncover bad motives. And long may that continue.

But what happens when injustice isn’t anyone’s fault; if it’s – essentially – everyone’s fault?

That’s at least one of the takeaways from Daniel Kahneman’s interesting new book, Noise, written with Olivier Sibony and Cass Sustein. It’s about how, beyond bias, discrimination and prejudice, simple randomness – caused by human frailty (“A flaw in human judgment”) – can also lead to injustice and inequality. And it offers a lesson for journalists: We should look for and expose systems that are failing, regardless of whether there’s systemic bias in them.

The book offers example after example of how what should be consistent judgements made on the merits of an argument often aren’t, even when the people making those calls are trying hard to be impartial and fair.

A study of thousands of juvenile court decisions found that when the local football teams loses a game on the weekend, the judges make harsher decisions on Monday (and, to a lesser extent, for the rest of the week.) Black defendants disproportionately bear the brunt of the increased harshness. A different study looked at 1.5 million judicial decisions over three decades and similarly found that judges are more severe on days that follow the local city’s football team than they are on days that follow a win.

Or this:

A review of 207,000 immigration court decisions over four years found a significant effect of daily temperature variations; when it is hot outside, people are less likely to get asylum.

The fundamental thesis of the book – which, admittedly, can be a little hard to get through – is that we don’t spend enough time looking at “noise” in our systems: the factors that produce wide variations in what should be much more standardized decisions. That’s not to say that bias and systemic bias isn’t an issue, and certainly journalists expend a huge amount of effort to find cases of discrimination, both blatant and subtle. But we don’t focus as much on the noise that can cause as much injustice in outcomes. If immigration cases aren’t judged purely on the merits, if the results are somewhat random, that’s just as bad an outcome as if cases are regularly stacked against a type of applicant or group.

I won’t go into all the factors that the book lists as reasons why our systems of reasoning aren’t as robust as we like to think they are; you should read the book. Part of it has to do with the normal human desire to see cause and effect in the world and an aversion to thinking statistically about events.

Some variation in results is caused by the difference is how different people – immigration judges, for example – see the world and the applicants before them. And that could be driven by bias. But other variation can be caused by the events of that day and simply how decision makers feel at that moment.

Uri Simonsohn showed that college admissions officers pay more attention to the academic attributes of candidates on cloudier days and are more sensitive to nonacademic attributes on sunnier days.

Or it could be the sequence of events.

Asylum judges in the United States, for instance, are 19% less likely to grant asylum to an applicant when the previous two cases were approved.

Decision makers sometimes can’t even agree with themselves.

Fingerprint experts – who are supposed to make quasi-scientific decisions about whether prints found at the scene of a crime match that of a suspect – turn out to be influenced by extraneous factors, such as extra information that suggests the suspect is innocent or guilty. And when asked to check on findings they had made months ago, they changed their mind 10% of the time.

A 2012 study commissioned by the FBI replicated this finding on a larger scale by asking seventy-two examiners to look again at twenty-five pairs of prints they had evaluated about seven months earlier… About one decision in ten was altered.

OK, but so what? The world is an unfair, and random place. What’s the value in pointing that out if there isn’t much that can be done about it? For one, it can be useful to show how systems aren’t working, even if there aren’t ways to fix them. But more importantly, as the book notes, there are ways to decrease variability, to drive for more fairness.

It’ll take another post to write about those ways to reduce the noise in systems – not least in making better and less biased hiring decisions – but the main message is simply that bad outcomes are bad, regardless of whether bad actors make them, or the fault lies in all of us. And we should expose them. As the book notes:

System noise is inconsistency, and inconsistency damages the credibility of the system.


  1. The final point about something that damages the credibility of ‘the system’ reminds me of the Gallup poll that has shown a decline in the trust of institutions. Do you get a sense that the authors are talking about institutions or something a bit more vague and overarching? I’ve been thinking quite a bit about the loss of faith in institutions, which to me are constituent parts of the overall system. Finding fault with institutions is relatively easy, but there seems to be less effort in reforming them. Of course there has been some writing about a post-institutionalist society, but most of them seem overly abstract. At any rate, good to read you again.

    I have a lot of pent-up blogging energy, but most of my monthly word quota is taken up with my master’s degree at the moment and for the next six months.

    • Kevin, hey! Nice to hear from you. I think – to your question – it’s both. I think confidence in institutions falls when there’s sense, justified or not, that it fails to do what it says it does. And that “Noise” demonstrates that there are certainly flaws in the consistency of outcomes that we expect from our institutions. And also that it’s not as hard as it sounds to at least improve them, if not reform them.

  2. […] riffing off my recent riffing off on Noise, the new book by Daniel Kahneman, Olivier Sibony and Cass Sunstein: OK, so people are […]

  3. […] And that’s something we should think about too, as we write about complex systems – to resist the temptation to just look for for bad guys but instead to help readers really understand how the world works, even if terrible outcomes are the result of small flaws or human frailty. […]

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s


%d bloggers like this: