Some very smart people, and Harvard prawf Cass Sunstein, offer a counterpoint to our current obsession with bias, and tacitly with adoration of empiricism. They call it “noise.”
Society has devoted a lot of attention to the problem of bias — and rightly so. But when it comes to mistaken judgments and unfortunate decisions, there is another type of error that attracts far less attention: noise.
To see the difference between bias and noise, consider your bathroom scale. If on average the readings it gives are too high (or too low), the scale is biased. If it shows different readings when you step on it several times in quick succession, the scale is noisy. (Cheap scales are likely to be both biased and noisy.) While bias is the average of errors, noise is their variability.
Why call it “noise”? Other than Sunstein’s bias toward words beginning with “n,” I dunno. It’s not the word I would choose. It’s not specific, it fails to convey the specialized meaning like “bias” does and it’s far too common and easily conflated with other issues. But then, they’re intellectuals, so what do I know? In any event, their point is valuable.
Although it is often ignored, noise is a large source of malfunction in society. In a 1981 study, for example, 208 federal judges were asked to determine the appropriate sentences for the same 16 cases. The cases were described by the characteristics of the offense (robbery or fraud, violent or not) and of the defendant (young or old, repeat or first-time offender, accomplice or principal). You might have expected judges to agree closely about such vignettes, which were stripped of distracting details and contained only relevant information.
To be fair, I would have never expected judges to closely agree, but then I know sentencing is voodoo and can vary wildly. One factor omitted from their study is the impact of advocacy on the variability of sentences. A curious omission, if common in academia.
So why isn’t this “noise” revealed by our plethora of empirical studies that we are reliably informed will save us from our biases?
Noise causes error, as does bias, but the two kinds of error are separate and independent. A company’s hiring decisions could be unbiased overall if some of its recruiters favor men and others favor women. However, its hiring decisions would be noisy, and the company would make many bad choices. Likewise, if one insurance policy is overpriced and another is underpriced by the same amount, the company is making two mistakes, even though there is no overall bias.
This raises one of the problems that’s intuitively recognized by some of us in the trenches, and ignored by proponents of statistical fixes. You may have a 50-50 split of ex-prosecutor judges and ex-public defender judges (note, not criminal defense lawyers, but PDs, because they’re marginalized and trendy). But if you get wheeled out to one, what difference does it make that the other exists? You get what you get, and the stats may equal out but they won’t do you any good. Individual bias will impact any individual outcome, but the big number will look unbiased. That’s noise.
In the sentencing realm, the United Stats Sentencing Guidelines were an effort to eliminate variables, to quiet the noise. They were horrible, not so much because the concept of bringing consistency to sentencing was a bad idea in itself, but because they were formulated out of whole cloth for political reasons and demonstrated, yet again, the one size does not fit all, Trying to eliminate noise might be a good thing, but the execution of such a task is hardly simple.
A third source of noise is less intuitive, although it is usually the largest: People can have not only different general tendencies (say, whether they are harsh or lenient) but also different patterns of assessment (say, which types of cases they believe merit being harsh or lenient about). Underwriters differ in their views of what is risky, and doctors in their views of which ailments require treatment. We celebrate the uniqueness of individuals, but we tend to forget that, when we expect consistency, uniqueness becomes a liability.
Are we not individuals? Are we not intersectional? Are we not unique? We celebrate rugged individualism on one side and empathy of the other, both of which tend to be highly idiosyncratic. Perhaps we think of ourselves as fairly normal, and that our values are correct and shared by most similarly normal people, but why do so many people disagree with us so strongly when we’re so obviously right? They must all be crazy, instead of smart, decent and normal like us.
Recognizing the existence of “noise” is the precursor to dealing with the variables that produce noise errors. But knowing that a problem exists doesn’t mean there’s a solution to it.
Once you become aware of noise, you can look for ways to reduce it. For instance, independent judgments from a number of people can be averaged (a frequent practice in forecasting). Guidelines, such as those often used in medicine, can help professionals reach better and more uniform decisions. As studies of hiring practices have consistently shown, imposing structure and discipline in interviews and other forms of assessment tends to improve judgments of job candidates.
There’s an old adage, everybody is entitled to their opinion, but they’re not entitled to their own facts. There’s another old adage that opinions are like assholes, everybody’s got one. The irony of noise reduction techniques is that they serve to create consistency at the expense of individualized assessment of variables that differ from person to person, situation to situation, by reducing complex decision making to its lowest common denominator.
There’s another old adage (actually a quote from Ralph Waldo Emerson’s 1841 essay, “Self-Reliance” ), “a foolish consistency is the hobgoblin of little minds, adored by little statesmen and philosophers and divines.” Sometimes, you need to make a little noise. We’re hardly as “unique” as we wish we were, but reducing our world to the mean and medium and ignoring differences that matter is just as foolish.