I’ve been taking advantage of some time off work to get some reading done. I tend to look in the business section but am noticing that a lot of the ebooks I find on Overdrive that are readable are in sociology. That’s where I came across Noise and it has been occupying my mind for most of the holidays. The basic idea is that our decision-making is impacted by noise and bias, and the book takes you through examples of noise, particularly using the criminal justice system as an example.

One of our kids is a methodical reader. They read a book, then flesh out their reading list with citations from the bibliography, and so on. Me, I’m a bit more like a really bad weekend fisher. I fire up Overdrive and poke around for books that look interesting. Sometimes I come up with a catch and just as often I don’t.

Noise is a book I already know I’m going to be re-reading. There’s a lot in it and I find myself reading a chapter and unfocusing as I think about how I could apply the concepts. It’s the first book in a long time that I’ve felt worth highlighting as I go.

I’m going to touch on two of those potential applications for a law library director. But anyone who is interested in the criminal justice system may want to give this a read. It talks particularly about how noise in judicial judgment and sentencing can create wildly different, and unjust, results in similar cases.

It also looks at why humans are particularly weak at removing noise from decision-making but repeatable methods – algorithms – can be better. And that, after discussing artificial intelligence and machine learning, explains that their ability to remove noise doesn’t mean they are a definitive answer to the problem.

No one should doubt that it is possible – even easy – to create an algorithm that is noise-free but also racist, sexist, or otherwise biased.

Noise, Daniel Kahneman, Olivier Sibony, Cass Sunstein, p. 455 (ebook)

But, as Tom Sawyer says, that’s too many for me. As I say, I need to go back and read Noise a couple more times. A first read was enough for me to think about how to start using the ideas, though.

Reference Work

One of the things the book breaks down is the difference between predictive and evaluative decision-making. It’s the difference between forecasting next year’s sales and determining the appropriate sentence for a criminal defendant. (Chapter 4) In particular, the book breaks down why similarly situated people – like judges – make different judgments given the same inputs.

Noise in evaluative judgments is problematic for a different reason. In any system in which judges are assumed to be interchangeable and assigned quasi-randomly, large disagreements about the same case violate expectations of fairness and consistency.

Noise (chapter 4, p. 77 ebook)

Reference librarians do not generally make life and death decisions. But our staff do engage in regular evaluative judgments about what information a researcher needs. In theory, a self-represented litigant asking a question at 9am on a Monday is going to get the same answer as a lawyer asking the same question at 2pm on a Wednesday.

They are both seeking information to help them answer questions about a legal issue. But do they get the same answer? The same resources? Do we direct the self-represented litigant to resources in plain English that may be less detailed? Do we assume the lawyer can navigate the looseleafs or online services better than other researchers? Are we confident that all reference interactions are handled equally across the staff?

She gave no indication that – after the initial apprenticeship phase – she had learned to agree with others, had checked to what extent [her responses] did agree with them, or had even tried to prevent her practices from drifting away from those of her colleagues.

Noise (chapter 2, p. 49 ebook)

A reference librarian that I have a lot of respect for has a mental template of the ideal reference response. Once you’ve gone through the interview and identified what, exactly, the researcher needs, there is an output to create. This librarian has a qualitative template that defines a sweet spot – 3 to 5 resources, annotated and hyperlinked – of what they hope to send to each researcher.

What I like about this template is that it creates a repeatable rule. In theory, it has become such a habit for them that, whether the question is first thing on Monday or mid-week, everyone will receive a similarly thorough answer.

But how do we know what reference is doing? Even a solo law librarian may send a wide range of responses depending on what the authors of Noise call occasion noise. Sometimes we lack the time we would like to make a thorough response. Sometimes it’s due to other resources.

Occasion noise is the product of a second lottery. This lottery picks the moment when the professional makes a judgment, the professional’s mood, the sequence of cases that are fresh in mind, and countless other features of the occasion.

Noise (chapter 7, p. 114 ebook)

One thing that the authors discuss is doing a noise audit to determine the variability in judgments. How different are the responses that your reference staff give to an answer? The concept takes me back to law school legal research class, where we are given a fact pattern that we need to research. Even as potential professionals, we didn’t always get the same answer and, even if we did, we may not have reached it the same way.

It makes me think more about how much templating we could do outside the reference librarians’ head. And about the type of oversight a reference team might exert itself over its output. This might manifest itself as:

  • e-mail templates that are pre-populated with the target number of response items (a list with three bullets, for example, as a reminder);
  • a weekly review of sent reference email, to look for differences in approach, and to determine whether those differences impact quality of answers or customer service
  • a checklist at the reference desk to ensure that responses in certain situations are repeatable. Directional reference? Point to the resource’s location, explain the content’s purpose, leave the reference desk and lead researcher to the resource.

Another idea might be to have each reference question reviewed before it goes out. Yes, I completely understand that this creates an enormous amount of friction, even if any of our law libraries had enough people to engage in this sort of editorial review. The authors of Noise cite a situation where a second-level review was incorporated:

[The researcher] suggested … that it would be preferable to mask the first reader’s ratings [on admissions applications] so as not to influence the second reader. The school’s reply: “We used to do that, but it resulted in so many disagreements that we switched to the current system.”

We do not have the resources to do this sort of review in most of our law libraries. But we should also be aware that we may not be rigorous in our review if it is likely to cause inter-personal friction within our teams. Each attempt to remove noise from our judgments can have consequences.

Board Herding

I am constantly on the alert for my tendency to a negative first reaction: why we can’t do something. It causes me to shut up and listen, which has its own added benefits. Law library directors can have an outsized impact on group discussions and input gathering for decision-making. I’ve mentioned “highest paid person in the office (HIPPO)” before.

Chapter 8 of Noise is a fascinating look at the challenges around group judgments, including social influences and early movers. You start to see how easy it can be to influence a group merely by creating enough momentum (downloads, up votes) to influence movement in one direction or another.

I was also struck by information cascades. And I immediately envisioned how this works at a board of directors or board of trustees meeting. This can be particularly fraught for law libraries with governance boards.

Board members bring value to law library governance. Each one should, in theory, have a set of skills and background that makes them able to help the law library director. It is a tricky balance because the law library director also brings subject-matter expertise to the board table. Everyone at the table needs a bit of humility about what they’re good at and what they’re not, and to keep in mind what the Board’s role really is: strategy, accountability oversight, and not operations.

I’ve had the opportunity to see a lot of governance boards in action. After you’ve been at an organization long enough, you may actually see the board membership change. At my current organization, there’s a bit of change every 2 years. At the ABA, there was an annual influx and outflow of participants.

The challenge a law library director and Board chair face is uninformed consensus. The director needs to worry about this as it can complicate the ability to successfully make operational decisions. The Board chair needs to be aware of it to ensure that the board is making the best decisions possible.

Information cascades can make this decision-making worse. If you have a board member who always speaks first – regardless of their knowledge of the topic at hand – you may find that this first input shifts how the Board responds. Subsequent speakers or other decision-makers may infer that earlier or more ardent speakers have information that they themselves lack.

[I]nformational cascades make noise across groups possible and even likely…. [P]eople tend to neglect the possibility that most of the people in the crowd are in a cascade, too…. We might think that their shared agreement reflects collective wisdom, eve if it reflects the initial views of just a few people.Noise (chapter 8, p. 143 ebook)

Lawyers and judges and other board members use law libraries. They do not necessarily understand what a law library does. If your law library board does not change frequently, you can miss out on new perspectives and skills. But you can probably be confident that they, at least, understand how the law library operates.

I’ve seen information cascades at the board level blow an issue completely off course. It doesn’t take much and, if there are other, latent divisions in the board, you may find a topic become a proxy for some other problem the board is experiencing. Board decisions have impact and bad decisions will have negative impacts for the law library.

The director isn’t the chair, though. So how do you help to derail information cascades that are unhelpful as a bystander? This is what I’ve been thinking about doing in group discussions, whether with staff or a board or with peers in a professional association:

  • note who speaks first on a topic. Is one person always the first person out of the gate, on every issue? We have someone at my current organization who is like that, and he both wastes the group’s time and contributes no value because he mistakes enthusiasm for expertise. If it’s always the same person, take action (or highlight it for the board chair after the meeting) to try to bring in other voices earlier.
  • understand who has what expertise even if they do not overtly share it. This requires interacting with board members and others outside the meeting. This makes sense politically, to understand unstated currents underlying the board’s makeup. But you may find out someone has an idea or expertise that they may not share without prompting.

Fortunately, this overlaps with other goals of group interaction. If you’ve got people or representation who tend to be underrepresented or unheard, you can steer the discussion to them. It can be easier to just let the discussion happen but I think committee and board chairs, and directors in staff discussions, should be more mindful about their role. Just as I try not to ever speak first in a discussion, to avoid constraining ideas, I can make sure everyone has a chance to speak and to break up cascades.

And information cascades aren’t all bad. If your experts speak first and thoughtfully, they can save the board time. A consensus decision can reflect that a lot of the people at the table may not have the same subject matter expertise. But law library boards are frequently populated by law school graduates, who may not be self-aware about their subject matter expertise limitations and unable to admit not knowing. A bit of vigilance on the part of the law library director and board chair can help to avoid uniformed cascades.

I’m going to wrap up. This post is plenty long and reflects, perhaps, how interesting the concept of reducing noise in decision-making is for me! I used to buy books for my governance board – like a book club, so everyone could read the same things – and this is the first time in a long time that I think I’ve found a book that everyone should read.