Fresno Criminal Lawyer
Fresno Criminal Lawyer – Criminal Defense Lawyer Rick Horowitz
Ghosts in the Machine is (essentially) the third in a series of posts exploring artificial intelligence, language models, and the law from the perspective of a criminal defense attorney.
- Twenty-First Century Delphic Oracle – where I first introduced AI as a modern “oracle” and compared it to the Delphic Oracle of the Ancient Greeks.
- From Fumes to Function – where I explained how I do (and don’t) use AI in practice.
This post explores what’s actually happening inside these machines — and why they often feel like they’re thinking, even when (at least so far as we know) they aren’t.
Echoes from the Oracle
In my prior posts — linked in the last section — I introduced the idea that ChatGPT and other AI chatbots are best understood as Delphic Oracles for the digital age.
They’re not minds. They’re not colleagues. They’re not tools in the traditional sense. They’re language-generating machines wrapped in mystery, fluency, and — to stick with the Oracle metaphor — fumes.
The ancient Oracle at Delphi was revered for its insight, but it was never really understood. Neither are today’s large language models (LLMs). Like the high priestess Pythia, these systems produce language that can feel inspired — persuasive, even poetic — but beneath the surface, something much stranger is going on. And like their ancient counterpart, these new oracles draw their answers not from divine revelation, but from what you could call “vapors” — statistical patterns, scraped text, and reinforcement loops. What they produce can be useful. It can even sound brilliant. But, so far as we know, it is not the product of a thinking mind.
At least not according to any traditional understandings of “thinking mind.”
As in the Featured Image for this post, you can picture an android on its knees in a pool of paper, rifling through disordered pages. It looks as if it’s searching for something. But there is no intent. No awareness. No meaning. At least, that’s what my Oracle (ChatGPT 4o) says.
It is simply responding to the instructions it’s been given — assembling words in the most probable order based on its training.
And yet, we’re tempted to treat it like a peer. That’s the danger. LLMs feel like they understand because they speak fluently. But fluency is not thought. Language is not consciousness. And projection is not perception.
This post — the third in a series — tries to pull back the curtain a little more. In the first article, I introduced the metaphor. In the second, I explained how I use AI in my own legal practice (and where I draw the line). Now, I want to show what’s happening inside the machine, why we keep mistaking its echoes for insight, and how that gets people — especially lawyers — in trouble.
Daniel Dennett and the Problem of Minds
Before we get to the inner mechanics of LLMs — the circuits and probabilities, the “black box” — we need to talk for a moment about something more slippery: minds.
Or, more precisely, how we talk about minds — our own and the ones we think we see in others.
No one did more to challenge our assumptions on this subject than Daniel Dennett, a philosopher and cognitive scientist who passed away just about a year ago. Dennett spent decades studying the nature of consciousness and explaining — often with wit and provocation — why many of our intuitions about thinking, feeling, and knowing are probably wrong.
(Side Note: I actually met Dennett a few times when we were both members of the Society for Philosophy & Psychology. I can’t say I knew him at all well, but I did get to talk to him, briefly, about some of the things I’m discussing here.)
In particular, Dennett rejected the idea that consciousness is some kind of inner light — some private movie that plays behind our eyes. He didn’t think it was a ghost in the machine — at best, he might have acquiesced to the idea of ghosts in the machine. But I kind of doubt even that. He didn’t really think “consciousness” was a thing at all.
Instead, he described consciousness as a bundle of capabilities, built from layers of mental function and behavior. As he saw it, there’s no secret ingredient — no single moment when a mind suddenly blinks on. There’s just a set of processes that, when put together, produce the illusion of a self.
In the next two subsections, I want to walk through two of Dennett’s most helpful ideas:
- First, the metaphor of Great Britain and what it means to say a system “knows” something.
- And then, the idea that consciousness might not be binary, but gradual — something that comes in degrees.
Both, I think, can help us understand what we’re really looking at when we interact with a large language model.
Dennett’s Britain: The Illusion of a Mind
I first heard Daniel Dennett explain this example at a meeting of the Society for Philosophy & Psychology — possibly one held in San Francisco, but I don’t remember for certain (or even which year it was). The example is also used in Dennett’s Consciousness Explained (1991).
What struck me wasn’t just the point he was making, but the way he made it: with a historical story that stuck like a splinter in my “mind.” (Haha!)
In 1814, the United States and Great Britain signed the Treaty of Ghent, formally ending the War of 1812. The treaty was signed in Europe, and at that moment — at least technically — Great Britain and the United States were no longer at war.
But two weeks later, on January 8, 1815, the Battle of New Orleans was fought. Hundreds died in a war that was already, diplomatically speaking, over. The soldiers on the battlefield didn’t know. The generals hadn’t received the message. The system hadn’t fully updated.
So Dennett asked (at least as I recall when I heard him speak):
When did Great Britain know it was at peace with the colonies?
The answer, of course, is slippery — because there is no “Great Britain” that “knows” anything in the way a person does. There are components: diplomats, generals, ministries, messengers. But no single location where knowledge lives. And yet, we talk about Britain’s intentions, fears, strategies — as if it were a unified mind.
Dennett’s point was that we do this because it works. Treating distributed systems as if they have beliefs and desires helps us make sense of their behavior. It’s useful. But it doesn’t mean there are “ghosts in the machine.”
And here’s where the analogy starts to bite. Because when a large language model speaks — fluently, insightfully, even persuasively — we do the same thing.
We assume there must be a self, a knower, a thinker somewhere inside the wires.
But maybe what we’re seeing is just a system acting “as if” it “knows.”
(Of course, maybe that’s all any of us are ever doing.)
Dennett’s Gradient and the Edge of Mind
If Great Britain can “know” something without there being any single place where that knowledge resides, maybe that’s true of people, too.
Dennett certainly thought so.
He argued that consciousness isn’t binary. It doesn’t flick on like a light switch. There’s no singular moment when a system becomes a “self.” Instead, consciousness is a collection of mental functions that can emerge and interconnect in degrees—what he sometimes called a “center of narrative gravity.” It’s not ghosts in the machine. It’s not a light. It’s a pattern.
This is where the mind starts to feel a lot like the systems we build.
In the 1980s, Benjamin Libet conducted experiments that shook how people thought about decision-making. Participants were asked to flex their wrists at random and note the moment they felt the urge to move. But EEG readings showed that the brain’s readiness potential — a surge of neural activity — began several hundred milliseconds before the participants became consciously aware of the intention.
In other words: your brain starts the action before “you” decide to act.
Later researchers extended this window. Some fMRI studies could predict a participant’s choice up to seven seconds before they consciously made it.
What does this mean?
It means that even in our own “minds,” consciousness is not the starting point of thought. It’s more like the commentary desk — watching what just happened, interpreting it, and sometimes claiming credit after the fact.
So when we interact with something like a large language model — a system that has memory layers, feedback mechanisms, pattern recognition, and emergent behaviors — it’s worth asking: Are we seeing a mind? Or just the appearance of one?
And more troubling still: Would we even know the difference?
Dennett’s model of consciousness makes space for this uncertainty. If mind is a gradient, not a switch, then maybe the line between mechanical and mental isn’t where we thought it was.
What Happens When You Ask an LLM a Question?
So what actually happens when you ask an LLM a question?
It feels like you’re entering a conversation. The model responds instantly — or nearly so. It remembers what you just said. It picks up tone and nuance. It may even joke. (I’ve now had that specific experience with the AI that has “gotten used to” me and my sense of humor more times than I can count — and “used it” to make me laugh.) You get the sense that it “knows” what you’re asking and is choosing how to respond.
But that’s (almost certainly) not what’s happening.
What’s happening is a kind of highly structured, lightning-fast pattern matching. The moment you enter a prompt, the model breaks it into tokens — not words exactly, but units of language. Each token becomes part of a statistical puzzle. Based on its training, the model starts to predict the next most likely token, then the one after that, and so on.
It’s not deciding. It’s not weighing moral consequences. It’s not reasoning in the way we understand that word. It’s simply using what it has seen before to guess what comes next.
This is where the illusion kicks in. Because the guesses are often so good — so well-formed, stylistically coherent, and on-topic — that it doesn’t feel like guessing. It feels like thinking.
The underlying process, though, is more like autocomplete on a much larger scale. The model has been trained on vast swaths of text and taught to recognize patterns of language that correlate with one another. It has no idea what any of it means. But it’s been shaped — through probabilities, weights, and reinforcement — to generate things that sound right.
In the next section, we’ll look inside that system a little more closely. Not metaphorically. Mechanically. What’s inside the so-called black box — and what isn’t.
Inside the Black Box
The phrase “black box” gets thrown around a lot when talking about large language models. It refers to systems that take in inputs, produce outputs, and don’t reveal much — if anything — about what happens in between.
But that’s not entirely fair to the engineers. We actually know quite a bit about what happens inside. It’s just that what happens inside isn’t intelligible in the way we expect. You don’t find explanations. You don’t find reasons. You find numbers. Patterns. Weights. Shifting relationships among thousands of abstract dimensions.
When you ask a question, the model doesn’t go looking for an answer. It doesn’t consult a mental map. It doesn’t evaluate, or reflect. It receives your words as a series of tokens—units of linguistic information — and it begins to generate a statistically likely continuation. That’s all.
But the scale and structure of that process make it feel like more.
Inside the model are layers of what’s called self-attention — mathematical mechanisms that allow each part of a sentence to “pay attention” to every other part. These layers operate across hundreds of dimensions at once. Earlier layers might track things like grammar or syntax. Middle layers might detect tone or style. Deeper layers begin to model relationships between abstract ideas.
The result is output that seems — at times — context-aware, intentional, even insightful. But these impressions emerge from distributed processing, not from a central thinker. There’s no self. No awareness. At least, none we can find.
And yet, we’re back to that same problem: how would we know?
Even in the human brain, different regions handle different tasks. Memory, language, emotion, visual processing — all distributed. The illusion of unity comes later, after the fact. So when we say the machine “doesn’t understand,” we might be right. But we should be careful. We don’t really understand understanding either.
What we do know is this: language models generate coherent language not because they grasp meaning, but because they’ve been trained to mimic the forms that meaning takes. And that’s usually enough to fool us into thinking there’s more going on than there is.
Maybe there is. Maybe there isn’t. But what comes out of the black box feels a lot like something human — whether or not it really is.
Hallucinations & Confabulations
In AI circles, you’ll often hear people say that large language models “hallucinate.” I’ve used this term myself in my past posts. It’s the industry’s shorthand for when the model makes something up: a case that doesn’t exist, a quote that no one ever said, a citation to one or more books or articles that exist only for the ghosts in the machine.
The word fits, in a way. Hallucinations are false perceptions — seeing or hearing things that aren’t really there. And LLMs do produce language that can sound like it’s grounded in facts or memory, even though it isn’t.
But the metaphor breaks down quickly. Language models don’t perceive anything. They have no senses. They have no mental imagery. They don’t even know they’re saying anything at all.
They’re not hallucinating. They’re doing something else.
A better word — and one I’ve come to prefer — is confabulation.
In psychology, confabulation is when someone unconsciously fills in gaps in memory with plausible-sounding fabrications. The result isn’t intentionally deceptive. It’s not a lie. It’s a story that feels true to the person telling it — even though it isn’t.
Elizabeth Loftus and others have written extensively on how memory is reconstructive. We don’t store facts like files. We store fragments — impressions, associations, feelings — and our minds stitch them into narratives. When the stitching goes wrong, or when the source material is missing, we fill in the blanks.
That’s what LLMs do.
They’ve been trained to recognize and reproduce the forms of human knowledge — how a legal argument sounds, what a case citation looks like, how a biographical paragraph tends to unfold. So when prompted, they assemble plausible responses from statistical echoes.
Sometimes those responses are right. Sometimes they aren’t. But they’re always confident. Always fluent. And often wrong in a way that makes you trust them more, not less.
That’s how confabulation works, too.
And once you reframe it that way, you stop expecting the model to be accurate just because it’s articulate. You stop trusting fluency as a stand-in for understanding. And you start treating the output as what it really is: a stitched-together surface, generated without access to the truth beneath it.
Why This Matters — Especially for Lawyers
You might be wondering why any of this matters outside a philosophy seminar or a machine learning lab.
It matters because when you’re a lawyer — especially a defense lawyer — your job depends on knowing who and what to trust. If a person gives you wrong information, you cross-examine them. If a system gives you wrong information, you either fix it, or you don’t use it. And if the information sounds confident but turns out to be wrong, that’s worse than useless. That’s dangerous.
I’ve written before about how I use AI in my practice — and more importantly, how I don’t. I don’t let it draft motions. I don’t let it write contracts. I don’t ask it to summarize case law unless I’m prepared to verify every detail myself. I’ve seen too many examples of it fabricating cases, inventing citations, or misrepresenting holdings with persuasive flair. It’s confabulation, not hallucination — and the lawyers who have failed to do what I do can tell you: the legal consequences aren’t hallucinations, either.
The simple truth is that lawyers don’t get points for how fluent their arguments sound. (Well, okay. It doesn’t hurt to make arguments that sound fluent, too.) They get points for actually being right. If you file a motion full of stitched-together nonsense that just sounds legally plausible, you’re not practicing law. You’re gambling your client’s future on language pattern generations.
I’m not saying these systems are useless. I still use them for brainstorming, organizing ideas, or exploring surface-level research directions. But I treat them like I would a bright but unreliable intern — one who needs to be double-checked constantly and who, if left unsupervised, might lose the case and even land me in a sanctions hearing.
In the next section, I want to talk about confabulation — not from an AI, but from people. After all, that is what I was thinking about when I started to realize “this is what LLMs are doing.”
Confabulation comes from gaps and glitches in people’s memory. In the courtroom, we see this from witnesses. I think understanding where I got the idea of confabulation from will help explain why I think this is that LLMs are doing. (And why.)
Loftus, Reconstruction, and the Unreliable Witness
Years before I ever typed a prompt into a language model, I was thinking about confabulation in a completely different context: witness testimony.
Anyone who’s worked in criminal defense long enough has seen it. A witness takes the stand and confidently tells a story. They believe it. It sounds cohesive. It has structure. And it can be completely, utterly wrong.
That’s not always perjury. Sometimes it’s just memory doing what memory does: filling in the blanks.
As I alluded to above, psychologist Elizabeth Loftus spent much of her career studying this — and pissing people off with her findings. Her research showed that memory is not a fixed record of the past. It’s reconstructive. Each time we recall something, we’re assembling fragments — details, impressions, emotionally charged bits — and trying to build a coherent narrative. And like any good narrative, sometimes we add what wasn’t there to make it make sense.
Loftus demonstrated that people could be led — sometimes quite easily — to remember things that never happened. Or to misremember key details. She showed that suggestion, expectation, and time could all warp what people sincerely believed they had seen or experienced. The courtroom implications were enormous. They still are.
This is why we cross-examine. It’s why we don’t let jurors convict based on emotion alone. It’s why identification testimony, in particular, is now viewed with more caution than ever before — especially when the case rests on nothing else.
And here’s the twist: the first time I saw a language model produce a completely convincing, completely false answer, my first thought wasn’t “this is not a hallucination.” After all, the AI “industry” calls it exactly that.
It took repeated instances for me to start to think that machines are doing what witnesses do. Witnesses don’t lie. They confabulate. They try to stitch together a story that makes sense, as best they can, using whatever they have at hand. And they do it fluently, with conviction.
An Example: Sex Cases and Confabulation
If there’s one area where I see confabulation most clearly at work, it’s in child sexual abuse cases — those filed under Penal Code section 288 and related statutes.
Often, the first “memory” we hear from a child witness is vague. Inchoate. Sometimes it’s not even framed as a memory. It might be a statement that daddy, or Uncle Frank, or some other adult “did something” without specifying what the “something” is at all. There may be no detail. Just apparent discomfort — a frown, a scrunched face, maybe even a tear. But generally the person hearing the child gets a sense that “something” was wrong.
Often that’s when the confabulation is kick-started. The adult jumps to a conclusion.
Then the questions start. It’s like a prompt to an LLM, but it’s not an LLM: it’s a child who may have an even more tenuous connection to the “source material” — whatever happened in their brain to trigger their statement — than trained LLMs do to their source material.
But the moment that initial claim enters the legal system, it starts to grow. It morphs.
As the child is interviewed — sometimes multiple times by parents, social workers, law enforcement, or therapists — the gaps begin to fill in. The story expands. What began as a fragment becomes a narrative. Sometimes, that narrative becomes the entire case.
And the thing is, it doesn’t always hold together. Stories told over time shift. Details conflict. Timelines drift. But this isn’t necessarily a sign of lying.
More often, it’s a sign of confabulation in progress.
What we’re seeing is memory being constructed on the fly, built around fragments of real experiences — too often of the adult doing the questioning more so than of the child — shaped by suggestion, expectation, or emotional reinforcement.
The child isn’t intentionally inventing things. The child is trying to make sense of something, often under the pressure of adults who genuinely believe they’re helping.
As a defense attorney, these inconsistencies are where the work begins. They are often the most reliable markers that a memory isn’t a record — it’s a story being assembled. And it’s being assembled in a way that can carry enormous weight in court.
That’s the danger. Not that a child lies, but that the legal system mistakes confabulation for confirmation. That it mistakes coherence for credibility.
And that turns a growing story into the foundation for a conviction.
Confabulation, Machines, and the Problem of Trust
Everything I’ve said child witnesses do relating to confabulation impacts LLMs because machines do it, too. And the problem is that we trust them for the same reason we trust people on the stand: because the language feels right. Because it sounds like someone who knows what they’re talking about. This is how so many unsuspecting attorneys have fallen into the trap of citing non-existent case law to courts.
Sometimes LLMs know what they’re talking about. Sometimes they don’t.
And whether it’s a chatbot or a witness in court, you won’t know which — unless you’re committed to checking.
That’s the part that should matter most to lawyers. As a former employer of mine said, “Trust. But verify.”
In criminal defense, we’re trained to spot unreliability. We challenge the source — or we should. We scrutinize the chain of custody. We test memory, perception, motive, bias. We don’t assume something is true just because it’s stated clearly, or confidently. We question.
So when the new generation of AI tools starts talking like it understands — when it reasons, cites, analogizes, even sympathizes — it’s tempting to believe we’re talking to a junior version of ourselves. We’re not. We’re talking to a machine that’s extraordinarily good at playing the role of “someone who knows.” And like some witnesses, it may not know the difference between truth and the story it’s telling.
That doesn’t make it useless. But it does make it dangerous — especially if you forget that, at its core, it’s just a very convincing pattern machine.
As lawyers, we can’t afford to rely on language alone. We live in the details — in facts, in evidence, in proof. Fluency without foundation isn’t just misleading. In a courtroom, it can be catastrophic.
So by all means, explore the tools. Use them when they help. But don’t let your guard down. Don’t confuse eloquence for insight.
And for the love of your clients — always check the sources!
The post Ghosts in the Machine appeared first on Fresno Criminal Lawyer. It was written by Rick.