I mention AI and the law, and I am apt to be jumped by a lawyer telling me that AI is not to be trusted—look at the reports of hallucinations.
Get to legal blogs and things get much scarier. How could one find a legal blog credible? Put thousands of legal blogs together, in the form of an LLM, on which tens of thousands of lawyers have published. Crazy.
Well, some LLMs are more reliable than others, depending on how they’re trained, what they’re trained on, and how they’re used.
I’m still learning on the AI front—who isn’t? But here’s a possible outline for evaluating an LLM based in part on legal blogs—or maybe even built entirely on legal blogs.
What Makes an LLM More Reliable?
- Quality of Training Data
- An LLM trained on high-quality, verified content (legal blogs from credible lawyers and law firms whose reputations are on the line) will produce more accurate and grounded responses.
- One trained heavily on noisy web data or low-quality sources will be more prone to hallucinations.
- Alignment and Guardrails
- Some LLM’s undergo extensive training—meaning they’re better at refusing unsafe or speculative answers, and stick closer to their training data’s intent.
- Open-source models may be faster or cheaper to use, but less tightly aligned.
- Specialization or Fine-Tuning
- An LLM tuned on legal-specific data (like lawyer publishing—-blogs, articles, insights et al) will perform far better in legal contexts than a general-purpose model.
- Think of it as the difference between a generalist and a seasoned legal researcher.
In Legal Contexts, “Reliable” Means (Understand that Human Oversight is Still Needed):
- Cites actual sources (or tells you when it can’t)
- Doesn’t invent facts or legal principles
- Handles nuance and exceptions in the law
- Reflects jurisdictional differences when asked
Bottom Line:
We are a ways off from a perfect LLM that’s trained exclusively on legal blogs or an LLM that’s trained in part on blogs. However, things are moving awfully fast in the AI world.
We’re apt to jump from questioning one legal blog article or blog publication at a time to trusting the authority of an LLM trained on legal blogs before long.