There’s no doubt that one of the hottest trends in business (legal and otherwise) is artificial intelligence (AI) and machine learning (ML). Nearly all legal technology vendors are looking for creative ways to add AI/ML components to their existing software. Meanwhile, major corporations like Microsoft, Google, and Amazon are investing in commoditizing ML utilities, thereby reducing entry barriers for small-to-medium-sized businesses. We’re entering a new and exciting digital revolution that will be equally if not more impactful than the Internet or even the original development of computers.

This is the first of a set of blog posts discussing the use and applicability of AI and ML in the legal setting. Before we get there, it’s important to have a baseline understanding of the differences between AI and ML and to know what to look for when evaluating these tools for legal use.

What’s AI/ML, and What Isn’t?

AI is often conflated with ML, but technically, they’re different concepts. ML is the use of algorithms to determine the best approach for a computer to reach a conclusion. Basically, you supply the variables and the answer, and the computer determines the equation. A common example of ML is predicated on the sinking of the Titanic; an algorithm accounts for demographic information (age, gender, ticket class, and cabin location) to guess whether or not you would’ve survived. If you want to try this yourself, Microsoft has a convenient tutorial using this example. AI, however, is far more robust, and the definition is in the name. True AI is basically a computer acting as an artificial human, capable of intelligently responding to various inquiries. The closest we’ve come so far are ubiquitous digital assistants, such as Amazon’s Alexa, Google’s Assistant, Apple’s Siri, and Microsoft’s Cortana. These tools use deep learning techniques to see, hear, and respond appropriately.

Despite these technical differences, both AI and ML tools tend to get lumped together under the AI title, which is easier for the untrained person to comprehend. That leaves us with the question of what isn’t AI. Keyword searches, regular expressions, and complicated, self-created workflows are sometimes referred to as AI but aren’t (nor are they ML or deep learning, for that matter). A simple way to explain the difference is that with AI/ML, you know the source and the destination, and you let the computer figure out the best way to go from point A to point B. Non-AI tools involve someone specifically programming the path of the process. There’s nothing wrong with keyword searches, regular expressions, and thoughtfully designed workflows. However, if a vendor slaps “AI” on a tool and the price increases by 50%, the consumer is misled. Be wary of vendors offering AI with no added value.

How Accurate Is Accurate Enough?

Once you determine that the tool you’re evaluating is truly an AI/ML tool, you can get into more subjective areas about what’s good and bad. Unfortunately, there’s no simple answer to this question, but you can develop one based on your business needs.

One phrase you’ll hear from vendors is “the quality of the model.” By “model,” they basically mean how the computer was trained to answer a specific question. A model’s quality is evaluated based on its accuracy. For example, when you’re trying to use an AI algorithm to route tech-support calls to the proper end user, if the call is directed to the right person 80% of the time, then it’s 80% accurate. In other cases, such as when an AI algorithm is trying to estimate sales figures, the closer it is on average, the better the accuracy. Note that 100% accuracy is never going to be realistic, and I immediately discredit any vendor who pretends otherwise.

Now, the tough part: how accurate is accurate enough? I wish I could simply give you a number; however, it’s not that simple. Using the tech support scenario as an example, let’s say the algorithm routes the calls correctly 60% of the time. That’s not great, but if the existing process only has a 50% accuracy rate, then it’s an improvement and could be a worthwhile investment.

It’s important to note that AI algorithms not only provide a predicted answer, they also provide a confidence level. A confidence level tells the end user how accurate the algorithm thinks it is in its prediction. Make sure your legal technology vendor exposes this information, as it’s hugely useful. Using the tech support example again, if human accuracy is 70% and the model’s is 60%, you’d probably forgo the tech. However, if you automatically route items with very high confidence levels (say, 80% or above) and have low confidence items reviewed by a person, you can potentially reduce the number of initial contacts your tech support team has to make. Depending upon the volume of calls and the cost of the utility, you can potentially save money and time even if the model is only somewhat accurate.

My recommendation when evaluating an AI model is to never expect perfection. Go in with an understanding of what your minimal accuracy threshold is. Then, weigh that threshold with an understanding of how much overhead can be reduced if all the “most certain” answers were fully automated.

Who Owns the Model?

So if the legal technology tool you’re evaluating passes your accuracy test, will it always be this accurate? The first way to look into this is to determine who owns the model. If the vendor owns the model, find out how often it retrains it and how it maintains the tool’s accuracy. If you have an algorithm that looks at a picture and says whether it’s a cat or a dog, there isn’t much need to regularly retrain it, as cats and dogs rarely change. But if you have an algorithm that predicts traffic situations, you’ll want to be sure that it’s constantly retrained, as road conditions constantly change. Vendor-owned and properly managed models are great for most business situations; however, they tend to be generic and don’t work as well as a model custom-tailored to your business.

Custom models allow a business to create AI using its own data, ensuring that the results are properly skewed according the differences in how your business operates relative to another. So why don’t we use custom models exclusively? Primarily because the cost, burden of accuracy, and risks are transferred from the vendor to the business. In some cases, these custom models are worthwhile investments, but in others, they’re huge burdens. For example, Microsoft has a utility based on a model it created; called Computer Vision, it can identify items in images. This tool works well in most situations—but not all. If you’re a manufacturer that makes 10 different kinds of widgets and you want a camera to be able to automatically distinguish them, I’d recommend the custom-model route so you can fine-tune your algorithm on just the stuff that matters. This could dramatically improve accuracy. With a custom model, you can retrain at your discretion if the specifications on your widgets change, if new items are introduced, or if the accuracy decreases.

Anything Else (Am I Done Yet)?

Not quite. Now that you’ve evaluated the AI/ML aspect of the tool, you should put it through the same rigor and vetting as any other legal technology solution. Confirm whether the user interface works as expected and that it produces an acceptable user experience; then, gauge that user acceptance. Keep in mind: user acceptance is different for an AI/ML tool than it is for other solutions. Many end users tend to be afraid the machine will take their job and may resist using an AI/ML tool. On the other hand, others may refuse to use a tool that isn’t 100% accurate.

In future posts, we’ll discuss how you can prepare your business and help users understand and embrace the new AI frontier.