In the world of AI, particularly in legal tech and contract analysis, accuracy is often the golden standard. It’s the first question people ask: How accurate is your model? But is this the right question? While accuracy is essential, it doesn’t tell the whole story—especially in complex, real-world applications where nuance, context, and usability matter just as much.
So why does accuracy dominate the conversation?
While accuracy is important, we should be asking deeper questions. Here are some other key metrics or dimensions that can provide a fuller picture of an AI system’s value:
1. Scope of data (Depth of Coverage & Granularity)
It’s one thing to be highly accurate on the narrow slice of concepts but contracts are complex, and legal teams need tools that can more fully capture what's in them. This where depth of coverage and data granularity become more relevant metrics.
Real-world contracts aren’t limited to a neat list of 20 generic terms like governing law, effective date, or termination for convenience. They’re full of nuanced clauses that reflect business priorities, regulatory obligations, deal-specific risk allocations, and industry-specific language. Think exclusivity carveouts, regulatory change provisions, or indemnity structures — the kinds of clauses that often hide the most consequential obligations and risks.
A tool that extracts 20 standard fields at 99% accuracy might look good on a benchmark, but it's far less helpful when the real business question hinges on a less common — but highly material — concept. In contrast, a system that can identify 100+ critical and nuanced concepts, even at 95% accuracy, delivers far more value. Greater depth of coverage expands the scope of what’s visible, actionable, and ultimately controllable in the contract portfolio.
Granularity, the level of precision and structure in what the system returns, goes hand in hand with depth of coverage.
There’s a big difference between a system that highlights a paragraph and one that can pinpoint the specific obligation, threshold, counterparty, and time frame within that paragraph. The latter is significantly more useful in contract analysis and should accelerate review cycles.
Additionally, it’s worth asking: does the system return a large, unstructured chunk of text, or does it provide precise, actionable extractions? Large, unstructured chunks of text may technically contain the answer, but they still require a human to sift through, interpret, and reformat that information into something usable. That’s not true automation. For example, just saying there is a Limitation of Liability (LOL) clause may not fully address an important risk. If the cap is set very high, for example, or there are very sweeping carveouts, then a LOL clause may be little better than no LOL clause at shielding you from risk.
Granularity turns data into decisions. The more precisely a system can dissect and label the components of a provision, the more valuable it becomes — especially at scale. When evaluating AI tools, don’t just ask what it extracts — ask how much structure and specificity it provides.
In fast-paced environments — Private Equity, Hedge Funds, Asset Management, M&A, or crisis response — timing is everything. A perfect answer that arrives too late is functionally no better than being unable to answer the question. That’s why the speed at which an answer can be provided should be considered when evaluating contract AI tools.
The traditional obsession with accuracy assumes that legal review is happening in a vacuum — that time is unlimited, and the goal is perfection. But that’s rarely the case in practice. When the market moves, when a regulator knocks, or when a deal’s closing window tightens, legal and business teams need rapid clarity. Decision-makers need real-time insights, not perfection delivered too late to act on.
This doesn’t mean accuracy doesn’t matter. It just means speed must be evaluated alongside accuracy — especially when responding to a crisis where the value of the insight diminishes with each passing hour. Legal tech that prioritizes speed to answer doesn’t just streamline workflows; it aligns with how decisions are actually made under pressure.
Consistency of data saves time and reduces risk of missing something. It’s the difference between extracting “laws of the State of Pennsylvania”, “laws of the Commonwealth of Pennsylvania”, “Pennsylvania Law”, and recognizing that they all mean the same thing.
Normalization is about turning varied language into a standardized, predictable set of outputs. Why does this matter? Because contracts are written by different parties, across different jurisdictions, using different styles. Without normalization, your dataset becomes a patchwork of inconsistent labels — technically accurate, but operationally messy. With normalization more powerful searches, meaningful comparisons, and analytics are possible. Normalization turns extracted data into usable data. For example, if you don't normalize party names across a large portfolio of contracts, you might not realize that an amendment is linked to a specific agreement and make decisions based on incorrect data.
It’s not enough for an AI tool to provide an answer — businesses are looking for answers that can be trusted meaning they need to know where that answer came from.
Not all AI hallucinates. Modern AI solutions are now capable of linking every extracted data point back to its exact location in the source document — down to the clause, sentence, or even phrase. This traceability transforms AI from a black box into a transparent partner. It empowers the business to quickly verify findings, confirm context, and defend decisions with confidence.
Without clearly grounding the data, trusting every answer from an AI system is an act of faith. That might work in lower stakes use cases, but not when you're operating on critical deals, regulatory filings, or fire drill exercises to resolve critical business issues. You need to be able to say, “This is the provision. This is what it means. And here’s where it lives in the contract.”
Data traceability also accelerates review workflows. Instead of manually hunting through a PDF to validate an extraction, users can jump directly to the source. That’s not just convenient—it’s crucial when you’re under pressure and need to review results quickly. A solution that provides a traceable path from output to source delivers data that can be trusted by the user.
Another consideration for businesses is whether the AI solution can provide the data your business cares about the most. This is where relevance of data comes into play. It’s not enough to just have data, it must be relevant data based on the context for it to add value.
The notification obligations after a cybersecurity data breach are much more critical than notification obligations for a fund administrator change. The notification obligation for the data breach might require immediate notification and carry significant regulatory, reputational and financial implications. Having this insight enables the business to understand the risks and respond to the incident.
Complexity of answers reflects a tool’s ability to handle legal language not just as data, but as layered meaning. It’s what separates tools built for demos from those built for real-world legal work. The more sophisticated the questions your business needs to ask of your contracts, the more important this becomes.
It’s one thing to pull out simple fields like governing law or effective date. These are relatively easy targets: short, structured, and usually labeled clearly. But contracts are full of clauses that are conditional and context-dependent.
Take for example, termination rights. You might need to capture who can terminate, under what conditions, with what notice period, and whether any fees or penalties apply. A provision might grant termination “for cause,” but only after a 30-day cure period — except in cases of insolvency. These are not one-line answers. They’re multi-dimensional data points buried in legal nuance. Does the accompanying data model handle this complexity?
Now ask yourself: if an AI system returns “accurate” answers — meaning they technically include the right information — but those answers are vague, incomplete, or missing key attributes, are they really that valuable?
Two sets of answers might score 99% on a traditional accuracy test. But are they equally useful? That’s the leading question — and the answer is no. Oh and us humans are not capable of perfect accuracy either.
To data scientists, legal practitioners, and decision-makers: let’s move beyond just asking about accuracy. Let’s start asking questions like:
Ultimately, these questions will lead to better AI solutions—ones that fit real business needs, save time, and reduce the risk of disillusionment. If you’re working on defining metrics for some of these areas, let’s connect and push the conversation forward.