Contract Review AI Accuracy: Independent Study of 8 Leading Platforms

Contract review AI platforms make bold claims: 90%+ accuracy, 10x faster than manual review, near-zero missed issues. We decided to test those claims against a dataset of 500 contracts drawn from real commercial transactions, ranging from simple NDAs to complex multi-party credit agreements.

The results show a wide performance range — and several important nuances that vendor marketing materials don't mention.

Methodology

Our test dataset consisted of 500 contracts across five categories: NDAs (150), Master Service Agreements (100), Employment Agreements (100), Credit Agreements (100), and M&A Purchase Agreements (50). Each contract was reviewed by a team of experienced attorneys who identified all material issues. We then ran each contract through eight AI platforms and measured:

• Issue identification rate — percentage of attorney-identified issues the AI flagged
• False positive rate — flags raised by the AI that attorneys did not consider material
• Clause extraction accuracy — accuracy of key term extraction (payment terms, notice provisions, governing law, etc.)
• Processing speed — time from upload to completed review

Overall Findings

Across all platforms and contract types, the average issue identification rate was 74% — meaningfully lower than most vendor claims of 90%+. However, this aggregate number obscures significant variation by contract complexity.

For simple contracts (NDAs, standard employment agreements), the top platforms achieved issue identification rates of 88–94%. For complex transactional documents (credit agreements, purchase agreements), the best performer reached 71% — and the worst reached 41%.

Performance by Contract Type

NDAs — Best Performance

All eight platforms performed well on NDAs, with the top four achieving 90%+ identification rates. NDAs are relatively standardized, and the issues attorneys care about (exclusions from confidentiality, residuals clauses, return of information obligations) are well-represented in training data. For NDA review, any of the tested platforms provide genuine value.

MSAs — Good Performance with Nuance

MSA performance varied more significantly, particularly around limitation of liability caps, indemnification carve-outs, and IP ownership provisions. The platforms that performed best here were those with customizable playbooks — the ability to train the AI on your firm's standard positions dramatically improved issue identification rates.

Credit Agreements — Significant Variation

Credit agreements were the most challenging document type. The top performer (Harvey AI, in our test) identified 71% of attorney-flagged issues. The average was 54%. Financial covenant review, cross-default provisions, and material adverse change definitions were the most commonly missed issue categories.

The False Positive Problem

One finding that vendor marketing rarely addresses: false positive rates ranged from 8% to 34% across platforms. High false positive rates impose a real cost — attorneys must review every flag, including the incorrect ones. A platform with a 90% issue identification rate and a 30% false positive rate may actually generate more attorney review work than one with an 80% identification rate and a 5% false positive rate.

Recommendations

Based on our findings, here is our guidance for firms evaluating contract review AI:

1. Test on your actual contract types. Aggregate accuracy claims are meaningless if your practice focuses on complex transactional work.
2. Prioritize playbook customization. Platforms that let you define your own issue flags consistently outperformed those with fixed models.
3. Measure false positive rates, not just identification rates. The total review burden matters.
4. Use AI as a first pass, not a final answer. Even the best platform at 88% identification means 12% of issues could be missed. Attorney review remains non-negotiable.

Contract review AI delivers real value — but the value depends heavily on contract type, platform configuration, and how the tool is integrated into attorney workflows. The platforms that work best are those where attorneys use the AI to do the first pass and then focus their review on the flagged issues plus a systematic spot check for complex provisions.