Legal Tech · Sample Case Study
From hallucinating prototype to governed RAG, 94% fewer factual errors
A legal-tech scale-up building contract-review software
- fewer factual errors vs. the prior pipeline
- 94%
- fewer factual errors vs. the prior pipeline
- p95 query latency
- 3.2s → 640ms
- p95 query latency
- enterprise deals unblocked by the new audit trail
- 2
- enterprise deals unblocked by the new audit trail
The Challenge
The product's contract-review assistant answered questions against uploaded contracts using a single-pass embedding search, no re-ranking, no citation back to source text. It demoed well and sold terribly to enterprise legal teams, because reviewers kept catching it confidently citing clauses that didn't exist. Every prospective deal stalled at security review on the same question: how do we know what it told a lawyer is actually in the contract?
Engagement Lead
Engagement Lead — AI Engineering
Engineering Approach
[01]
Audited the retrieval pipeline and traced the hallucinations to their root cause: flat cosine-similarity search with no re-ranking, retrieving semantically related but factually wrong clauses as top results.
[02]
Rebuilt chunking to respect the contract's actual clause and section structure instead of a fixed token window, so a retrieved chunk was always a complete, coherent clause rather than a fragment split mid-sentence.
[03]
Added hybrid dense plus sparse retrieval with cross-encoder re-ranking, and constrained the model to answer only from retrieved text with an inline citation back to the specific clause. When retrieval confidence was low, it refused to answer instead of guessing.
[04]
Layered in PII redaction, prompt-injection guardrails on any text extracted from uploaded documents, and full query-level audit logging so every AI-generated answer had a traceable source and a compliance-reviewable trail.
Business Results
Factual error rate dropped 94% against a held-out set of 500 contract questions, verified by the client's own legal review team, not just an automated eval.
p95 query latency dropped from 3.2s to 640ms after re-ranking and serving optimizations, despite the pipeline doing strictly more work per query.
Passed two enterprise security reviews that had previously stalled specifically on the lack of an audit trail for AI-generated output.
Two contracts that had been blocked pending AI governance sign-off closed within the following quarter.
“The turning point in every enterprise security review was being able to show exactly which clause an answer came from. Before, we were selling a demo. After, we were selling something legal teams could actually defend using in front of their own clients.”
Stop guessing.
Start shipping.
Thirty minutes with a senior engineer. No sales deck, no junior SDRs. Just a technical conversation about what's actually breaking or slowing you down.