Legal AI Tools: If You Can't Tell Me Where the Knowledge Comes From, You Don't Have a Product

Written by Nick Thayer, Lex Tecnica Law Clerk

Image generated with ChatGPT (OpenAI), 2026.

There’s a question that tends to stop my AI vendor conversations cold, and it works in pretty much any industry. Where does your knowledge base actually come from? Not the marketing version. The real answer. What’s in it, how was it sourced, how current is it, and how do you know it’s accurate? I ask this a lot, especially with legal AI tools, and the answers are almost always some version of the same non-answer. “Proprietary legal data.”, “Trained on a comprehensive corpus of industry materials.”, “Curated content from trusted sources.” Sometimes the conversation just pivots entirely and suddenly there’s a new feature to show you, a client logo to name-drop, a benchmark number to throw out. It’s all very smoke and mirrors.

This Isn’t Just a Legal Problem: Why Legal AI Tools Are Hard to Trust

I come at this from the legal world so that’s where my examples stem from, but the same issue exists across every industry where AI tools are being sold as specialized solutions. Healthcare AI that claims to draw from “comprehensive clinical literature.” Financial tools trained on “proprietary market data.” Compliance platforms built on “curated regulatory content.” HR systems with “industry-specific knowledge bases.”

In every one of these cases, the vendor is asking you to make professional decisions based on a knowledge foundation they won’t describe in any meaningful detail. And here’s the thing that most are missing: if the underlying knowledge base is bad, incomplete, or just a general-purpose model dressed up in a vertical-specific interface, then you don’t have a specialized tool. You have a chatbot with better branding. The same foundation model you could access directly for a fraction of the price, with a layer of marketing on top. The pitch sounds sophisticated. The substance is often a lot thinner than that.

The “Proprietary Data” Trap

There’s a version of this that’s especially common right now where vendors use the phrase “proprietary data” as if it’s a feature. It’s not. Proprietary just means they own it or licensed it exclusively (hopefully). It says nothing about whether it’s good, current, accurate, or actually relevant to your use case. In July 2024, Paxton AI released benchmarking data claiming 93.82% accuracy on legal research tasks. Sounds great. When you read into it, that number came from testing against a subset of Stanford hallucination benchmarks that Paxton itself curated, excluding certain task categories due to what they called “specific alignment considerations with our current testing framework.” That 94% ends up doing a lot of heavy lifting in sales conversations. The methodology supporting it does not. This is not unique to that company. It’s a pattern. Self-reported benchmarks, self-selected test sets, accuracy numbers without independent verification. The feature list keeps growing. The foundational question keeps not getting answered.

Why It Matters More Than People Admit

Most people focus on AI accuracy in the sense of “does it give the right answer.” That’s important but it’s actually a downstream problem. The upstream problem is whether the tool has access to the right information in the first place. A model that’s drawing from outdated, jurisdiction-wrong, or just plain incomplete source material isn’t going to hallucinate in an obvious way. It’s going to give you confident, fluent, well-structured answers that are wrong in ways you might not catch. That’s worse than obvious nonsense. Obvious nonsense you throw out. Plausible-sounding wrong answers end up in final products. In legal this is especially stark.ABA Formal Opinion 512, the national ethics guidance issued in July 2024, requires lawyers to have a reasonable understanding of the capabilities and limitations of the specific AI tools they use. Not AI in general. The specific tool. That means knowing what it draws from and where it’s likely to fail. You can’t do that if the vendor won’t tell you. The case Mata v. Avianca is the most visible example of how this goes wrong. Attorneys submitted ChatGPT-generated citations to federal court. The citations looked real. The cases didn’t exist. $5,000 in sanctions and a national cautionary tale, all because someone trusted output from a system without understanding what it was drawing from.

Legal gets the dramatic examples because the stakes are public and the consequences are documented. But the same dynamic plays out quietly in every industry where people are deploying AI tools they don’t fully understand against decisions that actually matter.

What Transparency Looks Like When It’s Real

The vendors who actually have a solid knowledge foundation are usually not vague about it, because specificity is a selling point, not a liability. There’s a real difference between “We retrieve from federal case law indexed through PACER, updated weekly, covering all circuits...” and “Proprietary legal AI trained on comprehensive legal materials.” One of those you can evaluate. The other you’re just supposed to trust. The tools worth taking seriously are the ones that show their work. Citations linked to actual source documents. Explicit confidence indicators. Honest disclosure of what’s in scope and what isn’t. That’s not a high bar. It’s the minimum you’d expect from any source you’d cite professionally.

The Question to Ask

Before deploying any AI tool in a professional context, ask directly: What is your knowledge base, where does it come from, how current is it, and what’s out of scope? If the answer is specific, that’s a vendor that understands your professional obligations and has built something worth evaluating. If the answer is vague, or if the conversation suddenly has a lot more features to show you, that’s telling you something important about what’s actually under the hood. The more impressive the demo, the more important it is to figure out what’s holding it up. And “we work with [insert big firm / hospital / bank name]” is not an answer. A client logo is not a knowledge base. Name recognition is not accuracy. That’s blind trust, and smart professionals across every field are increasingly unwilling to give it without something real underneath.

This article has been reviewed and approved for legal accuracy by Nick Anderson, Esq. It is intended for informational purposes only and does not constitute legal advice.

See more from Nick Thayer at https://stratemy.substack.com/

The information presented in this post is provided as-is for general informational and entertainment purposes only. While every effort was made to ensure accuracy at the time of publication, facts, laws, and best practices can change quickly, and unintentional errors or omissions may occur. Nothing here should be construed as professional, legal, medical, or financial advice. Always verify details independently and consult a qualified professional before acting on any information contained herein. This article was created with the assistance of artificial-intelligence tools and subsequently reviewed and edited by the author. Any views expressed are those of the author alone and do not necessarily reflect the viewpoints of the AI developers or any other entity. By reading, using, or sharing this content, you agree that the author and publisher are not liable for any loss, injury, or damages arising from its use