In partnership with
A confident answer to the wrong question
I watched a senior lawyer use an AI assistant to research a contract clause last year. She knew the area well-fifteen years in commercial law. She asked the tool a precise question, got a precise-sounding answer, and moved on.
The answer was wrong. Not obviously wrong. Wrong in the specific way that only becomes visible when you already know the territory well enough to have asked the right question in the first place.
She caught it. But she caught it after she had already structured her advice around it-and only because a junior colleague happened to flag a discrepancy in a footnote.
The paradox is this: expertise does not protect you from AI errors. In certain conditions, it makes you more exposed to them.
Scale AI support on AWS, see how July 9
Customer expectations keep rising. Support budgets don't. On July 9, Fin and AWS are hosting a live executive session on how leading enterprises close that gap: scaling AI-powered support while simplifying how they buy it.
You'll see how to resolve an average 76% of conversations with Fin on AWS enterprise-grade infrastructure, procure through AWS Marketplace to put committed cloud spend to work, and turn the Fin and AWS collaboration into lower support costs. Register for the live session to see how.
Why expertise creates a blind spot
Here is what happens. When a novice uses an AI tool, they tend to treat outputs with general scepticism-they know they do not know, so they verify more.
When an expert uses the same tool, something different occurs. Their deep familiarity with the domain makes a fluent, confident-sounding answer feel right, even when it is not.
The signal they are using to detect errors-whether the output matches what they already know-is exactly the signal that fails when the error is at the boundary of their knowledge.
Psychologists call this fluency miscalibration: the smoother and more coherent a piece of information sounds, the more credible it feels, independent of whether it is accurate.
AI-generated text is almost always fluent. That fluency is a feature for communication. It is a trap for verification.
The confidence-competence gap
Think of it as a gap between two curves. As expertise rises, competence rises with it-but the felt confidence in AI outputs can rise faster, because the expert can evaluate surface coherence more readily. The gap between those two curves is where costly errors live.
What happened to Daniel
Daniel, a financial analyst with eight years in equity research, began using an AI assistant to draft preliminary sector summaries. His review process was efficient: he would scan the output for obvious errors and move on.
Six weeks in, a summary contained a subtly inverted relationship between two metrics-the kind of mistake that only makes sense to someone deep in the sector. He approved it. It went into a client brief. The error surfaced in a follow-up call.
Daniel was not careless. He was calibrated for a different kind of mistake.
Your support queue gets a head start every morning.
Viktor reads overnight tickets, tags them by product area, summarizes the patterns, and posts a brief in #support. The agent picks up the queue already triaged. The PM sees recurring requests rolled up by Friday.
What the research shows
A 2023 study by researchers at the University of Waterloo, published in Computers in Human Behavior, found that participants with higher domain knowledge were less likely to detect AI-generated errors in that domain than participants with moderate knowledge-a 23-percentage-point gap in error detection rates.
The researchers attributed this to 'expertise-induced fluency acceptance': experts process domain-relevant language faster and with less scrutiny, making them more susceptible to coherent-sounding errors.
The study also found that prompting experts to slow down their review-specifically by asking them to articulate why each claim was correct-closed the detection gap by roughly 60%.
The fix, in other words, is not more expertise. It is a different reviewing posture.
The one change that closes the gap
You do not need to distrust AI outputs more. You need to verify them differently.
Novice verification asks: does this seem right? Expert verification needs to ask: what would this look like if it were wrong? Those are not the same question. The first check confirms coherence.
The second one looks for the specific failure mode that your expertise would normally catch-except that your expertise is also what makes it easy to miss.
This week: the next time you review an AI output in your own domain, write one sentence before approving it: the most plausible way this could be wrong.
Not a general caveat- specific one. That single step is what the Waterloo study found closed most of the gap.
I have written more about where AI verification breaks down-and the four conditions that predict it-at promptnproductive.com.
Reply and tell me the domain where you trust AI most. That is probably the one to watch.
—Prompt N Productive—