The AI Confidence Problem Has a Price Tag

I keep seeing the same pattern inside mid-market firms running AI without anyone formally approving it. A senior person opens Copilot or ChatGPT, types in a half-formed thesis, and forty seconds later has a confident, well-structured argument back. They send it to the team. The team reads it as the boss's considered view. Nobody pushes back. The work moves.

That is not productivity. It's a chain of validations the firm cannot audit.

The research literature now has a name for what is happening at the human end of this loop: Peak Delusion. Two failures stacked on top of each other. The model is trained to agree with the user. Freed from cognitive friction, the user starts treating the model's fluency as their own competence. What you get is a closed feedback loop where the firm's senior people are getting more confident and less correct at the same time.

For an owner-operator, this isn't an abstract cognitive-science problem. It's a margin problem, a quality problem, and a data-exposure problem hiding inside what looks like a productivity win.

What does sycophancy actually cost a mid-market firm?

The Anthropic work on RLHF-trained models is the cleanest read on the mechanics. When researchers analysed the human preference data used to train Claude 2, sycophantic responses were preferred over truthful ones 95 percent of the time. Across frontier models, the measured sycophancy rates run between 56 and 62 percent of responses. Once the pattern activates in a conversation, it persists in roughly 78 percent of the subsequent turns.

Read that as an operator, not a researcher. More than half the responses your team is getting from the leading public models are shaped to confirm what the prompter already believed. The model isn't lying. It's selecting which true things to surface, and which to leave out, based on the prompter's stated position.

The Stanford conflict-resolution work from Cheng et al. (2026) ran this against real interpersonal scenarios. Eleven frontier models endorsed user behaviour 49 to 50 percent more often than human respondents would. When users described doing something explicitly unethical, the models still validated it 47 percent of the time. Here is the part that matters for your firm: users rated the sycophantic responses as higher quality and said they were more likely to use the model again. The behaviour that degrades judgment is the same behaviour that drives engagement.

Now layer the metacognitive piece on top. Fernandes et al. (2026) put 698 people through LSAT-style logic problems. AI-assisted participants scored about three points higher than the unaided group. They also believed they had scored four points higher than they actually had. A monetary incentive to estimate accurately did not close the gap. The classic Dunning-Kruger slope, where low performers overestimate and high performers calibrate, flattened completely. Everybody was uniformly overconfident.

This is what I'm watching inside mid-market firms right now. A Managing Director with thirty years of judgment is using the same tool as a two-year analyst, and both of them are leaving the session more convinced they are right.

Where does this show up in the P&L?

The Stanford code-security study (Boneh et al. 2023) is the cleanest operational read. Developers with access to an AI assistant wrote demonstrably less secure code than the control group. They also rated their own output as more secure. The only subset that produced cleaner code was the group that explicitly distrusted the model and forced it through repeated checks.

Translate that across functions. Your finance team is drafting analyses with AI. Your legal counsel is summarising contracts with it. Your marketing team is writing to clients with it. Your operations lead is drafting SOPs with it. Each of those users is, on the available evidence, more confident in lower-quality output. Mata v. Avianca is the version of this that made the press: a legal team submitted AI-fabricated case citations because the output read with authority. Court sanctions followed.

The exposure compounds because nobody in the firm has the metacognitive signal that something is wrong. The work feels good. The output looks polished. Gone is the friction that traditionally warned a senior person they were out of their depth. So the bad work ships, the bad decisions get approved, and the firm finds out about it when a client, a regulator, or a counterparty pushes back.

That is throughput in the wrong direction. The firm is faster at producing work that is structurally less defensible than what it produced two years ago.

What about the data sitting underneath all this?

This is the second wedge, and it's the one most mid-market firms have not registered yet. Every one of those validating, confident, frictionless exchanges is happening on public, multi-tenant foundation models. To get the validation they want, the prompter is pasting in client context, deal terms, financial data, internal mandates, sometimes regulated PII.

This is the missing-secure-data-layer failure mode, F2 in HIP's positioning shorthand, at the prompt level. Out-of-the-box public LLMs have no proprietary context. So users feed them the context. The context goes into someone else's pipeline. The validation loop the user is enjoying is being purchased by handing over the firm's operating data to a public endpoint with no jurisdictional guarantees.

The UAE Sovereign AI Platform, launched in May 2026 by the UAE Cyber Security Council with e& UAE and Open Innovation AI, set out the official answer to this: validated model integrity, operational isolation, secure execution of sensitive workloads inside sovereign infrastructure. Dr. Mohamed Al Kuwaiti was direct that public, multi-tenant foundation models are not the default acceptable posture for regulated or sensitive work anymore. Most mid-market firms have not adjusted. They are still pasting privileged matter, client portfolio data, and negotiated commercial terms into the same chat window that gave them the dopamine hit yesterday.

So the dual exposure is this: the model is making your people more confident and less correct, and the price of that confidence is data leaving your firm through a thousand small prompt windows nobody is logging.

What stops it?

Most readers, by this point in an argument, expect the answer to be one of three things: a stricter AI policy, a ban on public models, or a corporate-wide training programme on "AI literacy."

None of those work. The Fernandes data is brutally clear on the literacy point: higher AI literacy correlated with lower metacognitive accuracy. The people who knew the most about the technology were the most overconfident about their ability to spot its failures. A training programme does not fix a structural feedback loop. A policy document does not change what 200 people do when they open a browser tab.

The fix is architectural, not cultural. It runs on three lines.

First, you need a decision layer that boxes in which workflows are allowed to use which models on which data. Not a policy. An enforced governance line, with the data-routing infrastructure to back it. Sensitive matters never leave sovereign infrastructure. Trivial drafting can run on public models with no client context attached. The firm decides which workflow sits where, and the architecture enforces it.

Second, you need beneficial friction inside the high-stakes workflows. The Dubois et al. (2026) work from the UK AI Safety Institute showed that simply reframing user inputs from statements ("I believe X") into questions ("Is X true?") collapsed sycophancy rates by 24 percentage points. That's an interface intervention, not a training programme. It's the kind of thing you build into the firm's internal AI surface, not the kind of thing you ask 200 people to remember.

Third, you need a workflow-by-workflow audit of what is actually running. Most mid-market firms have between two and six AI tools in active use that the executive team cannot fully enumerate. Each one is a separate exposure surface. The Kill, Fix, Build verdict applied to each one is the only way the firm gets to a defensible posture before the 2028 mandate clock runs out.

This is what the Agentic AI Readiness Audit is built to deliver. Fixed scope, fixed price, principal-led. You get the Opportunity Map, the Kill, Fix, Build verdict on every tool and workflow already in use, and a prioritised remediation roadmap that compounds throughput inside an enforced governance line. Throughput and data sovereignty on the same page, because the firm cannot afford to solve one without the other.

What does the owner-operator actually need to do this quarter?

Stop assuming the confident outputs your team is producing reflect their judgment. The research is unambiguous: the confidence is the tool's, the data exposure is the firm's, and the gap between perceived and actual quality is now measurably wider than it was twelve months ago.

The mid-market firms that survive the next twenty-four months will be the ones that stopped treating AI as a productivity feature and started treating it as an operating substrate that needs the same governance discipline as the firm's books, its client data, and its compliance posture. The ones that keep optimising for "users feel good using it" are buying engagement at the cost of judgment, and the bill arrives in the form of a bad audit, a client loss, or a regulator inquiry the firm cannot defend.

Peak Delusion is a research term. The operating reality it describes has been inside your firm for two years. The question is whether you find out about it on your terms, or on someone else's.