The more we trust a name, the less we question the machine wearing it, and the more that name has to lose when the machine is confidently wrong.
A few weeks ago my team spent an hour arguing about whether to lie to people.
We were designing the onboarding for an AI product, the first time user experience that tries to prove the product is worth their time. The fairly insignificant thing we were fighting about was whether to add a little animation that made it look like the software was thinking hard, adding a few seconds to the flow when really it could have been instantaneous.
A spinning “analyzing your answers…” moment where we could inject some language and brand feel into otherwise straightforward flow. A fake insight or two we’d basically hard-code to make it feel personal and “magical,” a wildly overused word.
We knew what it really was, though. It was smoke and mirrors, but it really divided us, because we knew it would work. It would have been a great first impression. What stopped us was the idea of a bait and switch where we get someone genuinely excited about something that wasn’t actually in the product (yet).
We talked ourselves out of it, not because we’re saints, but because the same worry kept coming up from every one of us. Someone finishes that slick little intro, thinks “huh, that was cool,” and then realizes it didn’t do a single useful thing for them. You’d have spent your most valuable moment, the one where a person decides whether to trust you, teaching them the magic was fake.
Which, obviously, it was because magic isn’t real.
What stuck with me afterward was that this was a low-stakes debate. We were literally deciding whether to fake a loading animation.
What happens when the smoke and mirrors comes with larger stakes? When the confident, capable-sounding thing on the other side of the screen is wearing the name of someone, or some company, you already believe, and you have no reason left to check whether it actually knows what it’s talking about? What does it cost that name when it’s wrong?
The tell we lost
We were never going to say the product was smart, we were going to make it feel smart, and let the user’s own brain fill in the rest. That works because of a little embarrassing truth about people: we trust confidence over almost anything.
We, so often, don’t fact-check each other. We read how sure someone sounds and we take it as a stand-in for whether they know what they’re talking about. Psychologists have even measured this directly in studies on what they call the status-enhancement theory of overconfidence.
People who simply seemed more confident were judged more competent and handed more influence, whether or not they were actually any good, and observers couldn’t tell earned confidence from the empty kind, because from the outside they look identical.
It’s the reason you nod along to the guy at the party swearing a ribeye needs exactly four minutes a side, when anyone who’s cooked one knows it depends on the ribeye.
AI doesn’t do any of that. It hands you the wrong answer with exactly the same fluency as the right one. Researchers at Carnegie Mellon put numbers on this showing that when people fail at something, they get less sure of themselves afterward, which seems like the normal, healthy response.
The AI models they tested got more confident as they got things wrong. One of the researchers put the human cost of it plainly, that we’re not skeptical enough of AI because it asserts everything with a confidence it hasn’t earned, and it never gives off the hesitation cues we’ve spent our lives learning to read.
We spent more time arguing about that animation than it would have taken to build it. What we were actually debating was whether to hijack something we hadn’t actually earned, which was the user’s decision whether or not the product knows what it’s talking about.
Obviously, we were creating a lot of tension over something relatively small. Now picture the same decision made by something you’d actually stake a choice on. A brand you’ve trusted for years. A name whose whole reputation is being right. That decision is being made right now, in rooms exactly like ours, and the stakes are growing larger every day.
The oldest trick in the book
The idea of creating an onboarding experience to feel smarter than it is isn’t new and, frighteningly, it’s quite common. This type of thing even has a name.
Harry Brignull has spent fifteen years cataloging the ways interfaces trick people into doing things they wouldn’t do if they understood what was happening. He called them “dark patterns,” then renamed them, in a 2023 book, deceptive patterns. Deceptive is the right word here.
The fake countdown timer. The pre-checked box.The unsubscribe button hidden like it’s the Ark of the Covenant in that government warehouse. These are historically effective ways to get people to do what you want, and in our case, we wanted to use them to preemptively earn a user’s trust.
Those tricks go after your behavior, they nudge you to click, to stay, to buy. What we were tempted to do went one level deeper. We didn’t want to manipulate what the user did. We wanted to manipulate what they believed, to make them trust the thing before it had done anything to deserve it. That’s a darker pattern than any pre-checked box, because once someone believes you, they stop questioning you.
For a hundred thousand years, confidence was a decent thing to trust, though not for the reason we think. We tell ourselves we can read a liar, the shifty eyes, the nervous pause, the tell. The reality is, we mostly can’t. Decades of research have shown those cues are faint and unreliable, and most people are mediocre lie detectors.
So we’re bad at spotting liars. Historically, that was survivable, because the grifters eventually got found out sooner or later (at least I like to think so). Talk a big game about something you don’t actually know, and eventually you’re exposed, and that will cost you the deal, the job, your credibility.
Being confidently wrong used to have consequences, so trusting confidence made sense. You paid for it eventually.
Today, that reliance on trust has become a bit more dangerous and should start to worry you.
In the movie Rounders, Mike (Matt Damon) beats Teddy KGB (John Malkovich) because he catches his tell, the way Teddy handles his Oreo cookie when his hand is strong. That’s how it’s supposed to work. You read the tell, you know the truth underneath the bluff.
AI is Teddy KGB with no Oreo. It hands you the wrong answer in the same voice as the right one. Sure, sloppy AI still has its tells and people have catalogued them: the ubiquitous em dash, the “here’s what nobody’s telling you” opener, the sentence that’s grammatically perfect and says nothing. Those are the early mistakes of a system that’s learning fast and that’s why they’re getting patched out by the day.
Cleaning up the tells doesn’t make these tools harder to catch when they’re wrong and the worst part is that they get more confident the more wrong they are.
So we’ve built a machine that’s confidently wrong and impossible to read. Now we’re about to add a layer that could make people stop checking for the truth altogether, a name they already trust.
That’s where this stops being a design problem and starts being a very expensive one.
The name does the lying for you
There’s a reason con artists don’t pose as nobodies. In Dirty Rotten Scoundrels, Lawrence Jameson fleeces rich women by posing as royalty in exile. He barely has to lift a finger, because the crown does the persuading for him. He doesn’t have to earn their trust because he’s already leveraging it.
Psychologists have a plain name for this called authority bias, the tendency to believe something more because of who, or what, it came from, regardless of whether the thing itself holds up.
It actually runs way deeper than most of us probably want to admit.
In the 1960s, Stanley Milgram got ordinary people to deliver what they thought were dangerous electric shocks to a stranger, simply because a man in a lab coat told them to keep going. A few years later, a study had nurses take phone orders from a “doctor” they’d never met, to give a patient a dose that was obviously too high.
Almost all of them started to do it. These were trained professionals who knew better, and they went ahead anyway, because the order came from a doctor. We tend to defer to authority we trust, even when our own judgment is screaming otherwise.
Let’s bring back the idea of what a productized expert actually is. Strip a real expert down to just their conclusions and that’s what a productized expert is. All of the authority, none of the person, bottled and sold at scale. It answers instantly, at any hour, for anyone who pays, and it never does the thing that used to make an expert worth trusting, hesitation.
It doesn’t say “let me look into that.” It doesn’t tell you when you’ve wandered past what it actually knows. It just answers, in the voice of someone whose whole career taught you to believe them.
What should terrify the companies racing to do this is that trusted name doesn’t just make you believe the copy. It puts the company’s whole reputation behind an answer no human ever checked. Every confident, wrong response goes out wearing a name that took decades to earn, and it spends that trust like it’s free.
It isn’t.
Paul Slovic spent his career studying how trust actually works, and he found it’s wildly asymmetric. It builds up slowly, through years of being right, and it collapses in a single visible failure, fast, and often for good. One bad answer could cost you a chunk of the thing you spent twenty years building.
Think about what happens when you automate that. A human expert who’s confidently wrong damages one relationship at a time, but a productized one is wrong at scale, thousands of times a day, in a million conversations you’ll never see, each one signed with your name or attributed to your brand.
You’ve taken the one asset that made you worth trusting and pointed it at your customers, and handed the trigger to a machine that doesn’t know when it’s wrong.
You can watch that asymmetry happen in real time. When Google first showed off its AI, Bard, in a 2023 ad, the bot confidently answered a simple question about the James Webb telescope, and got it flatly wrong. This, coming from a company whose entire brand is being the thing that gives you the right answer.
Within a day, Alphabet had lost around $100 billion in market value. Decades of “Google means correct” undercut by a single confident mistake, in the company’s own launch ad.
At least Google only lost money. Air Canada got told, in a courtroom, that it’s liable for the words coming out of its AI mouth.
In 2022, the Air Canada’s website had a chatbot, and when a grieving customer asked about bereavement fares, the bot confidently told him he could claim the discount retroactively. He couldn’t. That was never the airline’s policy; the machine simply made it up, in Air Canada’s voice, on Air Canada’s site.
When the customer sued, the airline tried an extraordinary defense that argued the chatbot was a separate entity, responsible for its own words. The tribunal didn’t buy it. A chatbot on your website is your website. You don’t get to keep the trust your name generates and disown the mistakes it makes. Air Canada put its name behind a confident machine, the machine was confidently wrong, and Air Canada paid for it.
The research on this is getting uncomfortably specific.
A 2025 study on algorithmic versus human advice found that whether people trust a machine’s recommendation over a human’s comes down to how much authority they’ve assigned to each, and that the more competent and credible the source seems, the more people hand over the call without checking it themselves.
That same study lands on the only real defense, informed self-trust, which feels pretty uncomfortable.
You have to be willing to trust your own gut over the confident expert. Which is a hard sell, because everything about authority bias is built to talk you out of exactly that.
This isn’t hypothetical, though. It’s now actual case law. In 2023, a New York lawyer named Steven Schwartz asked ChatGPT to help with a filing, and it handed him a set of court cases to cite, confidently, by name.
They didn’t exist. He filed them in federal court anyway, because they sounded real and he had no reason to think the machine would just make them up.
A lawyer, in the one job where checking the citation is kind of the whole point, took the confident answer on faith.
Sadly, he’s no longer a cautionary outlier. Courts have now flagged roughly 900 of these fake-citation filings, by trained lawyers whose entire profession is not taking someone’s word for it.
If a lawyer will stake his license on a confident answer he didn’t fact check, imagine how fast a company will stake its brand on one. The lawyer only embarrassed himself. When the name on the answer is yours or your company’s, every confident mistake burns trust you can’t buy back and can maybe only slowly rebuild if you’re lucky.
The only thing that works is annoying
The obvious and glamorous solution is to keep a human in the loop, someone who checks the machine before anyone trusts it, but it feels like busy work, which is why almost no one does it right.
That would be the ideal answer, except we’re wired to stop looking. Researchers call it automation bias, and the foundational study on it found the tendency to over-trust a machine and quit scrutinizing it shows up in experts as much as novices, and it can’t be trained away.
The MIT Sloan Management Review describes what that looks like in practice, review without real scrutiny becomes “a dangerous illusion of control,” a person only notionally in charge, approving things they never actually examined. Which, arguably, is worse than no human at all, because now the mistake moves up the ladder with a human’s blessing.
Europe took it seriously enough to create its own law around this. The EU’s AI Act demands meaningful oversight of high-risk systems, and regulators test whether it’s real by checking how often the human actually overrides the machine. If nobody ever says no, there’s no oversight happening. There’s just a human being blamed for the machine’s mistakes.
The solution should never have been “just have a smart person glance at it.”
The solution is a reviewer with something to lose, their name, their license, their reputation, so that when the machine sounds sure and it’s wrong, they’re the one holding the bag.
One company I work with, whose entire business is being the trusted voice in its industry, understood this to its core. When they built AI into their product, they created one unbreakable rule that the machine can gather, draft, and summarize, but it is never allowed to be the thing ultimately gives conclusions.
In their case, a real person, with a reputation to lose, signs off on every answer before it goes out under the company’s name. Their phrase for it was, bluntly, that people conclude, machines don’t. Yes, it’s slower and more expensive, but it’s the only kind of oversight that actually protects their credibility, which is the thing they’re really selling.
A person with no stake will click approve. A person who fears the consequences of any mistake reads every line carefully.
This is why almost nobody will do it. A human who reads every line is slow, and slow loses to a competitor who just lets the machine answer. So the companies with the most trust to spend will be the most tempted to spend it, quietly, a little at a time.
The real question has nothing to do with whether the machine is “good.” It has a lot of good qualities that many of us are taking advantage of right now.
The real question is whether anyone left has the power to overrule it, and a real reason to bother.
At most companies, that person is already gone. Many organizations found that person to be too slow, too expensive, and too willing to kill a good-sounding answer. That person was traded for speed, and speed is what we got. Now the confident machine speaks in the company’s voice, and no one left in the room has the standing, or the reason, to tell it it’s wrong.
I don’t think anyone’s being evil here. That’s what worries me. The company saves money, the customer trusts the brand, the machine doesn’t know it’s wrong, and everybody moves on. We’re about to find out how much quiet damage a confident, trusted, wrong answer can do before anyone bothers to check.
My guess is a lot.
References and further reading
On why we trust confidence
- Anderson, Brion, Moore & Kennedy, “A Status-Enhancement Account of Overconfidence” (Journal of Personality and Social Psychology): confident people are judged more competent and handed more influence, and observers can’t tell earned confidence from the empty kind.
- Cash & Oppenheimer et al., via Carnegie Mellon: humans lose confidence when they fail; the AI models tested grew more confident as they got things wrong, and give off none of the hesitation cues we use to read people.
- Vrij, Hartwig & Granhag, “Reading Lies” (Annual Review of Psychology): the nonverbal “tells” we swear by are faint and unreliable, and people are mediocre lie detectors.
On deceptive design
- Harry Brignull, Deceptive Patterns: fifteen years cataloging the interface tricks that get people to act against their own interest, renamed from “dark patterns” because deception is the honest word. (book)
- OpenAI on the em-dash “tell” (TechCrunch): the giveaways of AI writing are actively being patched out.
- Hunting the Muse, “How to Tell If Writing Is AI”: a field guide to the vanishing tells, including the “here’s what nobody’s telling you” opener.
On borrowed authority
- Authority bias: the tendency to believe something more because of who it came from, regardless of whether it holds up.
- Stanley Milgram, the obedience experiments: ordinary people delivered what they believed were dangerous shocks because a man in a lab coat told them to continue.
- Hofling’s hospital study: nurses followed an unknown “doctor’s” unsafe phone orders against their own training.
- Acta Psychologica (2025), on algorithmic versus human advice: trust tracks the authority we assign a source, and “informed self-trust” is the one real defense.
On what it costs the name
- Paul Slovic, the asymmetry principle: trust builds slowly through years of being right and collapses in a single visible failure, often for good.
- Google’s Bard demo error (CNN): one confident wrong answer in a launch ad wiped roughly $100 billion off Alphabet’s value in a day.
- Air Canada’s chatbot case (Forbes): a tribunal held the airline liable for its chatbot’s confident, invented policy, rejecting the argument that the bot was a separate entity.
- Steven Schwartz’s fake citations (Forbes) and the roughly 900 filings since (Cronkite News): trained professionals staking their credibility on confident answers they didn’t check.
On why oversight fails, and what actually works
- Parasuraman & Manzey, “Complacency and Bias in Human Use of Automation” (Human Factors): the tendency to over-trust a machine shows up in experts as much as novices and can’t be trained away.
- MIT Sloan Management Review, on rubber-stamping: review without real scrutiny creates “a dangerous illusion of control.”
- The EU AI Act, Article 14: meaningful human oversight, tested by whether the human ever actually overrides the machine.
We’re gorging on borrowed trust and it’s going to cost us. was originally published in UX Collective on Medium, where people are continuing the conversation by highlighting and responding to this story.
