A few years back, a law firm used a consumer chatbot to help prep a case. The tool handed back a beautifully written brief, citing precedent after precedent. The lawyers filed it. The problem? The cases didn’t exist. The AI invented them. The firm got sanctioned and very nearly went under.

Everybody loves to tell that story as a cautionary tale about AI being unreliable. I tell it for a different reason. That chatbot didn’t malfunction. It did precisely what it was designed to do, and once you understand why, you’ll stop being surprised by it and start building around it.

Let me walk you through it. No math. No PhD required. Two minutes of intuition.

A language model is a guessing engine

This is the whole trick behind ChatGPT, Claude, Gemini, Copilot, all of them. You give it some words, and it predicts the next one. Then the next. One word at a time.

“The fox jumps over the…” and it fills in “lazy dog.” Why? Not because it knows anything about foxes or dogs. It has no database it looks things up in. It has no concept of truth. It absorbed an enormous amount of how people write, and it’s matching the most plausible continuation. That’s it.

Your own brain does the same thing, by the way. Show you a short, stocky dog with droopy ears and a stubby nose and you say “bulldog” instantly, not because you ran a database query, but because you recognized a pattern. The model recognizes patterns too. That’s the source of all its magic, and the source of all its trouble.

Because this is what “predicts the next word” really means. The model is always guessing. Even when it’s right, it’s guessing. Ask it the same question five times and you’ll often get five different answers. That’s not a glitch. That’s the machine.

It’s also built to make you happy

Now layer a second truth on top of the first.

These models are trained to be helpful. To please you. And there’s a cold business reason for it. If it makes you happy, you keep using it. So it learned to be accommodating. Tell it “no, that’s not what I wanted,” and it will fall over itself to give you something you’ll like better.

Put those two things together, a guessing engine that desperately wants to satisfy you, and you get the law firm. The lawyer asked for supporting precedent. The model didn’t have real precedent at its fingertips, so it generated the most convincing precedent it could invent. Confident. Cleanly formatted. Completely fabricated. It wasn’t lying to be malicious. It was filling a gap to make someone happy. That’s the whole anatomy of a hallucination.

So can we please retire the word “hallucination”? It’s a misnomer. It makes people picture the AI going off the rails, having some kind of digital fever dream. It isn’t. It’s a confident, plausible, wrong answer produced by a system doing exactly what it was built to do. Knowing that is what keeps you out of the headlines.

The fix isn’t a better model. It’s grounding.

This is the part that actually matters for your business.

You don’t fix this by waiting for a smarter model. You fix it by changing what the model is allowed to answer from.

Out of the box, a model answers from the patterns it learned in training, that vast sea of public text. Grounding forces it to answer from your material instead. Your files. Your contracts. Your records. And this is the key. It forces the model to cite the source for every claim, so you can trace it back to a real document.

The industry term for this is RAG, and it sounds fancier than it is. In plain English, you ask a question, the system first goes and searches your data for the relevant material, staples that material onto your question, and then tells the model to answer using only that. The model stops inventing from memory and starts working from the evidence you handed it.

That’s the entire reason enterprise Copilot behaves differently from the free chatbot your team is using on their phones right now. Same underlying model, completely different guardrails. Copilot grounds in your tenant, answers with citations, and stays inside your security boundary. The consumer tool grounds in nothing but the open web and its own training, which is exactly where invented case law comes from.

How to solve for this

Grounding is the single biggest improvement available in enterprise AI. It typically drops the hallucination rate by something like five to twenty times.

But I’m not going to sell you a fairy tale, because nobody in this space should. Grounding reduces hallucinations. It does not eliminate them. Run the math. A model that’s 99 percent accurate still produces 100 wrong answers for every 10,000 questions. At enterprise scale, 99 percent is not “done.” It’s “keep a human in the loop.”

So here’s what you actually do with all this. Three moves, today, no new budget required.

Ground it. Stop letting people paste sensitive work into consumer chatbots and pray. Point your AI at your real, governed data and make it cite sources. An answer you can’t trace is an answer you can’t trust.

Keep a checkpoint on anything that matters. AI is wonderful at the repetitive, heavy reading, low stakes work. The first draft, the summary, the quick catch up. The moment it’s about to publish, submit, commit, or cite something to a judge, a human reads it first. Not most of the time. Every time.

Write a sharper prompt. A vague prompt forces the model to guess harder, and guessing is exactly when it invents things to please you. Tell it the role, the context, the task, the format, and the guardrails, including “if you don’t know, say so.” Treat your prompt like a spec, not a wish.

None of this requires you to become an AI expert. It requires you to stop thinking of the model as a knowledge oracle and start thinking of it as a brilliant, eager, slightly unreliable new associate. You wouldn’t let that person file a brief unread. Don’t let the AI either.

Steve discusses how he uses Claude projects to create the guardrails David calls out in Building an AI research partner, personalized for you .

The technology is genuinely ready to take real work off your plate. It just isn’t ready to be trusted blindly, and the companies that internalize that difference are the ones that will pull ahead while everyone else is still cleaning up after their chatbot.

So before you trust the next confident answer, ask the only question that really counts. Can you trace it back to something real? Ground your AI before it grounds your reputation.

David Tyler is Director of Data & AI at Trofeo Solutions, where he helps organizations adopt AI in a way that’s governed, practical, and actually trustworthy at scale. Follow David on LinkedIn for more insight into the state of AI.

This article was originally published on LinkedIn and has been republished here with permission, Your AI Isn’t Broken When It Lies to You. It’s Doing Exactly What It Was Built to Do. | LinkedIn.