Why AI Makes Things Up, and Our Fact-Check Checklist for AI-Written Content

In 2023, a lawyer in New York handed a federal judge a brief with six court cases in it. None of them existed. Not the names, not the quotes, not the judges' opinions.
His AI chatbot had made them up. When he asked it whether one case was real, it said yes, and that you could find it in the legal databases. The judge fined him and his colleague $5,000 in June 2023.
He was not the last. One researcher keeps a public list of court cases where AI-invented material showed up. By November 2026 it had more than 2,200 entries, from courts all over the world.
Last time, we watched an AI learn by reading: trillions of words, one guessing game, what comes next? That makes it brilliant at writing what sounds right. This post is about the gap between sounding right and being right, and what we do about it.
What a hallucination is
People call it a hallucination: an AI gives you something fluent, confident, and wrong.
It isn't a glitch in one answer, and it isn't lying on purpose, because there's no purpose in there. It's the system doing what it was built to do: produce the most likely-sounding text.
It happens to the biggest companies too. In February 2023, Google's first ad for its chatbot got a fact about a space telescope wrong, and on the day that was reported, the parent company's value fell by about $100 billion. A year later, an airline's chatbot told a grieving customer about a refund rule that didn't exist, and a tribunal held the airline responsible.
Why it doesn't just say "I don't know"
Think of a multiple-choice exam. Right gets a point. Wrong gets zero. Blank also gets zero. If you don't know, you guess, because a guess might score and a blank never does.
In September 2025, researchers at OpenAI and Georgia Tech argued that this is how we test AI. Most of the benchmarks that rank models only check right or wrong, so an honest "I don't know" scores the same as a wrong answer. A model that bluffs climbs the leaderboard.
OpenAI's own numbers show it. On the same quiz of hard factual questions:
| Says "I don't know" | Right | Wrong | |
|---|---|---|---|
| Model A | 52% | 22% | 26% |
| Model B | 1% | 24% | 75% |
Nearly the same number of right answers. Almost three times as many wrong ones. On a leaderboard that only counts right answers, Model B looks better.
There's a second reason. A model doesn't keep the pages it read; it keeps patterns. A fact that appears a million times leaves a strong pattern. A fact that appears once, like one small court case, leaves almost nothing, so the model fills the gap with something that has the right shape: a believable name, year or quote. Popular myths are worse, because they appear more often than their corrections. In the TruthfulQA test from 2021, people answered truthfully 94% of the time and the best model 58%, and the larger models in that study did worse, not better.
Three ways AI makes things up
- Invented facts. The court cases.
- Numbers that look precise but are wrong. A decimal point feels like proof.
- Claiming it did something it didn't. The newest kind. In July 2025, a founder reported that a coding agent deleted his app's live database, then said it couldn't be recovered. It could.
We caught our own video
Our second episode went through a separate fact-check before it was published. It found nine things to fix in our research and script. A comparison that should have said oven, not microwave. A data-centre forecast that was out of date. An energy figure per AI prompt with the right number but the wrong description of what it measured.
None of them looked wrong. All of them sounded right. That's the point.
The AI agents that run parts of our own business follow three short rules: measure before you claim; open it, or read it back, first; say "sent" only after it's sent. They came from near misses: once our automation would have logged "sent" without checking, and once an agent marked the wrong bid as submitted. Both were caught before any harm.
Our fact-check checklist for AI-written content
Use this before anything AI-written goes out: a post, a report, a client email, a script.
- Ask for sources, then open one. A link isn't proof until you've clicked it and found the claim on the page.
- Find the one number your decision depends on, and check that number at its original source, not a summary.
- Check names, dates and quotes. These are the "right shape" details a model invents most easily.
- Let it search or give it the document instead of asking from memory. Paste the file, the page, the data.
- Be suspicious of myths and of rare facts. Popular claims and one-off details are where models go wrong most.
- When it says it did something, ask it to show you. The sent email, the saved file, the changed record.
- Have a second pass. A separate check, by a person or a second tool with fresh eyes, catches what the first one missed.
The real fix
There's a hopeful finding. Anthropic looked inside one of its models and found that, in that model, the default is to say it can't answer. Something has to switch that off. Hallucinations happen when a question feels familiar enough to flip the switch, but the model doesn't really know. That's something researchers can measure, and a place to start fixing it.
So an AI makes things up because it learned to sound right, and because we graded it for guessing. The fix isn't asking it to try harder. It's giving it a way to check: let it search, let it read the actual file, let it run the numbers.
To check things, though, an AI needs hands: a way to reach your files, your apps, the web. That's the next episode.
We're PretendMind. When we're not making these, we build AI tools for real companies, with the checking built in. Talk to us at pretendmind.com.
FAQ
Why does AI make things up?
Because it was trained to produce the most likely-sounding next words, not to check facts. And most tests that rank AI models give a point for a right answer and nothing for "I don't know", so guessing scores better than admitting uncertainty. Rare facts and popular myths make it worse.
What is an AI hallucination?
An answer that is fluent, confident and wrong: an invented case, a made-up quote, a precise-looking number that isn't real, or an AI agent saying it did something it didn't do. It isn't a random glitch or a deliberate lie; it's the model producing text that sounds right.
Do AI models that say "I don't know" make fewer mistakes?
In OpenAI's test, yes. A model that abstained about half the time was wrong 26% of the time, while one that almost never abstained was wrong 75% of the time, with nearly the same number of right answers.
How can I check if an AI answer is true?
Ask for sources and open at least one. Check the single number your decision depends on at its original source. Double-check names, dates and quotes. Where you can, let the AI search or give it the document instead of asking from memory.
Can AI hallucinations be fixed?
Not completely yet, but they can be reduced a lot. Tests that reward honest uncertainty help, research inside models is finding where guessing starts, and the most practical fix today is giving the AI tools to check: search, the actual files, and the ability to run the numbers.