Why does AI state wrong things as if they were true?

Kohaku
KohakuSometimes an AI answers with great confidence about things that aren't true. Why might that be?
The answer in 30 seconds
KohakuIt comes down to how it predicts the next word, and how tests are scored.

An AI chatbot builds text by choosing words that are likely to come next, so even when it has few clues to the answer, it can still produce something that sounds right. On top of that, OpenAI explains that tests that only count correct answers give zero points for "I don't know," so guessing looks better on the scoreboard, and that way of answering tends to stick.

Kohaku
KohakuNow, let’s look at it in more detail.
A road on a hot day where the far end shines like a puddle: a mirage
A "mirage" on a road on a hot day. Water that isn't really there looks as if it is.Photo: Michael from Calgary, AB, Canada / CC BY 2.0 / Wikimedia Commons

Plausible mistakes are called "hallucinations"

The AI company OpenAI uses the word "hallucination" for text that an AI produces confidently and that sounds plausible but is not true. For example, when a widely used chatbot was asked for the title of one researcher's PhD dissertation, it gave 3 different titles, and none were correct. When asked for his birthday, it gave 3 different dates, all wrong.

Rare facts are the easiest to get wrong

The base model behind an AI first practices predicting the next word using huge amounts of text (pretraining). That text carries no "true" or "false" labels. Things that follow a fixed pattern, like spelling or closing brackets, stop going wrong as training goes on. But facts that rarely appear and can't be predicted from a pattern, like a pet's birthday, are much easier to get wrong.

Figure: Patterns versus rare facts
A map with clear roads, a map where word cards invent a road past a dead end, and two test sheets compared
  1. 1A question with plenty of clues (a map with clear roads)
  2. 2A question with few clues (a map whose road stops halfway)
  3. 3A road that shouldn't exist, made by lining up plausible words
  4. 4A test where blanks score nothing, but guesses sometimes score

On tests, guessing looks better

OpenAI compares this to a multiple-choice test. If you don't know the answer, a wild guess might still be right, but a blank always scores zero. Asked for a birthday, a guess of "September 10" has a 1 in 365 chance of being right. Because many evaluations only compare the share of correct answers, the guessing model ends up looking better. On a quiz called SimpleQA, gpt-5-thinking-mini held back on 52% of the questions, got 22% right and was wrong 26% of the time. o4-mini held back only 1% of the time; it got slightly more right, at 24%, but it was wrong 75% of the time.

Figure: Holding back versus guessing
gpt-5-thinking-miniHeld back52%Right answer22%Wrong answer26%o4-miniHeld back1%Right answer24%Wrong answer75%

From OpenAI's explanation of the SimpleQA quiz.

Kohaku
KohakuA blank scores zero, but a guess might be right. Even AI has test troubles, it seems.

Fixing the scoring, and checking as users

So OpenAI suggests changing the scoring: take off more points for confident mistakes than for "I don't know," and give partial credit for answers that say they are unsure. A guide from the Massachusetts Institute of Technology (MIT) also explains that generative AI works like an advanced autocomplete, built to produce plausible text, not to check whether it is true. For anything important, always check by asking an expert or comparing with sources such as research papers.

Want to know more?

  • Anthropic's research found that inside its AI Claude, a pathway that replies "I don't have enough information to answer" is switched on by default, and it gets held back when the question is about a well-known name.
  • OpenAI explains that a small model that has learned no Maori at all can simply say "I don't know" to a Maori question, while a model that knows a little has to judge how sure it is.
  • According to the MIT guide, setting the "temperature" (which controls how varied answers are) low, at 0 to 0.3, tends to give more focused, fact-based answers.
Kohaku
KohakuThe more convincing a sentence sounds, the more it is worth pausing to check. That is the secret to getting along well with AI.

Quick quiz: test yourself

KohakuRemember what you just read? Give it a try!

Why do AI models learn to guess?

Show the answer
  1. They were built to trick people
  2. Tests only count right answers
  3. Everything online is wrong

Tests only count right answers When only correct answers count, "I don't know" scores zero. Guessing looks better on the scoreboard, so that way of answering tends to stick.

Sources
  1. Why language models hallucinateOpenAI
  2. When AI Gets It Wrong: Addressing AI Hallucinations and BiasMIT Sloan Teaching & Learning Technologies
  3. Tracing the thoughts of a large language modelAnthropic

This article was written after checking the sources above as of 2026-10-03. Explanations may change as research progresses. Photo credits are shown below each photo.

Next fun fact ▶

TechnologyHow do video apps choose what to recommend?When you open a video app, it is full of videos you want to watch. How are they chosen?
NANI?! Animal Encyclopedia
Our book is on AmazonNANI?! Animal Encyclopedia

40 surprising animal facts with big photos and short, easy text. A fun-facts book for kids and grown-ups to read together.

See it on Amazon