How does an AI chatbot come up with its answers?

Kohaku
KohakuAsk a question and an AI answer flows right back. How is that text actually made?
The answer in 30 seconds
KohakuIn truth, it picks words one at a time and strings them together.

An AI chatbot takes your question and builds an answer by repeating one step: pick a likely "next word" and add it on. To do this, it is first trained on huge amounts of text to predict the next word, and then tuned so its answers move closer to the ones people rated as better.

Kohaku
KohakuNow, let’s look at it in more detail.
Rows of server racks lined up in a data center
Racks of servers (computers that do lots of calculations) lined up in a data center.Photo: PiDatacenters / CC BY-SA 4.0 / Wikimedia Commons

Splitting text into small "tokens"

The text you type into a chatbot is first split into small pieces called "tokens." A token can be a whole word, part of a word, or a single character. Inside the AI, your text is handled as a string of these tokens.

Practice predicting the next word, then fine-tuning

According to Anthropic, the company that makes the AI Claude, the base model is trained on huge amounts of text to predict the next word from the words that came before. This is called "pretraining." At this stage, though, the model is not good at answering questions or following instructions. So it is refined with further training and with a method called RLHF, in which people compare two or more answers and rank them, and the model is encouraged to give answers like the higher-ranked ones. This turns it into a helpful assistant.

Figure: How an AI chatbot is made
Pretrainingpredict the next wordTuning (RLHF etc.)people rank answersand the model moves closerA helpful assistant

Choosing one piece at a time

When it writes an answer, it looks at your question and the part of the answer it has written so far, picks one next token, and adds it on. Repeating this many times produces the full answer. The reason the same question can get a slightly different answer each time is that there is a setting that adds some randomness to how the next word is chosen.

Figure: How an answer is built
An AI that learned from huge amounts of text picks the next word piece from candidates and adds it, again and again
  1. 1The huge amount of text used for learning
  2. 2The row of word pieces that make up the text so far
  3. 3Candidates for the next word piece
  4. 4The word piece that was chosen
  5. 5Picking and adding a piece, again and again
Kohaku
KohakuEven its makers still don't understand much of it. Isn't that a curious kind of tool?

What happens inside is still being studied

These models are not built by people writing rules one by one. Instead, their ways of solving problems take shape during training. That is why even the companies that make them still don't understand much of what happens inside, and research continues. A 2025 study found that when writing a poem, the model lines up possible rhyming words for the end of a line before writing the line. It also found that when a "this seems familiar" signal switches on even though the needed information is missing, the model can produce text that sounds right but isn't true. For important things, check other sources too.

Want to know more?

  • In earlier Claude models, one token is said to be about 3.5 English characters long.
  • There is a limit to how much text the model can look back on while writing an answer, called the "context window." A bigger one makes long texts easier to handle.
  • A study on addition found two paths working at the same time: one that works out a rough answer, and one that works out the last digit exactly.
Kohaku
KohakuChoosing the next word one at a time to build an answer: that is how an AI chatbot works. For important things, please check other sources as well.

Quick quiz: test yourself

KohakuRemember what you just read? Give it a try!

How does an AI chatbot build its answer?

Show the answer
  1. It picks the next word, one at a time
  2. It copies an answer from the internet
  3. It looks it up in a fixed answer table

It picks the next word, one at a time It looks at the question and what it has written so far, picks one next token, adds it on, and repeats this to build the answer.

Sources
  1. GlossaryAnthropic(Claude の開発者向け資料)
  2. Tracing the thoughts of a large language modelAnthropic

This article was written after checking the sources above as of 2026-10-03. Explanations may change as research progresses. Photo credits are shown below each photo.

Next fun fact ▶

TechnologyHow does a phone screen feel your finger?A phone screen reacts to just a gentle touch of your finger. What exactly is it sensing?
NANI?! Animal Encyclopedia
Our book is on AmazonNANI?! Animal Encyclopedia

40 surprising animal facts with big photos and short, easy text. A fun-facts book for kids and grown-ups to read together.

See it on Amazon