/ an illustrated guide to language models

How does AI think?

When you type “bank”, an AI doesn’t know whether you mean money or a river. Watch it figure that out, layer by layer, measured from a real model.

Watch AI figure it out All playgrounds

This is a simplified illustration of model computation. It does not show a mind, feelings, or a literal stream of thoughts.

/ the mystery

How does AI know what you mean?

“I waited by the bank.”

River bank? Money bank?

On its own, the model can’t be sure.

Add a few words:

The river was rising, so I waited by the bank.

Which meaning will the model land on?

/ 01 break it apart

What a model reads: tokens

Back to the beginning. First, our sentence is chopped into pieces called tokens.

TextThe river was rising, so I waited by the bank

How many pieces do you think this becomes?

Runs in your browser with GPT-2’s tokenizer. Nothing is sent.

Token numbers and tokenizer

Each token is really a number, an entry in a vocabulary of 50,257. The model never sees letters.

4647850373739611523314134884162623331

These tokens come from GPT-2’s own tokenizer, the same model measured on the rest of this page. “Try your own” uses that tokenizer (r50k_base) through the open-source gpt-tokenizer package. Other models split text differently.

A token can be a whole word, part of a word, a space and a word together, or a punctuation mark. Common words are usually one token. Rare words break into several. Each piece is really a number: its place in a vocabulary of 50,257.

TakeawayAI doesn’t read words. It reads chunks of text, and each chunk is a number.

What’s simplified here?
  • Tokens are the model’s input units. They are not little thoughts.
  • These pieces come from GPT-2’s tokenizer. Other models split text differently.
/ 02 turn each piece into numbers

Each piece becomes a list of numbers

One tiny word. 768 numbers. Here is “bank” in two different sentences.

Measured from GPT-2 small · the word “bank”

“The river was rising, so I waited by the bank”
“I needed to deposit a check, so I waited by the bank”
Hidden until you guess

River bank and money bank. Are their numbers the same, or different?

You can’t meaningfully read those numbers one by one. Together, though, they act like coordinates describing how the model currently represents that piece. They are often called an embedding. They were learned in training, so pieces used in similar ways end up with similar numbers. And they are looked up the same way every time, whatever the sentence around them.

TakeawayBefore context, both meanings of “bank” are literally identical to the model.

What’s simplified here?
  • The numbers are not definitions. They are coordinates the model learned to find useful.
  • GPT-2 also adds a small signal for each word’s position before the first layer. It is not drawn here.
/ 03 figure out what it means

How AI re-reads a sentence, layer by layer

This is what you watched at the top of the page, slowed down. Press play, or step through the layers.

The river was rising, so I waited by the bank

Start: before any re-reading

Start: “bank” is the same in every sentence.

Here’s what “bank” paid attention toAt the start: how much it draws from each earlier word

Nothing has been re-read yet. Each token only knows itself.

Here’s where its meaning movedAt the start, the same in every sentence

This is the part most explanations skip. The numbers pass through a stack of layers: 12 in GPT-2 small, many more in today’s large models. In every layer, each token looks back at the tokens before it, a step called attention, and pulls in what it needs. Then its numbers are rewritten. Meaning is not looked up once. It is rebuilt, layer after layer.

A hint about a rising river pulls “bank” toward the river meaning from the very first layer. A check to deposit pulls it toward money. With no hint at all, GPT-2 drifts toward money, the more common use of “bank” in the text it learned from.

TakeawayThis is where context changes what a word represents.

One more experiment

What if “river” comes after “bank”?

After a long and busy morning, I waited by the bank. The river was rising.

Will “river” change what “bank” means?

What’s simplified here?
  • The meaning map flattens 768 numbers into 2.
  • Attention is averaged over the 12 heads in each layer. Individual heads do different things.
  • GPT-2 parks much of its attention on the first word, a known quirk, so that bar is capped.
  • The hint gets the most attention only in the first layers, yet the meaning keeps moving. Averaged attention is not the whole story of how information moves.
/ 04 pick an answer

Watch an answer take shape

You’ve watched the model work out what “bank” means. But understanding the sentence isn’t enough. It still has to decide what comes next, and you can read that decision forming, layer by layer.

The river was rising, so I waited by the bank of the ___

The final answer is “river”. When do you think it takes the lead?

Measured from GPT-2 small. Layers 1 to 11 are read out with the “logit lens”, an approximation: each layer’s numbers are run through the model’s final step early. Layer 12 is the model’s real output. Orange marks the final answer wherever it appears.

In the first layers, the guesses are generic words. Around the middle, the right idea appears. By the last layers, one answer has pulled ahead. The answer is not retrieved from a list. It is built up as the text is re-read.

TakeawayThe answer doesn’t suddenly appear. You can watch it become more likely.

Surely AI knows 2 + 2, right?

Complete it: “Two plus two equals ____”

What’s simplified here?
  • Layers 1 to 11 are read out with the “logit lens”, an approximation: each layer’s numbers are run through the model’s final step early. Layer 12 is the real output.
/ 05 do it again

One word at a time, then all over again

The model picks one piece, adds it to the text, and starts the whole trip again.

Measured from GPT-2 small · real probabilities along one path

The river was rising, so I waited by the bank of the

To add 5 more pieces, how many layer passes does the model run?

0 pieces added× 12 layers =0layer passes

Scores for the next piece

  1. ·river48%
  2. ·stream4%
  3. ·River3%
  4. ·lake2%
  5. ·creek1%
  6. All other pieces42%

After the last layer, the model scores every token in its vocabulary of 50,257. One is selected and added to the text. Then the whole trip runs again, through every layer, to pick the one after that. A single paragraph of reply takes about a hundred of these trips.

That is the whole journey for our sentence: text, then 11 pieces, then 768 numbers for each, then 12 rounds of re-reading, then “river”, and then all of it again for the next word.

TakeawayEvery reply is built this way: one piece at a time, re-reading everything each time.

What’s simplified here?
  • This path always takes the top score. Chat products usually draw at random, weighted by the scores, so lower-ranked pieces sometimes win.
  • Scores are not feelings, intentions, or confidence that a statement is true.
How this was measured
  • Model. GPT-2 small (openai-community/gpt2, revision 607a30d): 124 million parameters, 12 layers, 768 numbers per token, 12 attention heads. Measured once on a laptop with the open-source transformers library. This page never runs a model; it shows the saved results.
  • Start and layers. “Start” is the word’s lookup numbers. GPT-2 then adds a small signal for word position, which is not drawn. Layers 1 to 12 are the running numbers after each layer, before the model’s final normalization step.
  • Lean. For each word, 12 reference sentences per meaning. At each layer, all of the word’s numbers are centered, and the lean is its position along the line between the two meaning averages, scaled so each average sits at +1 or −1. The reference sentences are never the ones shown. Word position explains little of the lean: its correlation stays under 0.31 for “bat” and “mouse”, and under 0.51 for “bank”.
  • Meaning map. The two directions along which the word’s numbers vary most at that layer, turned to line up with the previous layer so the picture does not flip.
  • Attention. The word’s attention to each earlier token, averaged over the 12 heads of that layer.
  • Answer read-outs. Each layer’s numbers passed through the model’s final normalization and output step early (the logit lens). Layer 12 is the real output.
  • Next piece. The model’s real output probabilities at each step, following the top choice.
  • Examples. These words were chosen because their meanings separate clearly. Other words may not behave as neatly.
  • Sources.Vaswani et al., 2017, the transformer; Radford et al., 2019, GPT-2; nostalgebraist, 2020, the logit lens; Ethayarajh, 2019, contextual word representations.
/ the big question

So, is this thinking?

What you watched is a model computation, illustrated for people. Numbers are looked up, re-read and rewritten through 12 layers, then a next piece is scored and selected, over and over. There is no mind or feeling in the picture.

Whether any of it deserves the word “thinking” is a question about words as much as machines. What the measurements show is narrower and more useful: in a language model, meaning is built up by re-reading, and every answer is chosen one piece at a time.

Similar-looking answers can come from very different processes. A model produces an output; the application around it decides what happens next. See how a business puts AI outputs to work →That demo uses a different model. This guide shows a general language model.

Where this analogy breaks down

A person reading

  • Draws on lived experience, senses, goals, and feelings.
  • Can have a personal reason to say something.
  • Can tell you why, and the reason can be real.

A language model

  • Uses learned numerical patterns and the text it was given.
  • Scores possible next pieces of text and selects one, over and over.
  • Can produce a convincing explanation that does not reveal how the answer was computed.
  • The meaning map flattens 768 numbers into 2. Most of the structure is not shown.
  • Attention here is averaged over the 12 heads in each layer. Individual heads do quite different things.
  • Attention shows where information can flow. It is not a complete explanation of an answer.
  • Reading out the middle layers (the "logit lens") is an approximation.
  • A token does not line up neatly with one idea, and one idea can span many tokens.
  • GPT-2 small is a small model from 2019. Today’s models are far larger and go through extra training, such as learning from human feedback, that this page does not cover.
  • A next-word probability is not factual certainty.
  • Chat products wrap the model in instructions, tools, and safety systems that are not shown here.
How modern models differ

GPT-2 small is from 2019. It has 124 million parameters and 12 layers, small enough to measure on a laptop, which is why it is used here. Today’s large models are built on the same core design, the transformer, with many more layers and far more parameters. Exact sizes are often not published.

Some newer models also write out intermediate steps before they answer, then read those steps back as more text. That is a second loop on top of the one shown here, and it is still text being scored and selected.

Researchers who study model internals, a field called interpretability, keep finding more structure than one page can show, such as internal features for specific concepts and steps that happen inside a single pass. The picture here is the starting point.

/ questions

Questions about how AI works

How does AI think?
Not the way a person does. A language model turns your text into tokens, turns each token into a long list of numbers, and passes those numbers through a stack of layers. At every layer, each token looks back at the tokens before it and its numbers are rewritten. After the last layer, the model scores every possible next token and one is selected. Then the whole process runs again for the next one.
What is a token?
A token is a piece of text the model reads as one unit: a whole word, part of a word, a space and a word together, or punctuation. GPT-2 has a vocabulary of 50,257 tokens.
Does AI understand words?
It represents them. Measurements show that a word like "bank" ends up with different numbers after a sentence about a river than after one about a check. Whether that counts as understanding is debated. What the model has is learned patterns, not experiences of rivers or money.
What are layers in a language model?
Layers are repeated processing steps. GPT-2 small has 12, and large modern models have many more. Each layer lets every token draw information from earlier tokens, a step called attention, and then transforms the result. Meaning and predictions build up across layers instead of appearing all at once.
How does AI choose the next word?
After the last layer, the model gives every token in its vocabulary a probability. A selection method then picks one, either the top choice or a random draw weighted by the probabilities. The chosen token is added to the text and the model runs again.
Is a high score the same as being right?
No. A score says how likely a token is to come next, given patterns in the text the model learned from. It does not say whether a statement is true. Asked to finish "Two plus two equals", GPT-2 small’s top choice is "one".

More AI demos: all playgrounds. Thinking about building with AI? Can I vibe code it myself?

Want AI working in your business?

Knowing how the model works is the start. The useful part is deciding where it helps, and wrapping it in rules you control.

Talk with me
RND·Tech & Design
Talk with me
},1791500332172]