You've used ChatGPT, Claude, or Copilot. You type a message, and text streams back — word by word, as if the model is thinking out loud. That streaming response is a Large Language Model at work.

An LLM takes text in and produces text out. It doesn't browse the internet in real time. It doesn't have a database it queries. It generates each word based on patterns it absorbed during training — patterns extracted from a massive slice of the internet.

That's the surface. Underneath, every behavior you've noticed — the confident wrong answers, the uncanny understanding of your question, the word-by-word streaming — traces back to a specific set of design decisions made over the last 25 years.

This module walks that path. By the end, you'll understand why LLMs work the way they do — not as magic, but as engineering.

Before diving in, here's the full journey we'll trace — each step a solution to the previous step's failure:

• ~2000 — Bag-of-Words: Count words. Works for spam filters, breaks on anything nuanced.
• 2013 — Word2Vec: Encode meaning as vectors. king − man + woman ≈ queen. But every word gets one fixed meaning.
• 2014 — Attention: Let words look at each other. "Bank" shifts meaning depending on neighbors. But still sequential — painfully slow.
• 2017 — Transformer: Process everything in parallel. Trainable on the entire internet.
• 2018+ — BERT & GPT: Two shapes of the same architecture. Encoders understand, decoders generate.
• 2020+ — Scale: Bigger Transformers develop emergent capabilities. The LLM era begins.

Every step solved a concrete failure of the one before. That's the story.

When you type "bank" to ChatGPT, it knows from context whether you mean a river bank or a financial institution. But computers don't understand words — they understand numbers.

The word "bank" means something completely different next to "river" than it does next to "loan." To a machine, it's just a string of characters until someone figures out how to turn meaning into numbers.

A vector is simply a list of numbers that represents something — a word, a sentence, an idea. The entire history of language AI is the story of building better vectors: lists of numbers that capture more and more of what words actually mean.

This is Nina's world. She's a product engineer — she's been using LLMs at work for drafting and code, but now her team is building an AI feature into their product. She realizes she doesn't actually understand how any of it works. Let's follow her journey.

The earliest approach was brutally simple: count the words.

In the Bag-of-Words model, a document becomes a vector of word counts. "Bank" and "loan" together score high for finance. "Bank" and "river" score high for geography. The technique dominated early NLP — spam filters, search engines, and document classifiers all relied on it through the 2000s.

But it has a fatal flaw: it destroys all context. Word order, grammar, and sentence structure vanish completely. "The dog bit the man" and "The man bit the dog" produce identical vectors — same words, same counts. The model has no idea that the meaning changed entirely.

For classification tasks where topic matters more than nuance, this was good enough. But for anything requiring real understanding — summarization, translation, conversation — counting words is a dead end. Something needed to capture not just which words appear, but what they mean.

In 2013, Word2Vec had a breakthrough: instead of counting words, position them in a mathematical space based on how they're used.

Words that appear in similar contexts land close together. "Puppy" and "dog" cluster near each other. "Puppy" and "database" are far apart. The model learns this by reading billions of sentences and noticing which words keep the same neighbors.

The famous result: king − man + woman ≈ queen. Meaning encoded as math — relationships between concepts captured as directions in vector space.

Each word gets an embedding — a vector of perhaps 300 numbers representing its properties. Think of it like a scorecard: "puppy" might score high on "animal" and "young" but low on "abstract" and "digital." These scores aren't hand-coded — the model discovers them by observing language patterns across massive text.

This was the leap from counting to meaning. But Word2Vec had its own critical flaw — one we'll see next.

Word2Vec gave every word one fixed vector. "Bank" got the same embedding whether it appeared next to "river" or "loan." The model couldn't tell which meaning was active in a given sentence.

In 2014, a mechanism called attention changed everything. Instead of a fixed vector, attention lets each word dynamically look at the other words around it and adjust its meaning.

When processing "I deposited money at the bank," attention shifts "bank" toward its financial meaning — because "money" and "deposited" are nearby. In "I sat by the river bank," the same word shifts toward geography. Same word, different vector depending on context.

Attention works by computing a relevance score between every pair of words. High score means "these words should influence each other's meaning." Low score means "ignore each other." The result: contextual embeddings — vectors that change based on the sentence they're in.

This solved the meaning problem. But attention was still paired with a crippling bottleneck.

Attention solved the context problem — but the models using it had a fatal bottleneck.

Early attention was paired with Recurrent Neural Networks (RNNs) — models that process text one token at a time, left to right. (A token is roughly a word or word-piece — the unit an LLM reads and generates.) To process token 50, you must first process tokens 1 through 49, in order. No amount of GPUs could speed up that sequential chain.

In 2017, a Google team published Attention Is All You Need. The claim: you don't need recurrence at all. Attention alone is sufficient.

The Transformer processes all tokens simultaneously. Every token attends to every other token in the same forward pass. On GPU hardware, this parallelism translates directly into speed — making it feasible to train on the entire internet.

This single architectural change unlocked the scaling era. The Transformer is the foundation of every major LLM today: GPT, Claude, Gemini, LLaMA.

Not all Transformers are the same. The architecture split into two variants.

Encoder-only models (BERT-style): Process text in both directions. Every token attends to tokens before AND after it — excellent at understanding text (classification, sentiment, search). But they can't generate new text.

Decoder-only models (GPT-style): Process text left to right only. Each token only sees previous tokens — called causal attention. This restriction enables text generation: the model predicts each next token without "cheating" by looking ahead.

Modern LLMs — GPT-4, Claude, LLaMA — are all decoder-only. When you send a prompt and get a streaming response, that's a decoder doing causal attention, one token at a time.

Encoder models still matter for search and classification. But the LLMs you interact with daily are all decoders. Many production systems combine both — an encoder understands your search query, a decoder generates the answer.

Now the story connects.

Decoder-only Transformers scaled remarkably well. More data, more parameters, more compute — and new capabilities appeared: arithmetic, code generation, reasoning. This is the scaling law: larger models become qualitatively more capable in predictable ways.

The race to scale these models produced what we now call LLMs. Each one is trained in two phases: first pre-trained on massive text (learning language patterns), then fine-tuned to follow instructions and be helpful. The base model is a pattern-completion machine; the chat model you interact with is the fine-tuned version.

Every behavior traces back to this arc. When an LLM hallucinates, that's lossy compression — trillions of words squeezed into billions of parameters. When it streams responses word by word, that's causal attention predicting one token at a time. When it seems to understand your question, that's contextual embeddings attending to relevant parts of your prompt.

Nina now had the history. But what IS an LLM, physically — two files on a laptop, not a cloud of magic. That's next.