Module 1 · Foundations

LLMs and Transformers

A large language model reads a prompt and generates the next token, over and over, until it decides to stop. This page animates what happens inside: how tokens become vectors, how attention lets words shape each other's meaning, and how modern variants trade memory for quality. Start here; the mechanisms below recur in every later module.

The autoregressive generation loop

auto-play

An LLM generates text one token at a time. Each new token is appended to the input and fed back in — a loop that runs until an end-of-sequence token appears.

token / text weights (Transformer) probability distribution

The first step of that loop — turning words into vectors — is the embedding matrix. Models actually work on tokens (usually sub-word pieces), not whole words. Embedding is a fixed lookup that maps each token to a point in a d_model-dimensional space, where similar meanings sit close together. But a raw embedding is context-free: the bear in "a brown bear" starts identical to the bear in "doesn't bear fruit". Attention is what pulls them apart.

Self-attention in one head

step-through

Watch a single attention head disambiguate "bear" by letting it attend to "brown". Project the tokens into Q, K, V, score every pair, mask the future, softmax, and mix the values back in.

token vectors (context) learned weights Wᵀ attention scores causal mask

That square k × k grid is the heart of attention: cell A[i,j] is how much token i attends to token j. It is not symmetric, and during generation the upper triangle is masked so a token can never peek at words that come after it. One head can only surface the single most prominent relationship, though — so Transformers run many heads at once.

Multi-head attention: many views in parallel

auto-play

A layer splits its work across h heads. Each has its own Wᶜ/Wᴷ/Wᵛ and learns a different relationship; the outputs are concatenated and mixed by a shared Wᴼ.

shared input per-head attention Wᴼ output projection updated hidden state

Why this matters for inference Every head keeps its own K and V vectors, and during generation those are cached and reused for every future token (the KV cache, Module 2). With many heads that cache gets large fast — which is exactly the pressure the next figure's variants relieve.

Shrinking the KV footprint: MHA → MQA → GQA → MLA

step-through

All four compute attention — but they differ in how many distinct K/V projections they keep, which sets how much memory the KV cache consumes. Step through the trade-off.

query heads stored K / V KV memory compressed latent

The lever is simple: fewer distinct K/V projections means a smaller KV cache, usually at some cost to quality. MQA shares one K/V across all heads (memory ÷ h); GQA shares within g groups (memory ÷ h/g) and is the popular sweet spot — an MHA model can be converted with mean pooling and only ~5% of the original compute to uptrain. MLA instead stores one small compressed latent per token and lets each head decompress its own K/V, reaching GQA-like memory with MHA-like quality.

Mixture-of-Experts: route each token to a few experts

auto-play

Attention is unchanged. The one dense feed-forward network is replaced by many small expert FFNs plus a router that sends each token to only its top-k experts.

token router active expert idle expert

Because each token visits only k of the experts, an MoE layer holds the same total parameters as a dense one but activates only a fraction per token — more capacity for the same compute. The catch is the router: if it funnels everything to one expert, the others starve and quality collapses. Training adds noise to routing scores and minimizes an auxiliary (load-balancing) loss so tokens spread evenly across experts.