Attention: How a Word Reads the Room

First entry in a new quest — Transformers from Scratch. Same rule as always: don’t state the formula, invent it from the problem up. So we start with a sentence that shouldn’t work:

“I sat on the bank and watched the water flow.”

You know instantly that bank means riverbank, not a place that holds money. But here’s the problem: a model first turns each word into a fixed vector — an embedding (plain: a list of numbers capturing the word’s general meaning, identical every time the word appears). And the embedding for “bank” is exactly the same in “river bank” and “money bank.” The dictionary entry can’t know which one you meant.

So what resolved it for you? The other words — “water,” “flow.” The information needed to fix “bank” was never in “bank”; it was sitting elsewhere in the sentence. This experiment builds the one mechanism that lets a word reach over and pull that information in. Verdict on the hypothesis that meaning is fixed by the embedding: FAR FROM IT.

The goal, stated plainly: rewrite each word’s vector as a blend of the other words’ vectors. The output we want is a contextual embedding — a new vector for “bank” that has absorbed the specific sentence it’s in — as opposed to the fixed static embedding it started with.

But not an equal blend. If “bank” mixed in “I,” “sat,” “on” as much as “water,” the clue would drown in junk. Each word has to listen more to some words and less to others.

↑ Click any word to make it the query — the rest glow by how much it listens. Click bank: it leans hard on water and flow, and its new vector becomes river-bank. That per-word list of “how much to listen” is what the rest of this post builds.

How does a word decide who to listen to? It has to search. And here’s the realisation that unlocks everything: a word plays two roles that shouldn’t share one vector — the thing it uses to go looking, and the thing it advertises so others find it. Those are different jobs.

So every word projects its embedding x through three learned matrices:

$q = xW_Q \qquad k = xW_K \qquad v = xW_V$

W_Q, W_K, W_V are three lenses the model learns during training — one turns a word into its search, one into its advert, one into its deliverable. Same word in, three different vectors out.

↑ Nudge the word and watch Query, Key and Value all shift together — they’re three views of one embedding, but never the same vector. The listing is not the house.

Now we match. How relevant is word j to word i? Take their query and key and run a dot product:

$\text{score}(q, k) = q \cdot k = \sum_{i} q_i k_i$

Big when they point the same way and are long; small when they’re orthogonal. Note we use the raw dot product, not cosine. Cosine — (q·k)/(‖q‖‖k‖) — divides the magnitude out. We deliberately keep magnitude: length is a learned volume knob a word can use to shout louder. We’ll tame the unboundedness a different way in a moment.

The scores are unbounded and don’t sum to anything tidy, so we push them through softmax to get weights that are all positive and add to 1:

$w_j = \text{softmax}(\text{scores})_j = \frac{e^{s_j}}{\sum_k e^{s_k}}$

And the new vector for the word is the weighted sum of the values — not the keys. The key was just the billboard that earned the match; the value is what actually gets delivered:

$\text{output}_i = \sum_j w_j\, v_j$

• ①For “bank,” softmax over its scores might land at water 0.30, flow 0.20, sat 0.15, … — so its output is ~50% water + flow. Mechanically, by a weighted average, “bank” has become riverbank.
• ②The keys and values come from every word, including “bank” itself. A word is always allowed to keep some of its own value.

The real formula has one term we haven’t justified: the scores are divided by √d_k (where d_k is the dimension of the query/key vectors) before softmax. This is the piece that separates people who memorised the formula from people who understand it.

Assume each component of q and k is roughly independent, mean 0, variance 1. The dot product sums d_k such terms, and variances add:

$\text{Var}(q\cdot k) = d_k \quad\Rightarrow\quad \text{typical } |q\cdot k| \approx \sqrt{d_k}$

So the score magnitude grows with the model’s width — ±2 at d_k=4, ±8 at 64, ±22 at 512. And a big score wrecks softmax: it saturates toward one-hot, putting ~all the weight on one word.

↑ Drag d_k up. The coral distribution (no scaling) collapses to one-hot and its learning signal p·(1−p) dies. The blue distribution (÷√d_k) never moves — the fix makes attention invariant to width.

Stack every word’s query into Q, keys into K, values into V, and the entire mechanism collapses to a single expression — self-attention, because every word queries every word of the same sentence:

$\text{Attention}(Q,K,V) = \text{softmax}\!\left(\frac{QK^{\top}}{\sqrt{d_k}}\right)V$

Every symbol earns its place: three projections because advertising ≠ delivering; a dot product because relevance is alignment; softmax because we need weights; √d_k because otherwise the whole thing stops learning.

One attention head produces a single distribution — one weighted average. But “bank” needs to attend to “water” for meaning and to “the” for grammar at the same time, and a single average can’t do both. And left unmasked, a word generating text could peek at the future it’s supposed to predict. Next up: multi-head attention, self vs cross attention, and the causal mask — why one head isn’t enough (EXP-014).