How Does It Even Know Word Order?

We’ve built the whole attention sub-layer — scaled dot-product, multi-head, masking, cross-attention. It works. And yet it has a hole so basic it’s almost funny. Consider two sentences:

“dog bites man”  vs  “man bites dog”

Same words, opposite meaning — the difference is entirely word order. So here’s the hypothesis I want to test: attention understands word order; feed it a sentence and it knows which word came first. We’ll prove it wrong on the very first interactive, then spend the rest of the experiment fixing it. Verdict, as usual, up front: FAR FROM IT.

Recall the mechanism: a word’s output is softmax(q·k / √d_k) weighted over the values. Every piece of that — the dot product, the softmax, the weighted sum — depends only on which words are present, never on where they sit. Permute the inputs and you permute the outputs, nothing more:

$\text{Attention}(\pi X) = \pi\,\text{Attention}(X)$

That’s the definition of permutation invariance (plain: reshuffling the inputs only reshuffles the outputs — no new information). Watch it directly. Below, the query word “bites” attends to the other words in two different orders:

↑ With positional encoding OFF, the output for “bites” is bit-for-bit identical across both orderings — it genuinely cannot tell who bit whom. Flip it ON and the two orderings finally diverge.

So we add a position signal to each word’s embedding. The obvious first tries both fail, and failing them tells us exactly what we need.

Attempt 1 — the raw index. Add the number pos (0, 1, 2, …) to the embedding. But the index is unbounded: by position 5000 you’re adding a value of 5000 to vectors whose entries are around ±1. The position doesn’t inform the word, it obliterates it. We need something bounded.

Attempt 2 — normalise by length, pos / length. Now it’s bounded to [0, 1]. But it’s inconsistent: the 3rd word is 3/5 = 0.60 in a five-word sentence and 3/50 = 0.06 in a fifty-word one. The same absolute position produces a different number depending on how long the sentence happens to be, so “third word” is never a stable signal. We need something bounded and consistent across lengths — the same position always encoded the same way.

Here’s the trick. Encode each position not as one number but as a whole vector of sines and cosines at geometrically-spaced frequencies:

$PE_{(pos,\,2i)} = \sin\!\left(\frac{pos}{10000^{\,2i/d}}\right), \qquad PE_{(pos,\,2i+1)} = \cos\!\left(\frac{pos}{10000^{\,2i/d}}\right)$

Every value is a sine or cosine, so it’s bounded to [−1, 1]. Low dimensions use long wavelengths (they crawl across positions); high dimensions use short ones (they race). Together they form a smooth binary counter — a unique fingerprint per position that means the same thing regardless of sentence length.

↑ Drag the position slider. Each row is one position’s code across all dimensions; the two waves show a slow dimension and a fast one. No two rows are alike, and every value stays in [−1, 1].

Uniqueness and boundedness we could get a hundred ways. The property that makes sinusoids special — the one worth understanding — is what happens when you move a fixed distance. Recall the angle-addition identity:

$\sin(pos + k) = \sin(pos)\cos(k) + \cos(pos)\sin(k)$

Read that carefully: the encoding at pos + k is a fixed linear combination of the encoding at pos. For a sin/cos dimension pair it’s exactly a rotation by the angle k·freq:

$\begin{bmatrix}\sin(pos+k)\\[2pt]\cos(pos+k)\end{bmatrix} = \underbrace{\begin{bmatrix}\cos k & \sin k\\[2pt]-\sin k & \cos k\end{bmatrix}}_{\text{depends only on }k}\begin{bmatrix}\sin(pos)\\[2pt]\cos(pos)\end{bmatrix}$

The rotation matrix depends only on k, not on pos. So “the word k positions back” is the same linear transform everywhere in the sentence — the model learns it once and it works at every location. That is why positions are sinusoids and not random unique vectors: relative position becomes linear, and linear is exactly what a neural network is good at.

↑ Drag pos: both dots swing around, but the angle between them — the shift of k — stays locked. A fixed rotation, independent of position.

Mock question. “You trained on sequences up to 512 tokens and it’s great. In production a user pastes a 2000-token document and quality falls off a cliff. Why?” This is length extrapolation, and it splits the schemes apart.

This is exactly why modern LLMs moved to RoPE (rotary embeddings) and ALiBi: instead of adding an absolute code, they bake relative distance directly into the attention scores — leaning on the very rotation property we just derived — and extrapolate to lengths far beyond training. The sinusoid was the right idea; RoPE is the idea taken to its conclusion.

• ①Learned absolute positions have a hard cap at the training length — position 2000 simply doesn’t exist for a 512-trained model.
• ②Sinusoids extrapolate numerically but not reliably; RoPE/ALiBi encode relative distance and hold up far better — the direct payoff of shift-by-k being linear.

With position injected, every attention block on the blueprint is now something we’ve derived by hand. Here’s Attention Is All You Need, Figure 1, with the whole attention path lit — including, finally, the positional-encoding inputs at the bottom:

↑ Every coral block is one we invented from its problem. What remains grey — embeddings, Add & Norm, feed-forward — is the plumbing that holds the attention together.

That’s the attention sub-layer, end to end: scaled dot-product (EXP-013), multi-head + masking + cross-attention (EXP-014), and now position (EXP-015). What turns it into a full transformer block is the un-glamorous scaffolding wrapped around it — residual connections, LayerNorm, and the position-wise feed-forward network — and then the architecture forks into BERT (encoder-only) and GPT (decoder-only). The hard, beautiful idea, though, you now own outright.