Order is information

A mask controls which positions are available. Positional features tell the computation where a state sits.

Remove a position before normalization

For a left-to-right prediction, a position must not consult future input tokens. The causal rule we use permits a key at position $j$ exactly when $j\le i$ , where $i$ is the query position. The current input token is included.

FIG-004Causal visibility is a row-by-row rule

Query rows read key columns. Allowed means key position ≤ query position.
Query ↓ Key →	grid	load	rises
grid	Allowed	Blocked	Blocked
load	Allowed	Allowed	Blocked
rises	Allowed	Allowed	Allowed

The diagonal is allowed: a token can read itself. Blocking happens before normalization, so each row forms a distribution over its own permitted positions. This three-token causal mask has no all-masked rows.

In the formula, add a mask $M$ to the scaled scores: zero for allowed entries and negative infinity for blocked ones. Operationally, the lab exponentiates only allowed entries.

A=\operatorname{softmax}_{\mathrm{rows}}\!\left(\frac{QK^{\mathsf T}}{\sqrt{d_k}}+M\right)

Blocking is not merely setting a score to zero: a zero score still gets $\exp(0)=1$ in the numerator. A blocked key gets exactly zero weight and is removed from the denominator. Renormalizing can change every remaining weight.

Every row in our causal mask has at least one allowed position. If you block all positions, there is no distribution to normalize. The companion rejects that case explicitly. This is a mathematical boundary, not a claim that all software libraries handle it identically.

S1 · §§3.1, 3.2.3 CL-007, CL-020

What a position-free mixture cannot notice

Imagine keeping a query fixed and moving each key together with its matching value. The scores and contributions simply move with their pairs. Adding the same terms in a different order gives the same output.

For all-visible self-attention with no positional information, permuting all input rows also permutes the query rows. The output therefore follows the same permutation: it is permutation-equivariant. It is not necessarily identical at each fixed row index.

For a short derivation, let $P$ be a matrix that reorders rows. Then the reordered weights are $PAP^{\mathsf T}$ , and:

O'=(PAP^{\mathsf T})(PV)=PAV=PO

This argument requires all-visible attention, or a mask reordered consistently with the rows. A fixed triangular causal mask changes which token may consult which other token when the order changes. Do not apply the unrestricted permutation argument to that different case.

Derived from shared row-wise projections and CL-001 · CL-011.

Supply a position-dependent signal

The original Transformer adds a positional vector to each input embedding. Its chosen encoding uses sine/cosine pairs at different frequencies. If $p$ is the position index and $r$ indexes a coordinate pair, let $d_{\mathrm{model}}$ be the token-state width. The original formula is:

\mathrm{PE}_{p,2r}=\sin\!\left(\frac{p}{10000^{2r/d_{\mathrm{model}}}}\right)

\mathrm{PE}_{p,2r+1}=\cos\!\left(\frac{p}{10000^{2r/d_{\mathrm{model}}}}\right)

In width two, the pair at position zero is $[0,1]$ and at position one is $[\sin(1),\cos(1)]$ . These are position-dependent numbers; adding them can make the same token at two positions produce different queries, keys, and values.

The paper also tested learned positional embeddings. Other choices exist; the sinusoidal design is the original choice described here. Our arithmetic lab does not add positional vectors. Its fixed rows and causal mask are deliberately exposed so there is no hidden position computation.

A mask and a position vector solve different parts of the problem. One restricts access; the other supplies features. Now we can place attention inside the rest of the architecture.

S1 · §3.5 CL-010