One state, three roles

A token needs a way to compare with its neighbors—and something to receive from them.

Start with a row of numbers

Read the phrase grid load rises. The word “rises” is more useful when connected to what rises. In a model, that connection has to become a numerical operation. Attention lets one token state combine information from other token states.

If you have seen an embedding, start there: a vector is an ordered list of coordinates. Put one vector on each row of a matrix $X$ . In a deeper layer, these rows are already contextual states produced by earlier computation. They need not remain the original embedding-table entries.

A recurrent model carries state along a sequence. Here we will instead calculate a set of weights from one row to the available rows, then form a mixture. We are tracing a forward calculation; the stored model parameters stay fixed.

TOY ASSUMPTION · CL-004

Our labels are grid, load, and rises, with invented states $[1,0]$ , $[0,1]$ , and $[1,1]$ . They were chosen for arithmetic, not learned from text. Neither coordinate is a measured electrical quantity or a verified semantic feature. The example makes no claim about forecasting or document understanding.

Compare with keys; carry values

For each row, create three projections. A query compares with keys. A value is the vector that will enter the mixture. These are roles played by numeric states, not three kinds of token.

Q=XW_Q,\qquad K=XW_K,\qquad V=XW_V

A projection is matrix multiplication by a parameter matrix. We use row vectors throughout. For our toy, choose these fixed matrices:

W_Q=\begin{bmatrix}1&0\\0&1\end{bmatrix},\quad W_K=\begin{bmatrix}1&1\\0&1\end{bmatrix}

W_V=\begin{bmatrix}1&0\\0&2\end{bmatrix}

For “rises,” multiply $[1,1]$ by $W_K$ : the first output is $1\cdot1+1\cdot0=1$ ; the second is $1\cdot1+1\cdot1=2$ . So its key is $[1,2]$ . The same row has query $[1,1]$ and value $[1,2]$ .

FIG-001One input row, three projected roles

Invented token states and their fixed query, key, and value projections. Each matrix has three rows and two columns.
Token / row	Input X	Query Q	Key K	Value V
gridrow 0	[1, 0]	[1, 0]	[1, 1]	[1, 0]
loadrow 1	[0, 1]	[0, 1]	[0, 1]	[0, 2]
risesrow 2	[1, 1]	[1, 1]	[1, 2]	[1, 2]

Synthetic coordinates. Read across a token row: the same input passes through three fixed matrices. The columns of each vector are coordinates, not probabilities. No position encoding is included in this toy.

The matching key and value coordinates for “rises” are a coincidence of these constants. Keys and values need not have the same width. In a trained model, learning changes the projection parameters; a new input changes the projected states even when those parameters are fixed.

S1 · §§3.2.2–3.2.3 CL-002, CL-003, CL-018

Read the shapes before the numbers

Let $n$ be the number of query rows, $m$ the number of key/value rows, $d_k$ the query/key width, and $d_v$ the value width. The transpose symbol $\mathsf T$ swaps a matrix’s rows and columns.

\underbrace{Q}_{n\times d_k}\;\underbrace{K^{\mathsf T}}_{d_k\times m}\;\longrightarrow\;\underbrace{S}_{n\times m}

\underbrace{A}_{n\times m}\;\underbrace{V}_{m\times d_v}\;\longrightarrow\;\underbrace{O}_{n\times d_v}

The first multiplication gives one comparison for every query/key pair. The second uses one weight per value row. Query and key widths must match; key and value row counts must match. $d_v$ may differ from $d_k$ . In our self-attention example, $n=m=3$ and both widths happen to be 2.

We have candidates and comparisons. Next we need to turn a row of scores into a controlled mixture.

Derived shape contract · CL-003, from S1 equation (1).