From scores to a mixture

Three comparisons become three weights. The result is a vector, not a selected word.

Score one query against every key

Take the last query, “rises”: $q=[1,1]$ . A dot product multiplies matching coordinates, then adds. Its raw scores against our three keys are:

qk_1^{\mathsf T}=2,\qquad qk_2^{\mathsf T}=1,\qquad qk_3^{\mathsf T}=3

For example, the third score is $1\cdot1+1\cdot2=3$ . These are compatibility scores under the chosen projections. They are not probabilities or verified statements of importance. Dot products also depend on vector magnitude; they are not automatically cosine similarity.

The original scaled dot-product rule divides each score by $\sqrt{d_k}$ . Here $d_k=2$ , so the scaled scores are approximately $[1.414,0.707,2.121]$ .

Why the square root? Under the illustrative assumption of independent, zero-mean, unit-variance query/key components, the variance of their dot product grows as $d_k$ . Dividing by $\sqrt{d_k}$ keeps that variance at one. Large score differences can push softmax toward saturation. Learned states need not satisfy those assumptions, so the scaling is a motivated design choice, not a universal bound on scores or gradients.

S1 · §3.2.1 and its variance footnote CL-001, CL-005, CL-006, CL-025

Normalize across the available keys

Softmax turns the scaled scores $s_j$ into positive weights. Exponentiation makes the numbers positive; division makes the row sum to one. For now, all three positions are allowed.

a_j=\frac{\exp(s_j)}{\sum_{\ell=1}^{3}\exp(s_\ell)}

The denominator is shared. A weight depends on its competitors’ scores as well as its own. For “rises,” this gives $a\approx[0.284,0.140,0.576]$ . These probabilities are over key positions, not over the vocabulary and not over whether a sentence is true.

One row at a time, the matrix formula is:

A=\operatorname{softmax}_{\mathrm{rows}}\!\left(\frac{QK^{\mathsf T}}{\sqrt{d_k}}\right),\qquad O=AV

For numerical stability, the lab subtracts the largest allowed score before taking exponentials. Multiplying every numerator and the denominator by the same factor leaves the weights unchanged. All displayed values are rounded to three decimals; the calculation keeps full floating-point precision.

S1 · §3.2.1, equation (1); stability is an algebraic derivation CL-001, CL-020

Mix vectors, not token labels

Now multiply each value vector by its weight and add the resulting vectors:

o=\sum_{j=1}^{3}a_jv_j

o\approx0.284[1,0]+0.140[0,2]+0.576[1,2]

FIG-002Weights choose a mixture; values supply its coordinates

Query: rises, q = [1, 1]. All three keys are allowed.

Default attention weights, value vectors, and weighted contributions. Bar lengths share a zero to one scale.
Key token	Weight · 0 to 1	Value	Weight × value
grid	0.284	[1, 0]	[0.284, 0.000]
load	0.140	[0, 2]	[0.000, 0.280]
rises	0.576	[1, 2]	[0.576, 1.152]

Sum of contributions [0.860, 1.432]

Derived from the fixed toy model. Each weight multiplies the entire value vector. The output is a two-coordinate state, not a prediction or a probability distribution. Values shown to three decimals; calculation uses full precision.

The output is approximately $[0.860,1.432]$ . It is a new two-coordinate state. Nothing in this tiny operation decodes a word or decides what happens to a grid.

Because the weights are nonnegative and sum to one, the result is a convex mixture of the allowed values: a point inside their convex hull. Each output coordinate lies between that coordinate’s smallest and largest allowed value. The values themselves can be negative or exceed one; probability bounds belong to the weights.

Before the lab, make one prediction: if the query becomes the zero vector, will the output become zero too?

Derived arithmetic and convex-mixture property · CL-001, CL-005, CL-008, CL-022.