STATES · COMPARISONS · MIXTURES Attention and One token at a time. Follow three states through a small calculation—and see exactly what changes.
Start reading → One query, three attention weights Synthetic example: query rises has attention weights grid 0.284, load 0.140, and rises 0.576. Each bar uses the same zero to one scale. The guide traces how those weights mix value vectors. ONE QUERY · THREE KEYS rises asks; values combine. grid 0.284 load 0.140 rises 0.576 Three weights, one mixture. Invented states make the computation visible; these are not learned token relationships.
A field guide for applied ML reading. Refresh vectors and notation as you go, calculate a single attention row, then place it inside a Transformer block. Six chapters, one runnable lab, and a complete script-free reading edition.
01 One state, three roles A token needs a way to compare with its neighbors—and something to receive from them.
↗ 02 From scores to a mixture Three comparisons become three weights. The result is a vector, not a selected word.
↗ 03 Change one thing Keep the parameters fixed. Intervene on a query, a visibility rule, or a carried value.
↗ 04 Order is information A mask controls which positions are available. Positional features tell the computation where a state sits.
↗ 05 Attention is one component Multiple mixtures feed a wider computation: projections, residual paths, normalization, and a shared transformation at each position.
↗ 06 Read the weights carefully A visible map tells you how one mixture was formed. It does not, by itself, establish why a complete model made a decision.
↗