Keep the definitions close.

Notation, primary sources, and a calculation you can reproduce.

A compact glossary

Token state, X

A row of features supplied to this layer. In the lab these are invented constants.

Parameter versus activation

A parameter such as W is stored and ordinarily learned. An activation such as Q, K, V, A or O is computed for the current input. Sliders here directly intervene on activations.

Query / key width, d_k

The shared number of coordinates used in a dot product. Values can have a different width d_v.

Score and attention weight

A score is a compatibility scalar. Row-wise softmax converts allowed scores into mixing weights summing to one.

Causal mask

At query i, permit keys j ≤ i. Include the current input; exclude future input positions.

Attention output, O

A weighted sum of values per query. It is a state vector, not a vocabulary distribution.

Residual path

An additive route around a sublayer. In the illustrated original encoder block, addition precedes layer normalization.

Projection and head

A head uses its own learned Q/K/V projections. Multi-head outputs are concatenated and projected back to model width.

Notation convention

Rows are tokens; vectors multiply parameter matrices on the right. Mask and worked-example position labels start at 1. Sinusoidal position p may start at zero; source-code row indices are 0, 1, 2. All toy coordinates are dimensionless.

Sources behind the explanation

Primary sources inspected on 4 September 2026. Citations beside the prose carry stable claim IDs. The project’s evidence ledger distinguishes source-backed facts, derivations, toy assumptions and qualified inferences.

1. Vaswani et al. — Attention Is All You Need

   Original architecture: §§3.1–3.5, equations (1) and (2), Figure 1; residual dropout in §5.4. 2017 paper, inspected arXiv v7.

2. Jain & Wallace — Attention is not Explanation

   NAACL 2019. §4.2, pp. 3548–3550: attention interventions in the tested NLP models. This is evidence about those settings, not every Transformer.

3. Wiegreffe & Pinter — Attention is not not Explanation

   EMNLP 2019. §§3.4–6, pp. 15–19: methodological critique, explanatory definitions, and task-dependent evidence.

4. Xiong et al. — On Layer Normalization in the Transformer Architecture

   ICML 2020. Abstract identifies the Post-LN / Pre-LN placement distinction. No optimization-performance claim is made here.

5. Ba, Kiros & Hinton — Layer Normalization

   §3, equation (3), and §5, equation (5): within-example feature statistics and learned gain and bias.

Reproduce the attention row

The canonical JavaScript module (local copy) supplies the lab and every static comparison. Run node model.mjs from the site folder for baseline, controlled contrasts and transfer output. Run npm test for the numerical and build checks.

Numerical tests (local copy) cover hand cases, stable softmax, invalid/all-masked rows, projection immutability, permutation behavior and all 702 UI states. These checks verify the implemented algebra, not reader understanding or real-model performance.

There is no random seed because no random sampling occurs. JavaScript double precision is used; printed UI numbers have three decimals. All state vectors and matrices are synthetic, and no optimizer runs. The lab omits positional features, multi-head combination, output projection, residual additions, normalization and feed-forward layers; chapter 5 explains those components without claiming to simulate them.