Read the weights carefully

A visible map tells you how one mixture was formed. It does not, by itself, establish why a complete model made a decision.

Keep the claim as small as the evidence

In the default lab, the “rises” value receives about 0.576 of the mixture weight. That is an exact statement about one head, one row, and these invented states. It does not mean “57.6% of the model’s reasoning came from rises.”

The value intervention gives a narrower, demonstrable result: identical weights can carry different content and produce different outputs. A complete Transformer adds heads, projections, bypass paths, normalization, and nonlinear transformations. A heatmap omits these.

There is also empirical and methodological debate. Jain and Wallace found different attention distributions with similar predictions in the NLP models they tested. Wiegreffe and Pinter challenged the scope and definition of explanation and argued for task-dependent tests. Those studies largely concern recurrent attention classifiers, so they are not a universal experimental verdict on every Transformer.

Our inference is deliberately bounded: attention weights alone do not establish a faithful explanation of a model’s final prediction. They can still be informative measurements of an internal computation. A stronger explanatory claim needs a definition, a target prediction, and evidence from suitable interventions or other checks.

S2 · §4.2; pp. 3548–3550 CL-017

S3 · §§3.4–6; pp. 15–19 CL-017

Try a case the controls do not answer

A new attention row has query $q=[0,0]$ . Only the first and third keys are allowed. The keys can be any finite two-coordinate vectors, and the values are:

V=\begin{bmatrix}2&-1\\9&9\\-2&3\end{bmatrix}

What are the three weights and the output? Would replacing the middle value by $[100,-100]$ change it? What if every key were blocked?

Reveal the worked answer

The zero query makes all finite-key scores zero. Normalize over the two allowed positions: $a=[1/2,0,1/2]$ .

o=\tfrac12[2,-1]+\tfrac12[-2,3]=[0,1]

The middle value contributes zero even after the replacement. The first output coordinate cancels although both allowed weights are positive. With all keys blocked, no normalized distribution exists; the companion rejects the input instead of returning a pretend attention row.

New synthetic transfer and derivation · CL-019, CL-020. It uses an unfamiliar mask and negative values, not the lab’s default case.

One more change: reorder the tokens

Remove positional features and use all-visible self-attention. Swap the first and third rows of $X$ , keeping the same projection parameters. Should every output stay at its old row index?

Reveal the order argument

No. Each token’s query, key, and value move together. The output rows follow the same permutation. If you instead keep a fixed causal mask, the allowed token relationships change with the order, so the all-visible equivariance argument no longer applies unchanged.

You can now reconstruct the mechanism from a shape check, a visibility rule, and a weighted sum—and state exactly where that explanation stops. The notes keep the symbols, source locators, and runnable lab together for a later reread.

Derived transfer · CL-011.