Attention is one component

Multiple mixtures feed a wider computation: projections, residual paths, normalization, and a shared transformation at each position.

Several views of the same states

Our lab has one head. In multi-head attention, each head has its own query, key, and value projections. Each computes a separate mixture, then the outputs are concatenated along their feature dimension.

H_h=\operatorname{Attention}(XW_Q^h,XW_K^h,XW_V^h)

\operatorname{MHA}(X)=\operatorname{Concat}(H_1,\ldots,H_H)W_O

Here $H$ is the head count; $H_h$ is head $h$ ’s output. If each head returns width $d_v$ , concatenation has width $Hd_v$ . The output matrix $W_O$ maps that width back to $d_{\mathrm{model}}$ , so it can be added to the input state.

Heads provide separately parameterized comparisons and mixtures. The architecture does not require one head to mean “grammar” and another “facts.” A readable semantic role would be a finding to investigate, not something the formula guarantees.

S1 · §3.2.2 CL-012, CL-024

Follow the bypass as well as the mixture

An encoder block in the original Transformer contains all-visible self-attention followed by a position-wise feed-forward sublayer. Each sublayer has a residual addition and then layer normalization. “Residual” means that the incoming state has an additive path around the sublayer.

FIG-005The original encoder block: add, then normalize

Original encoder structure, with dropout omitted from this drawing. Each bypass rejoins at + before LayerNorm (post-norm). The decoder additionally uses causal self-attention and encoder-decoder attention; a pre-norm variant places normalization differently.

U=\operatorname{LN}\bigl(X+\operatorname{MHA}(X)\bigr)

Y=\operatorname{LN}\bigl(U+\operatorname{FFN}(U)\bigr)

These equations show the original Post-LN order, with dropout omitted for the inference-time schematic. The original training setup uses dropout on sublayer outputs before the addition and normalization. Pre-LN places normalization inside the residual branch; it is a distinct variant, not the layout drawn here.

S1 · §3.1, Figure 1; §5.4 CL-013

S4 · abstract: normalization placement CL-016

Layer normalization acts across a token’s feature coordinates in this block. It centers and rescales them, then applies learned coordinate-wise gain $\gamma$ and bias $\beta$ . For token vector $z$ of width $D$ , one stabilized convention is:

\mu=\frac{1}{D}\sum_{r=1}^{D}z_r,\qquad \sigma^2=\frac{1}{D}\sum_{r=1}^{D}(z_r-\mu)^2

\operatorname{LN}(z)_r=\gamma_r\frac{z_r-\mu}{\sqrt{\sigma^2+\epsilon}}+\beta_r

The small positive $\epsilon$ avoids division by zero. This normalizes a feature vector; it is separate from softmax normalizing weights across key positions. For example, $z=[1,3]$ has mean 2 and variance 1 before stabilization.

S5 · §3, equation (3); §5, equation (5) CL-015

Mix positions, then transform features

Attention combines information across positions. The original feed-forward network then applies a two-layer transformation independently to each position, using the same parameters across positions within that layer:

\operatorname{FFN}(u)=\max(0,uW_1+b_1)W_2+b_2

The first linear map changes feature width; the elementwise maximum is the ReLU nonlinearity; the second map returns to the model width. If $u$ has width $d_{\mathrm{model}}$ , $W_1$ is $d_{\mathrm{model}}\times d_{\mathrm{ff}}$ and $W_2$ is $d_{\mathrm{ff}}\times d_{\mathrm{model}}$ . Different layers may have different parameters. This per-position processing does not introduce another direct token-to-token mixing step.

The residual paths, output projection, and feature transformations mean the single-head convex-mixture property does not describe the final Transformer state. Even the attention output is combined with information that bypassed it.

S1 · §3.3, equation (2) CL-014, CL-023, CL-024

Locate the prediction boundary

The original decoder differs from the encoder diagram above. It uses causal self-attention over decoder states, adds cross-attention to the encoder output, and has its own feed-forward sublayer. In cross-attention, queries come from the decoder; keys and values come from the encoder output. The two sequence lengths can differ, which is why we kept $n$ and $m$ separate in the shape rule.

Training targets are shifted: the state at one decoder input position is used to predict the next target token. Including the current input token in a causal mask does not reveal that future label. At the top of the decoder, a learned linear map and vocabulary softmax turn states into next-token probabilities. The attention softmax inside a head is a different distribution.

Learning would require a loss and parameter updates through this computation. Our lab performs neither. It stops at one attention output so that the mechanism remains hand-checkable.

S1 · §§3.1–3.4 CL-002, CL-007, CL-013, CL-018, CL-021