What changes when KL direction changes?

Forward KL weights by the teacher

At one held-fixed prefix, forward KL places the teacher distribution first:

D_F(p\|q)=D_{\mathrm{KL}}(p\|q)=\sum_i p_i\log\frac{p_i}{q_i}.

Teacher probabilities weight the log-ratios. The value uses natural logarithms, so its unit is nats.

Reverse KL weights by the student

Reverse KL swaps the arguments:

D_R(q\|p)=D_{\mathrm{KL}}(q\|p)=\sum_i q_i\log\frac{q_i}{p_i}.

Student probabilities now weight the log-ratios. KL is asymmetric, so forward and reverse values need not match even though the prefix and both vectors are unchanged.

Reduce the two visited positions once

OPD-FIG-003 · OPD-TOY-001Exact KL balance sheet

Full-precision route values before six-decimal display rounding
Held-fixed prefix	Student q	Teacher p	Reverse KL q ‖ p	Forward KL p ‖ q
`x`	[0.75, 0.25]	[0.5, 0.5]	0.130812035941137	0.143841036225890
`xB`	[0.6, 0.4]	[0.9, 0.1]	0.311238679583058	0.226289161185359
Unweighted mean	two visited positions		0.221025357762097	0.185065098705625

The canonical contract reports the reverse-KL column: 0.130812, 0.311239, then 0.221025 nats after one final rounding step. The forward column is a same-prefix comparison, not a different source.

The canonical student path is [x,xB]. Its reverse-KL losses are 0.130812035941… and 0.311238679583… nats (six-decimal displays: 0.130812 and 0.311239). The contract uses an unweighted two-position mean:

\widehat L_R=\frac{D_R(x)+D_R(xB)}{2}=0.221025357762\ldots\ \text{nats}\ \longrightarrow\ 0.221025.

Compute at full precision and round the final mean once. Averaging two already rounded display values can change the last displayed digit.

A local gradient is not a gradient through collection

With the prefix and teacher fixed, the student logits can receive a gradient from either KL. That derivative answers how to change the student distribution at this state. It does not differentiate through the earlier categorical token draw in the canonical estimator.

Estimator boundary. Sequence-level reverse-KL policy-gradient methods can retain score-function terms from the sampling law. They are distinct estimators, not a hidden form of this detached local calculation.

The same surface words can therefore describe different training semantics.