What does the teacher provide at xB?

What does the teacher provide at xB?

At the student-visited prefix xB, the student assigns [0.6,0.4] to [A,B]. Querying the teacher at that same prefix returns [0.9,0.1].

The teacher's probability vector is dense token-distribution supervision. It is not automatically a scalar reward, a correctness certificate, or process-level causal feedback.

Source-backed: GKD Eq. 2 evaluates teacher and student next-token distributions conditioned on the identical prefix. See the author manuscript §3.

The local comparison begins after the state is fixed

For a chosen state $s$, abbreviate the teacher probabilities by $p_i=p(i\mid s)$ and the student probabilities by $q_i=q_\theta(i\mid s)$. A declared divergence maps these aligned vectors to one scalar:

The state source is metadata about how this occurrence entered the batch. Changing the divergence does not rewrite that history.

Do not hide feedback granularity

This guide's canonical toy compares the complete two-token vocabulary. Other recipes may expose one sampled token, a hard teacher action, a top-k truncation, or a scalar reward. That objective granularity changes what information is available even when each signal is casually called “teacher feedback.”

Toy assumption. Teacher and student share the aligned vocabulary [A,B]. Cross-tokenizer mapping, EOS handling, and variable-length weighting remain outside the fixture.

With the state held fixed, KL direction finally becomes the question.