Two axes, one overloaded phrase

“On-policy” tells you where a trajectory came from. It does not tell you which way the KL points.

A shortcut that breaks the objective

Suppose a student language model generates a response, a teacher is queried along that response, and the loss is reverse KL. Which part makes the update on-policy?

The tempting answer is “reverse KL.” It is also the wrong coordinate. Reverse KL names the order of two distributions inside a local comparison. On-policy names the source of the response trajectory—and therefore the prefixes at which that comparison is evaluated.

This monograph follows that factorization all the way down. By the end, you will be able to point to the source measure, freeze one prefix, choose the local loss, and say exactly which dependence the gradient estimator keeps.

First, make “policy” small enough to inspect

A policy need not arrive with rewards, value functions, or a simulator. Here it is simply a conditional next-token distribution. Given a prompt $x$ and earlier response tokens $y_{<t}$ , the student returns

MATH-EQ-POLICY-001

q_\theta(\cdot\mid x,y_{<t})\in\Delta^{M-1}.

Read it as: at one prefix, the student assigns one probability to each of $M$ vocabulary items. The vector is unitless and sums to one.

A response is built one token at a time, so its probability factors into those conditionals:

MATH-EQ-AR-001 · derived

q_\theta(y\mid x)=\prod_{t=1}^{T}q_\theta(y_t\mid x,y_{<t}).

An early draw does more than add one token. It determines which later prefix is visited.

Call that prefix-state $s_t=(x,y_{<t})$ . Once a trajectory has been generated, it gives us a sequence of states $s_1,s_2,\ldots$ at which a teacher can be queried.

Put source and loss on different coordinates

The inspection plane

OUTER MEASUREWhere did the response and visited prefixes come from?

fixed data ↔ mixed ↔ current student

INNER COMPARISONHow are teacher and student compared at a held-fixed prefix?

forward KL · reverse KL · JSD · another declared divergence

Choose one point from each axis. Neither coordinate determines the other.

A fixed prompt set does not settle the outer coordinate. GKD samples $x$ from a prompt collection $X$ , then can sample the response $y$ from the current student. The prompt is fixed; the completion trajectory is on-policy for the student at collection time.

boundary “At collection time” matters. Reusing trajectories after the student changes can make the behavior policy stale. A method name does not reveal freshness; an implementation must.

Classify before calculating

Take two recipes:

Student response + forward KL. The trajectory is on-policy; the local direction is forward.
Fixed teacher response + reverse KL. The local direction is reverse; the supplied trajectory is off-policy relative to the current student.

Those combinations are not contradictions. They are two cells in a product space. The next chapter writes the outer coordinate as an actual measure rather than an adjective.

02

Where the measure lives

A sampled trajectory induces the prefix states. The local loss arrives only after those states exist.

Turn trajectories into a distribution over visited prefixes

The autoregressive product tells us how a response is generated. To expose the measure beneath the loss, we need one more object: the distribution over prefixes induced by a response source.

Let $\rho$ be any distribution over prompt–response pairs. Its normalized prefix occupancy is

MATH-EQ-OCC-001 · derived

d_\rho(s)=\mathbb E_{(x,y)\sim\rho}\!\left[\frac{1}{T(y)}\sum_{t=1}^{T(y)}\mathbf 1\{s=(x,y_{<t})\}\right].

Sample a sequence from $\rho$ , give each of its $T(y)$ visited positions weight $1/T(y)$ , then ask how much mass lands on state $s$ .

length boundary This preserves GKD's per-sequence mean. Pooling all tokens across variable-length sequences would weight long responses more heavily. The toy fixes two evaluated positions and does not define a universal EOS rule.

Do not let five operations collapse into one verb

“Distill on student outputs” compresses an entire computation. Expanding it reveals where the source measure stops and the local derivative starts.

MATH-FIG-001 · sourced + teaching decompositionFive operations, two kinds of arrow

01SampleStudent emits a response token. The draw is discrete.
02VisitThe emitted token becomes part of the next prefix.
03EvaluateTeacher and student are queried at that same prefix.
04CompareA chosen local divergence returns a scalar in nats.
05UpdateDifferentiate local student logits; the sampled path stays fixed.

solid data and state flow dashed gradient/update

Read the edge types. The sampled token travels along the solid path. The gradient begins only after a prefix has been fixed and the local loss has been formed. This five-label decomposition is ours; its operations follow GKD Eq. 2, Eq. 4, and Algorithm 1.

The separation is useful even if the real implementation fuses teacher and student forward passes. Conceptually, collection decides which states exist in the batch; the local loss decides how probability mass is compared at each state.

The toy makes one branch reproducible

OPD-TOY-001 At prefix x, the student distribution is [0.75, 0.25] in vocabulary order [A,B]. The contract supplies a uniform value of 0.8, so inverse-CDF sampling selects B and the next visited prefix is xB.

MATH-FIG-003 · toyA prefix string does not carry its own provenance

prefixxstudent [0.75, 0.25]

A · probability 0.75xAsupplied fixed occurrenceCould also be student-generated under another draw.

B · probability 0.25xBrealized student occurrenceu_token = 0.8 lands in [0.75, 1).

Exact branch intervals
Token	Student interval at x	This fixture
A	[0, 0.75)	not selected by u = 0.8
B	[0.75, 1)	selected → xB

The contract labels the supplied xA occurrence as the off-policy comparison. That label belongs to how this occurrence was collected—not to the characters “xA.”

The draw is fixed so we can audit every later number. It is still a sample realization, not the exact expectation over all student responses. At the same time, the displayed xA comparison should not be treated as intrinsically off-policy: another student draw could reach the same string.

Now place a divergence on the selected path

For each state $s_t=(x,y_{<t})$ on the selected response, query teacher $p$ and student $q_\theta$ at that identical context. GKD's token-distribution reduction is

MATH-EQ-REDUCE-001 · sourced form

\overline D(\theta;x,y)=\frac{1}{T(y)}\sum_{t=1}^{T(y)}D\!\left(p(\cdot\mid s_t)\,\middle\|\,q_\theta(\cdot\mid s_t)\right).

The response selects the states. The symbol $D$ then compares two full next-token distributions at each one and returns a scalar.

The author manuscript's Eq. 2 uses exactly this position-average structure. It does not force $D$ to be reverse KL.

Cross the source measure with the local choice

Write $\mathcal D_{\mathrm{fix}}$ for a fixed prompt–response distribution and $q_{\bar\theta}$ for a behavior-policy snapshot. At collection, $\bar\theta$ may equal the current student. During the canonical GKD-style update, the sampled response is held fixed:

MATH-EQ-MIX-001 · GKD factorization

\begin{aligned}L_\lambda(\theta;\bar\theta)={}&(1-\lambda)\,\mathbb E_{(x,y)\sim\mathcal D_{\mathrm{fix}}}[\overline D(\theta;x,y)]\\&+\lambda\,\mathbb E_{x\sim X,\,y\sim q_{\bar\theta}(\cdot\mid x)}[\overline D(\theta;x,y)].\end{aligned}

$\lambda$ is unitless and controls the expected contribution of student-generated output data. It does not occur inside the definition of forward or reverse KL.

Source special cases in GKD
λ	Output source	Careful reading
0	fixed / supervised	Not automatically SeqKD: fixed `Y` may be ground truth or teacher-generated, and the local objective is declared separately.
0.5	mixed	Equal expected source weight. Algorithm 1 chooses a source for a batch, so a finite batch history need not contain an exact half.
1	student-generated	On-policy for the fresh behavior policy at collection; prompts still come from `X`.

The paper evaluates these three settings and crosses them with forward KL, reverse KL, and several JSD coefficients. See §4, “GKD Variants.”

Detach the path, not the local student

Agarwal et al. state after Eq. 4 that they do not backpropagate through the student sampling distribution. In the toy, that means the 0.8 → B → xB branch is frozen during the update. It does not mean the student distribution at xB is constant.

HELD FIXED

prompt $x$ ; realized prefixes; teacher probabilities; vocabulary alignment; temperature; categorical outcome

DIFFERENTIATED

student logits $z_\theta(s_t)$ ; then student parameters through each state's network Jacobian

The next chapter performs precisely that fixed-state derivative.

03

Inside one fixed state

Hold the prefix and teacher still. Only then ask what forward and reverse KL do to student logits.

The same distributions, two orders

At one fixed state $s$ , abbreviate the teacher vector as $p_i=p(i\mid s)$ and the student vector as $q_i=q_\theta(i\mid s)$ . Assume a shared vocabulary of $M$ tokens and positive probabilities.

MATH-EQ-FKL-001

Forward KL

D_F(p\|q)=\sum_{i=1}^{M}p_i\log\frac{p_i}{q_i}

Teacher weights the log-ratio. Student appears in the denominator.

MATH-EQ-RKL-001

Reverse KL

D_R(q\|p)=\sum_{i=1}^{M}q_i\log\frac{q_i}{p_i}

Student weights the log-ratio. Teacher appears in the denominator.

KL is asymmetric, so these scalars need not match. Both are dimensionless log-ratios; natural logs express them in nats. Neither formula records how $s$ entered the batch.

Forward KL: the familiar q − p gradient

Let the student use temperature-scaled logits, $q_i=\operatorname{softmax}(z/\tau)_i$ . Its Jacobian is

MATH-EQ-SMX-001 · derived

\frac{\partial q_i}{\partial z_j}=\frac{1}{\tau}q_i(\delta_{ij}-q_j).

The Jacobian has shape $M\times M$ . The factor $1/\tau$ disappears only when $\tau=1$ .

Teacher entropy is constant in the local derivative. Differentiate the cross-entropy term:

MATH-EQ-FKL-GRAD-001 · fixed-state derivative

\begin{aligned}D_F(p\|q)&=\sum_i p_i\log p_i-\sum_i p_i\log q_i,\\\frac{\partial\log q_i}{\partial z_j}&=\frac1\tau(\delta_{ij}-q_j),\\\frac{\partial D_F}{\partial z_j}&=-\frac1\tau\sum_i p_i(\delta_{ij}-q_j)\\&=\frac1\tau(q_j-p_j).\end{aligned}

The last line uses $\sum_i p_i=1$ . At $\tau=1$ , the gradient is simply student minus teacher probability.

Reverse KL: center the student log-ratio

In reverse KL, the student probability appears both as a weight and inside the logarithm. Set $r_i=\log(q_i/p_i)$ and keep the state and teacher fixed:

MATH-EQ-RKL-GRAD-001 · fixed-state derivative

\begin{aligned}\frac{\partial D_R}{\partial z_j}&=\sum_i\frac{\partial q_i}{\partial z_j}(r_i+1)\\&=\frac1\tau\sum_iq_i(\delta_{ij}-q_j)(r_i+1)\\&=\frac1\tau q_j\left[(r_j+1)-\sum_iq_i(r_i+1)\right]\\&=\frac1\tau q_j\left[\log\frac{q_j}{p_j}-D_R(q\|p)\right].\end{aligned}

The baseline being subtracted is the student-weighted mean log-ratio, which is reverse KL itself.

Both local gradient vectors sum to zero. Adding the same constant to every logit leaves softmax unchanged, so a common-shift direction must have zero derivative.

One state under the microscope

toy At x, student $q=[0.75,0.25]$ and teacher $p=[0.5,0.5]$ . At temperature 1:

Local x comparison
Choice	Loss	Student-logit gradient [A,B]
Forward KL	0.143841036225890 nats	[0.25, −0.25]
Reverse KL	0.130812035941137 nats	[0.205989804125271, −0.205989804125271]

The similar loss magnitudes hide different derivatives. More importantly for our thesis, neither row says whether x was reached through fixed data or student generation. That provenance lives outside this table.

The term the canonical toy leaves out

If the current parameters control both response sampling and the local loss, a full derivative of the expectation contains two terms:

MATH-EQ-SG-BOUNDARY-001 · general identity

\nabla_\theta\mathbb E_{y\sim q_\theta}[\overline D_\theta(y)]=\mathbb E\!\left[\nabla_\theta\overline D_\theta(y)+\overline D_\theta(y)\nabla_\theta\log q_\theta(y)\right].

The first term differentiates the local loss on a fixed sample. The second is the score-function contribution from changing the sampling measure itself.

The canonical GKD-style estimator in this article retains the first term and stops the second. That is why the next numerical receipt must freeze the xB branch while perturbing its local logits.

do not generalize MiniLLM begins from a sequence-level reverse-KL target and derives policy-gradient terms. Its equations are not a hidden version of this local derivative; chapter 5 keeps them separate.

04

The derivation sheet

Every symbol, contribution, gradient, and rounding decision for OPD-TOY-001—open on the page.

Pin the symbols before the arithmetic

MATH-REC-TOY-001 · independent receipt

Epistemic status: invented deterministic teaching fixture. Natural logs; nats; vocabulary [A,B]; temperature 1; two evaluated positions; no explicit EOS.

Symbol table
Symbol	Meaning	Shape / unit	Local derivative
$x$	fixed prompt	token sequence	held fixed
$s_t=(x,y_{<t})$	visited prefix occurrence	one context	held fixed
$p(\cdot\mid s)$	teacher distribution	$\Delta^{1}$ ; unitless	held fixed
$z_\theta(s)$	student logits	$\mathbb R^2$	differentiated
$q_\theta(\cdot\mid s)$	student distribution	$\Delta^{1}$ ; unitless	differentiated
$D_F,D_R$	local KL scalar	nats	derived
$\lambda$	student-output source fraction	[0,1]; unitless	chosen

Probability vectors are positive and normalized.
The categorical u = 0.8 branch is detached.
xA is fixed by provenance in the supplied comparison occurrence, not by its text.
Conditioned-path mixtures are not the full response expectation.
Teacher probabilities are supervision, not a guarantee of truth or process causality.

Reverse KL, line by line

01 · state x

\begin{aligned}D_R(x)&=\frac34\log\frac{3/4}{1/2}+\frac14\log\frac{1/4}{1/2}\\&=\frac34\log\frac32+\frac14\log\frac12\\&=\frac34\log3-\log2\\&=0.304098831081123286-0.173286795139986327\\&=0.130812035941136959\ \text{nats}.\end{aligned}

The A contribution is positive and the B contribution negative; KL non-negativity applies to their sum, not each term.

02 · state xB

\begin{aligned}D_R(xB)&=\frac35\log\frac{3/5}{9/10}+\frac25\log\frac{2/5}{1/10}\\&=\frac35\log\frac23+\frac25\log4\\&=\frac75\log2-\frac35\log3\\&=-0.243279064864898629+0.554517744447956248\\&=0.311238679583057618\ \text{nats}.\end{aligned}

The teacher assigns only 0.1 to B, so the student's 0.4 weight produces the larger positive term.

03 · unweighted route mean

\begin{aligned}\widehat L_R&=\tfrac12[D_R(x)+D_R(xB)]\\&=\tfrac3{40}\log3+\tfrac15\log2\\&=0.221025357762097289\ \text{nats}\\&\longrightarrow\boxed{0.221025\ \text{nats at six decimals}}.\end{aligned}

Round the full-precision mean once. Averaging the displayed six-decimal components produces the tie 0.2210255; decimal round-half-up would show 0.221026 instead.

Forward direction, fixed route, and the expectation you did not sample

Full-precision local values · toy
Prefix	Reverse KL q ‖ p	Forward KL p ‖ q	Occurrence used by
x	0.130812035941137	0.143841036225890	both routes
xB	0.311238679583058	0.226289161185359	realized u = 0.8 student route
xA	0.582685302043240	0.534110808710307	supplied fixed route

MATH-EQ-FORWARD-MEAN-001

\widehat L_F(x\!\to\!xB)=\frac15\log3-\frac1{20}\log2=0.185065098705624673\ \text{nats}.

Same realized prefixes, different local direction. The route remains student-generated.

The supplied fixed route [x,xA] has mean reverse KL 0.356748668992188 and mean forward KL 0.338975922468099 nats. Mixing that one route with the realized student route gives a useful six-cell contrast—but not the exact expectation over all student responses.

MATH-EQ-OUTER-EXAMPLE-001 · variant-specific contrast

\mathbb E_{y_1\sim q(\cdot\mid x)}[L_R\mid x]=0.75(0.356748668992188)+0.25(0.221025357762097)=0.322817841184666.

This hypothetical branch expectation uses xA when the student itself draws A. It is not the contract's fixed u=0.8 sample receipt and still does not model EOS or later draws.

Analytic gradients meet finite differences

Local student-logit gradients at temperature 1
State	Forward $\nabla_zD_F$	Reverse $\nabla_zD_R$
x	[0.25, −0.25]	[0.205989804125271, −0.205989804125271]
xB	[−0.3, 0.3]	[−0.430022272614733, 0.430022272614733]
xA	[−0.5, 0.5]	[−0.469054366516490, 0.469054366516490]

Each vector has length two, units “nats per logit,” and sum zero. For the canonical two-state mean, keep state blocks separate:

MATH-EQ-MEAN-BLOCKS-001

\nabla_{(z_x,z_{xB})}\widehat L_R=[0.102994902062635,-0.102994902062635,-0.215011136307367,0.215011136307367].

Do not average gradients componentwise unless another toy deliberately shares one logit vector. A real network passes each block through its state-dependent Jacobian to common parameters.

executed receipt

MATH-LAB-001 perturbs frozen-state logits with central differences, step $h=10^{-5}$ , and requires absolute error at most $2\times10^{-10}$ with relative tolerance exactly zero. The observed maximum is loaded from the generated receipt and audited separately.

Inspect the JSON receipt (local copy) · Read the independent Python source (local copy) · Read the Markdown sheet (local copy)

Change one coordinate without smuggling in the other

Now the one approved interaction. Predict before touching it: if the trajectory source stays at $\lambda=1$ and only KL direction changes, should the on-policy label move?

MATH-EXP-001 — Static source × divergence matrix

The live edition lets a reader change λ and KL direction. This complete reading edition instead exposes every state at once.

MATH-FIG-002 · static fallback · toy conditioned-path values
Outer source coefficient	Local D_KL(student ‖ teacher)	Local D_KL(teacher ‖ student)	Provenance label
λ = 0Fixed route only	0.356748668992188reverse KL · nats	0.338975922468099forward KL · nats	fixed
λ = 0.5Equal conditioned-path coefficients	0.288887013377143reverse KL · nats	0.262020510586862forward KL · nats	mixed
λ = 1Realized student route only	0.221025357762097reverse KL · nats	0.185065098705625forward KL · nats	student / on-policy at collection

Invariant 1: moving across columns changes the divergence, not provenance. Invariant 2: moving down rows changes source coefficients, not KL direction. Values mix the supplied fixed route with the realized u = 0.8 student route; they are not the full student-trajectory expectation.

The table is not redundant decoration. It is the complete no-script and print path, and it makes every boundary state inspectable at once.

05

Methods that share a surface

Student generations and teacher feedback appear in several recipes. Their targets and gradients still differ.

Compare five fields, not one label

A method name is a poor substitute for an estimator. Ask about trajectory source, feedback granularity, objective, gradient treatment, and practical freshness.

Boundary map · primary-source status shown in final column
Method	Trajectory / prefix source	Teacher feedback	Training target	Gradient through source law?	Status
SeqKD	fixed teacher beam output	hard teacher-generated sequence	student NLL on that fixed sequence	no current-student source	EMNLP 2016
ImitKD	sequence-level mixture of fixed and student-generated examples; schedule may anneal	teacher optimal action or full token distribution at visited prefixes	oracle NLL or cross-entropy	sample generation is used as data; practical pools may be a few iterations old	EMNLP 2020
GKD / canonical toy	fixed, mixed, or current-student outputs via λ	full next-token distribution at each selected prefix	declared local divergence, averaged over positions	no; sampled response is detached in canonical estimator	ICLR 2024
MiniLLM	theory: student response distribution; practical behavior is teacher-mixed	teacher/student log ratios plus full-vocabulary single-step term	sequence-distribution reverse KL with policy-gradient decomposition	yes, through score-function terms—not pathwise through a token	ICLR 2024
Scalar-reward RL	often student trajectories, depending on algorithm	external scalar reward	expected reward / surrogate	algorithm-dependent score-function or surrogate	separate term in GKD Eq. 5
Sampled-token engineering recipe	student trajectories	teacher and student log-probability of sampled token	per-token reverse log-ratio as advantage in importance-sampling loss	RL-style loss implementation	Thinking Machines 2025 report
Top-k soft target	separate choice; docs example uses teacher completion	teacher probabilities renormalized over selected top-k tokens	soft-target cross-entropy over k entries	depends on surrounding sampler	current Tinker docs

feedback granularity Full vocabulary, one sampled token, and renormalized top-k probabilities are not equivalent observations. “Token-level” is too coarse a label to establish the estimator.

Fixed teacher sequence is not teacher distribution

SeqKD. Kim and Rush generate a fixed dataset with teacher beam search and train the student on a selected teacher sequence. Their EMNLP 2016 paper, Fig. 1 and the sequence-level method on pp.1318/1322, make the hard-sequence target explicit. This is not full-distribution matching at a student-generated prefix.

ImitKD. Lin et al. mix fixed sequences with student-generated sequences and query an oracle at the resulting prefixes. Their EMNLP 2020 paper, §3.1–3.4 and Algorithm 1, permits either the teacher's optimal action or full-distribution cross-entropy. Its $\beta_i$ schedule and optional sample pooling are not simply a constant GKD $\lambda=0.5$ .

MATH-NOTE-001 — MiniLLM starts from a different objective

bounded advanced note

Keep the equations in separate boxes

MiniLLM's peer-reviewed conference target is reverse KL between complete response distributions:

MINILLM EQ. 1 · sequence distribution

\mathcal L_{\mathrm{MiniLLM}}(\theta)=D_{KL}\!\left(q_\theta(y\mid x)\,\middle\|\,p(y\mid x)\right).

Here the student response distribution is part of the mathematical target, not merely a detached collector.

Its policy-gradient derivation contains a future cumulative log-ratio $R_t$ :

MINILLM EQ. 2 · score-function form

\nabla\mathcal L=-\mathbb E_{x,\,y\sim q_\theta}\sum_{t=1}^{T}(R_t-1)\nabla\log q_\theta(y_t\mid y_{<t},x).

This is a likelihood-ratio gradient. It does not make the categorical token pathwise differentiable.

The practical method then decomposes a full-vocabulary single-step term from a long-horizon score-function term and adds teacher-mixed sampling, approximate importance weighting, length normalization, clipping, and an auxiliary language-modeling loss. See the ICLR 2024 paper, §2.1–2.3 and Appendix A.2.

Boundary: the canonical GKD toy's fixed-state reverse-KL gradient is not a derivation of MiniLLM. Both may visit student-reachable prefixes; their objectives and retained gradient terms differ.

Sampled token, top-k, and full vocabulary

A 2025 Thinking Machines engineering report samples student trajectories, asks teacher and student for the sampled token's log-probability, and uses the negative reverse log-ratio as a per-token advantage in an importance-sampling RL loss. The report explicitly says its experiments did not use top-k logit distillation.

Current Tinker cross-entropy documentation separately exposes a top-k soft-target form, with teacher probabilities renormalized over selected target tokens. Truncating and renormalizing changes the compared distribution; one sampled token changes the available information again.

These are useful engineering choices, not evidence that one feedback granularity is universally better. The monograph records them to stop a notation shortcut from erasing implementation semantics.

What a method label still leaves unresolved

Vocabulary / tokenizer: The full-distribution KL assumes aligned entries. Cross-tokenizer mapping is outside this toy.
EOS and length: The paper writes a position mean; the toy fixes two evaluated positions. EOS inclusion and variable-length batching must be declared in an implementation.
Sampling temperature: GKD reports student temperature 1 during training. This fixture directly supplies probabilities at temperature 1.
Behavior freshness: Fresh per-step samples support the on-policy label. Replay or asynchronous collection may change it relative to current parameters.
Teacher semantics: A probability vector supplies dense supervision. It does not guarantee correctness, causal credit, or process supervision.
Teacher strength: A larger or stronger teacher is not guaranteed to help every student/configuration. No such claim is made here.

06

Carry the factorization with you

Two changed cases, then a five-question inspection rule for the next paper or implementation.

OPD-XFER-A — Reverse loss, fixed state source

COMMON TRANSFER CHECK · not a mastery score

A learner uses reverse KL on a fixed teacher-generated xA prefix. Is the divergence reverse KL, and is the state sampling on-policy? Explain both axes separately.

Inspect the reasoning

Local axis: yes, the formula is reverse KL because the student distribution is the first argument and weights the log-ratio.

Source axis: no for this supplied occurrence. The xA prefix came from a fixed teacher-generated trajectory, not the current student. The same textual prefix could be reached on-policy in another collection event; provenance belongs to the event.

OPD-XFER-B — Student state source, forward loss

COMMON TRANSFER CHECK · not a mastery score

A learner samples B from the student, queries the teacher at xB, and computes forward KL. Is this on-policy distillation? Explain why divergence direction does not change the state source.

Inspect the reasoning

Source axis: yes, provided the response was sampled from the current behavior policy at collection. xB is a student-visited prefix.

Local axis: forward KL compares teacher to student at that fixed prefix. Changing argument order changes the loss and gradient—not the already established provenance.

Five questions for any claimed OPD recipe

What samples the response?Fixed data, teacher, current student, a mixture, or a stale behavior policy?
Which occurrence is evaluated?Record prompt, prefix, and how that occurrence entered the batch.
What feedback is visible?Full vocabulary, top-k, one sampled token, scalar reward, or something else?
What objective is formed?Write its arguments and reduction instead of relying on “KL” or “distillation.”
Which dependence reaches the gradient?Detached collection, score-function term, importance weighting, or another estimator?

If those five answers are explicit, “on-policy” becomes one coordinate again rather than a bag of assumptions.

The measure beneath the loss

The outer measure chooses the states. The inner divergence compares distributions at a state. The gradient estimator decides which parameter dependence survives. Keep those three lines separate and the surrounding terminology becomes much easier to audit.

unproven hypothesis This mathematical form was designed to help a mathematically inclined reader resist one specific conflation. No learner study was run, and completing the transfer prompts does not demonstrate mastery or establish that this variant teaches better than another.

Open notation, evidence, and limitations → Complete script-free reading edition

Notation and evidence

Epistemic key

source-backed externally checkable statement with a primary locator

derived follows from displayed premises and arithmetic

toy invented input used only to expose a mechanism

boundary assumption or implementation choice that limits the result

unresolved not established by inspected evidence

Glossary

Policy: A conditional distribution over the next token in this article; it does not by itself imply scalar reward optimization.
Trajectory: A generated response sequence. Its prefixes are the states at which local token distributions can be compared.
On-policy: For canonical GKD here, the response was sampled from the current student behavior policy at collection time. It describes provenance, not KL direction.
Behavior policy: The policy that generated collected data. $q_{\bar\theta}$ exposes that it is held fixed during the canonical local update.
Forward KL: $D_{KL}(p_{teacher}\|q_{student})$ at a fixed state in this notation.
Reverse KL: $D_{KL}(q_{student}\|p_{teacher})$ at a fixed state in this notation.
Logit: An unnormalized student score. Softmax maps the length- $M$ logit vector to a probability vector.
Nat: The unit of information produced by a natural-log ratio. KL is dimensionless, but “nats” records the logarithm base.
Stop-gradient: An estimator choice that holds one dependency fixed during differentiation. Here it applies to the sampled trajectory, not the local student logits.
Feedback granularity: Which teacher information is exposed: a full vocabulary distribution, top-k truncation, one sampled token's score, a hard action, or a scalar reward.

Sources

SRC-GKD-ICLR-2024 · Peer-reviewed · ICLR 2024 official recordOn-Policy Distillation of Language Models: Learning from Self-Generated Mistakes
Canonical publication-status anchor. Supports the GKD framing, not this toy or a universal superiority claim.
SRC-GKD-ARXIV-V3 · Author manuscript v3 · 2024-01-17 · accepted at ICLR 2024Agarwal et al. author manuscript, arXiv:2306.13649v3
Navigable source for Eqs. 2–5, Algorithm 1, lambda cases, divergence choices, temperature, and the explicit stop-gradient statement.
SRC-SEQKD-EMNLP-2016 · Peer-reviewed · EMNLP 2016Sequence-Level Knowledge Distillation
Primary source for fixed teacher beam outputs used as hard sequence targets.
SRC-IMITKD-EMNLP-2020 · Peer-reviewed · EMNLP 2020Autoregressive Knowledge Distillation through Imitation Learning
Primary source for ImitKD sequence-level mixtures, oracle supervision, schedule, and practical sample pooling.
SRC-MINILLM-ICLR-2024 · Peer-reviewed · ICLR 2024 official recordMiniLLM: Knowledge Distillation of Large Language Models
Conference anchor for sequence-level reverse KL and its policy-gradient decomposition. The later arXiv v6 title changed in 2026; equations here remain pinned to the conference version.
SRC-TML-OPD-2025 · Author engineering report · Thinking Machines Lab · 2025-10-27On-Policy Distillation
Primary author report for the sampled-token reverse-log-ratio recipe and explicit statement that its experiments did not use top-k logit distillation. Not peer-reviewed.
SRC-TINKER-TOPK · Official engineering documentation · version-sensitiveTinker Cross-Entropy documentation
Primary documentation for renormalized teacher top-k soft targets. Not canonical GKD evidence.

Limits

Aligned vocabulary, two-position reduction, temperature 1, no explicit EOS, no stale replay model, no teacher-correctness guarantee, no real-model benchmark, and no learner study.