One mistake,
frame by frame.
The student sampled B. Keep that branch fixed while sample, prefix, teacher query, divergence, and update arrive one beat at a time.
Classify trajectory provenance separately from divergence direction, then recompute one token-distribution KL.
The spatial-continuity hypothesis is unproven. This build contains no learner study and makes no mastery claim.
The student draws B. It may not be the branch we wish it had taken. But once the student visits xB, the teacher must respond at xB.
That one decision clears up most of the phrase on-policy distillation. “On-policy” identifies the source of generated trajectories or prefix states. It does not name the divergence. Reverse KL, forward KL, JSD, or another discrepancy is a separate design choice.
sourced OPD-CL-001, OPD-CL-003 · Agarwal et al.’s canonical GKD formulation exposes student-data fraction and divergence as separate choices. See ICLR 2024 author manuscript v3, §3.1 and the official proceedings record.
You should leave able to name each boundary, recompute the toy loss, and classify two changed cases. Watching or answering is a self-check, not evidence of mastery.
Notation, just in time
x- a fixed input prompt
pS(· | s)- student next-token distribution at prefix
s pT(· | s)- teacher next-token distribution at the same prefix
- policy / prefix
- distribution used to act / tokens already generated
“On-policy” names a source.
The fixed prompt is not the crucial provenance question. Canonical GKD can draw x from a fixed set X and still sample the completion y | x freshly from the student.
sourced OPD-CL-006 · Agarwal et al., §3.1 Eq. 4 and Algorithm 1.
| Axis | Question | Examples |
|---|---|---|
| Trajectory provenance | Which distribution generated the completion or prefix? | current student; fixed teacher data; mixture |
| Training objective | How is the student changed at those states? | forward KL; reverse KL; JSD; NLL; policy-gradient estimator |
Putting reverse KL on a teacher-generated fixed prefix does not make the state on-policy. Putting forward KL on a freshly student-generated prefix does not make it off-policy.
OPD-TOY-001 · invented teaching fixture
Shared vocabulary (A, B). At the first prefix: pS(·|x)=(0.75,0.25) and pT(·|x)=(0.50,0.50).
These values are not real-model logits, a benchmark, or evidence that any method improves quality.
Preserve the sampled branch.
Choose a prefix; the feedback will preserve the actual trajectory.
Script-free prediction and answer
After the student samples B, query the teacher at xB. That is the visited prefix; xA remains off this trajectory.
FILM-CL-001 · The cumulative student intervals are [0,.75) for A and [.75,1) for B. The draw therefore selects B unambiguously, producing the visited prefix xB. Do not repair the trajectory before evaluating it.
Film FILM-VIDEO-001
Silent film with burned-in narration. A complete WebVTT track is available from the native captions menu but starts off to avoid duplicate text; when enabled, its cues sit at the top, away from the burned-in band. No autoplay.
Reduced-motion mode hides the moving film. The complete six-frame semantic version is expanded below.
Six inspectable beats FILM-EXP-001
The film is not the sole lesson. Every frame below includes the same branch, values, equation, narration, and relationship in selectable text.
“On-policy” names a source.
Static keyframe. Prompt x remains fixed; the student is the source of the completion.
A fixed prompt does not make a completion off-policy. Policy means the next-token distribution used to generate the trajectory; divergence is chosen later.
Infer from the layout: provenance and objective occupy different axes.
The draw lands on B.
Static keyframe. The B branch appears in the lower lane and will stay there.
Sampling is data collection. In this canonical GKD-style fixture, the discrete draw is detached.
The draw chooses the path; it is not yet the teacher query or loss.
The visited prefix is xB.
Static keyframe. xA remains visible only as an excluded comparison.
B makes xB the visited prefix. xA is a comparison branch, not part of this trajectory and not part of its canonical loss.
Do not swap in the branch the teacher might have preferred.
Ask the teacher at xB.
Static keyframe. The teacher’s dashed query line ends at the same xB prefix.
The teacher supplies a distribution at the same prefix. It does not replace the trajectory.
Same-prefix evaluation is the relationship to preserve.
Now measure disagreement.
Static keyframe. This is full-vocabulary feedback over the aligned toy tokens.
Reverse KL is one objective choice. It is not what makes the state on-policy.
Reduce unrounded state totals; xA remains excluded.
Only then: update the student.
Static keyframe. The stop marker and update arrow are separate visual edges.
No gradient passes through the discrete B draw in this canonical fixture. The local student distribution in the divergence is the differentiable part.
Transfer question OPD-XFER-B: student B + teacher at xB + forward KL. Is the state source still on-policy?
Yes. Forward KL changes the objective, not who generated xB.
sourced OPD-CL-002, OPD-CL-004, OPD-CL-005 · Canonical GKD compares teacher and student at intermediate prefixes and explicitly does not backpropagate through the sampling process. This stop-gradient boundary is not a universal definition of every method called OPD.
Reverse KL is one objective.
At a visited prefix s, this film chooses the discrete reverse KL:
| Token | Contribution | Six decimals |
|---|---|---|
| A | 0.60 ln(0.60/0.90) | −0.243279 |
| B | 0.40 ln(0.40/0.10) | +0.554518 |
| xB total | 0.311239 nats | |
An individual contribution can be negative; the complete KL is non-negative. At the first visited position x, the total is 0.130812 nats.
FILM-CL-002 · rounding trap
Averaging the displayed 0.130812 and 0.311239 gives 0.221026. The contract’s 0.221025 is correct because the full-precision totals are averaged before display rounding.
What this toy assumes
- Aligned student/teacher tokenization and the same two-token vocabulary.
- Full-vocabulary feedback. Sampled-token and top-k interfaces are not equivalent per update.
- Given distributions and
u=.8; no temperature transform is applied inside the fixture. - Exactly two visited positions, with no EOS, padding, mask, or variable-length weighting.
- A fresh behavior policy, with no distributed policy lag or importance correction.
toy boundary FILM-CL-007 · Canonical GKD reports student sampling temperature γ=1 during its experiments. This fixture also assumes aligned tokenization, two visited positions, no EOS, fresh behavior-policy sampling, and full-vocabulary feedback. None is treated here as a universal production default.
Runnable companion `FILM-LAB-001`. From the variant folder: python -B lab/opd_toy.py, then python -B -m unittest -v lab.test_opd_toy. The standard-library lab checks full-precision terms, branch selection, boundaries, xA exclusion, and forward/reverse anti-conflation.
Stop at the update boundary.
Only after the local discrepancies have been reduced do we write the symbolic student update:
The sampled B is detached data in this canonical GKD-style estimator. MiniLLM is a different construction: it starts from sequence-level reverse KL and derives a policy-gradient optimizer with additional single-step, long-term, mixed-sampling, importance-weight, and length choices.
sourced distinction OPD-CL-007 · GKD: Agarwal et al., §3–3.1. MiniLLM: Gu et al., current author manuscript v6, §2.1–2.3; ICLR 2024 record.
| Method | State / sequence source | Teacher feedback | Objective / update | Status |
|---|---|---|---|---|
| Canonical on-policy GKD (`λ=1`) | current student completion from fixed prompts | full next-token distribution at visited prefixes | chosen token-distribution divergence; sampling detached | ICLR 2024 |
FILM-CL-003Fixed teacher SeqKD | teacher-decoded static sequences | hard teacher sequence | supervised NLL / cross-entropy | EMNLP 2016 |
FILM-CL-004ImitKD | configurable fixed/student mixture; pool may lag | teacher argmax or distribution at those prefixes | imitation-learning update | EMNLP 2020 |
| MiniLLM | student / teacher-mixed sequence sampling | sequence-level teacher likelihood and derived terms | reverse-KL policy-gradient formulation | ICLR 2024 |
| Scalar-reward RL | usually student rollouts | scalar reward / return | policy-gradient family | different signal; combinable |
FILM-CL-005Thinking Machines recipe | student trajectories | teacher/student log-probability for realized tokens | negative log-ratio advantage via RL importance sampling | 2025 engineering report |
Feedback granularity is another axis. Full-vocabulary feedback exposes every aligned token probability. Sampled-token feedback observes the realized token. Top-k feedback retains only a subset. A 2026 paper formalizes these variants; it is recent preprint evidence, not peer-reviewed proof that one universally wins.
recent preprint FILM-CL-006 · Li et al., arXiv:2604.13016v2, §2.2. The 2025 engineering report explicitly says its experiments did not use top-k/logit distillation.
FILM-CL-008 · local evidence boundary
Three claims this signal does not earn.
- Teacher confidence is not proof of correctness.
- Tokenwise conditional feedback is not automatically causal attribution or process supervision.
- A stronger teacher does not guarantee a better student, and OPD does not universally beat SFT, offline distillation, or RL.
Separate the axes yourself.
OPD-XFER-A
A learner uses reverse KL on a fixed teacher-generated xA prefix. Is the divergence reverse KL, and is the state sampling on-policy? Explain both axes separately.
Model reasoning
The objective is reverse KL. The response prefix is fixed and teacher-generated rather than sampled by the current student, so its state provenance is not on-policy.
OPD-XFER-B
A learner samples B from the student, queries the teacher at xB, and computes forward KL. Is this on-policy distillation? Explain why divergence direction does not change the state source.
Model reasoning
Yes: the visited prefix is on-policy because the student generated B. Forward KL is the independently chosen objective. No gradient through the discrete draw is implied.
These are transfer-oriented self-checks, not a validated assessment and not evidence of durable learning.
The compact mental model. The student created xB. The teacher was evaluated at xB. Reverse KL was chosen separately. The draw was detached in this canonical fixture. The local loss then defined the parameter-gradient boundary.
Primary sources and status.
FILM-CL-009· Agarwal et al., “On-Policy Distillation of Language Models,” ICLR 2024Canonical peer-reviewed record. Current author manuscript: arXiv v3, revised 2024-01-17. Earlier manuscript versions used “Generalized Knowledge Distillation” in the title.
- Kim & Rush, “Sequence-Level Knowledge Distillation,” EMNLP 2016
Primary peer-reviewed SeqKD source.
- Lin et al., “Autoregressive Knowledge Distillation through Imitation Learning,” EMNLP 2020
Primary peer-reviewed ImitKD source.
- Gu et al., “MiniLLM,” ICLR 2024
Peer-reviewed record; current author manuscript v6 is newer and retitled.
- Kevin Lu, “On-Policy Distillation,” Thinking Machines Lab, 2025
Engineering report, not peer reviewed; page checked 2026-09-05.
- Li et al., “Rethinking On-Policy Distillation of Large Language Models,” arXiv:2604.13016v2
Recent 2026 preprint; used only for feedback-granularity taxonomy.