Experimental learning blog · Variant 03
The Prefix
Observatory
A causal laboratory for keeping trajectory source, KL direction, and feedback granularity in their own lanes.
Experiment status — unproven. This variant tests a design hypothesis: a manipulable 2×2 may make two independent OPD decisions easier to perceive. No learner study was run, so the build does not establish better comprehension, retention, or transfer. OBS-CL-011
“On-policy distillation” is easy to compress into the wrong shortcut: student samples, therefore reverse KL. That pairing appears in important systems, but the implication does not hold.
The canonical Generalized Knowledge Distillation (GKD) formulation gives us two switches: who produced the completion trajectory, and which divergence compares teacher with student at the visited prefixes. Its student-data fraction and divergence are independent inputs. “On-policy” identifies trajectory provenance, not the order of the arguments inside KL. GKD §3.1 and §4 OPD-CL-003
A fixed prompt set does not make the completion off-policy. A prompt x may come from a fixed dataset while the continuation y is drawn from the current student. The prompt is exogenous; the generated continuation determines later state occupancy. OPD-CL-006
01 · Separate the causes
The two questions
At an autoregressive prefix, teacher and student each return a normalized distribution over the next token. Sampling a token moves the generator to a new prefix. Ask these questions in order, but never collapse them.
Where did the state come from?
A fixed demonstration always takes branch A. The active student uses a replayable uniform draw u=0.8; at temperature one it takes B. Provenance belongs to the history that produced the string.
Key consequence If a colder student also takes A, its xA is still student-generated even though the fixed path spells the same text.
How are the distributions compared there?
Teacher and student are evaluated at the same selected prefix. A local loss then compares their next-token distributions. The teacher query does not retroactively create the prefix; it supplies supervision after the path selected it.
GKD Eq. 2 and Eq. 4 OPD-CL-001 OPD-CL-002
02 · Read the order literally
Just enough KL
For positive, normalized categorical distributions P and Q, the first distribution supplies both the weight and the top of the log ratio.
Following the GKD convention, this guide calls forward KL DKL(T ∥ S) and reverse KL DKL(S ∥ T). A single vocabulary contribution may be negative; only the complete supported KL sum is guaranteed non-negative. GKD §2 OBS-CL-001
At the canonical default—student path x → xB, reverse KL—the local values are 0.130812035941137 and 0.311238679583058. Their full-precision mean displays as 0.221025 nats. The implementation rounds only after reduction. OBS-CL-003
Toy Every probability and draw below is invented. The fixture is a hand-checkable mechanism, not a trained model, benchmark, or quadrant leaderboard.
03 · Predict before reveal
Which quadrant owns this case?
Static answer and canonical matrix
Without JavaScript, the intended cell is student-generated / forward: the student produced the completion, while T ∥ S names forward KL.
| Completion source | Forward T ∥ S | Reverse S ∥ T |
|---|---|---|
Fixed teacher path x → xA | 0.338976 nats | 0.356749 nats |
Student path x → xB | 0.185065 nats | 0.221025 nats |
These numerical differences change both provenance and visited prefix content; they are not a causal estimate of an “on-policy benefit.”
04 · Manipulate the mechanism
Inside the observatory
The 2×2 never rotates: rows are completion source, columns are divergence direction. Temperature changes every displayed student distribution and the active student behavior policy. Feedback aperture remains a separate third dimension.
OBS-EXPLAINER-001 · deterministic local model
Prefix and divergence instrument
Applied consistently to the student behavior and displayed loss policy; no hidden stale-policy correction. OBS-CL-002
- Source
- STUDENT
- Direction
- REVERSE
- Second prefix
- xB
OBS-FIG-001
Trajectory provenance
Same text can keep a different causal history.
Fixed lane: x → A → xA (teacher demonstration).
Student lane: x → B → xB (u=0.8, τ=1.00).
OBS-FIG-002
Two switches, four cells
Select a cell; geometry stays fixed.
| Completion source | Forward · T ∥ S | Reverse · S ∥ T |
|---|---|---|
| Fixed demonstration | ||
| Student-generated |
The normalized tables beneath the glass
| Prefix | T(A) | T(B) | T sum | S(A) | S(B) | S sum |
|---|
OBS-FIG-002B
Every signed contribution, then the exact sum
D_KL(S || T)
x: 0.130812; xB: 0.311239
(0.130812 + 0.311239) / 2 ≈ 0.221025 nats
Path mean nats
Student-generated completion visits xB. Teacher and student are evaluated at x and xB; reverse KL is reduced across both positions. The student categorical draw selects the path and is detached in this canonical GKD-style toy.
OBS-FIG-004 · separate third dimension
Feedback aperture
full vocabulary mode; reverse KL. Source and trajectory controls are unchanged.
Static canonical observation
Student-generated · reverse KL · τ=1 · u=0.8. The active student draws B, so the visited path is x → xB. The fixed demonstration instead follows x → xA.
| Prefix | T(A) | T(B) | T sum | S(A) | S(B) | S sum | Reverse KL |
|---|---|---|---|---|---|---|---|
| x | .500000 | .500000 | 1.000000 | .750000 | .250000 | 1.000000 | .130812 |
| xB | .900000 | .100000 | 1.000000 | .600000 | .400000 | 1.000000 | .311239 |
Signed reverse-KL terms are +0.304099, −0.173287 at x, and −0.243279, +0.554518 at xB. Complete local sums are non-negative; their full-precision mean is 0.221025 nats.
τ=0.5
The active student has Sx=(.900,.100); u=.8 selects A, so its path spells xA but remains student-generated.
τ=1
Sx=(.750,.250); u=.8 selects B. Canonical student path: xB.
τ=2
Sx≈(.634,.366); u=.8 selects B. The flatter distribution changes values, not the meaning of provenance.
05 · Locate the gradient · OBS-FIG-003
The update begins after the draw
In Agarwal et al.’s GKD estimator, the discrete student trajectory selects visited states but is not differentiated through. Gradients flow through the local student distribution inside the divergence at those states. GKD §3.1 after Eq. 4 OPD-CL-004 OPD-CL-005
- Sample / select
Student or fixed demonstration supplies the completion.
- Record prefix
The chosen token determines xB.
- Evaluate
Teacher and student return distributions at the same prefix.
- Compare
reverse KL, D_KL(S || T)
- Reduce
Two-state mean: 0.221025 nats.
- Update θ
Differentiate the local student distributions.
Stop-gradient boundary: no gradient through the categorical draw in this canonical GKD-style toy. This is method-specific, not a definition of every system now called OPD. Teacher confidence does not by itself establish correctness, and matching its token distribution is not causal process supervision. OBS-CL-010
MiniLLM is not merely “the reverse cell.” It targets sequence-level reverse KL and derives a policy-gradient decomposition with an exact local term and sampled long-horizon term. MiniLLM Eq. 1–3 OPD-CL-007
06 · Open a third dimension
Feedback aperture is not another matrix axis
Full-vocabulary, sampled-token, and top-k feedback transmit different information. The separate fixture OBS-FEEDBACK-001 uses four tokens so truncation actually hides a tail.
| Policy | A | B | C | D | Sum |
|---|---|---|---|---|---|
| Student S | .55 | .25 | .15 | .05 | 1.00 |
| Teacher T1 | .40 | .20 | .30 | .10 | 1.00 |
| Teacher T2 | .40 | .10 | .30 | .20 | 1.00 |
Full vocabulary. Exact reverse KL is 0.092306 for T1 and 0.230935 for T2; forward is 0.105249 versus 0.266192.
Sampled token. For P ∥ Q, one draw v ~ P yields ln[P(v)/Q(v)]. Its expectation is KL, but one observation may be negative and is not the weighted exact term. OBS-CL-004
Top-2. Both teachers expose A=.40, C=.30, and omitted mass .30. Their payloads are identical while their full KL differs, so that payload cannot identify full-vocabulary KL. Renormalizing over {A,C} yields a separately declared conditional objective: 0.101682 reverse or 0.115090 forward for T1. OBS-CL-013
Different aperture, different information. State source, divergence, payload granularity, and gradient estimator each need a name.
Current Tinker documentation specifies one renormalized top-k soft-target rule; it is an implementation contract, not a universal definition. A separate Thinking Machines engineering report uses sampled-token teacher/student log probabilities and explicitly says its experiments did not use top-k logit distillation. OBS-CL-005 OBS-CL-006
07 · Keep nearby methods distinct · OBS-FIG-005
Same neighborhood, different addresses
| Method | Trajectory / prefix source | Supervision | Update distinction | Source status |
|---|---|---|---|---|
| Canonical GKD on-policy endpoint | Student completion from fixed prompts | Full teacher and student next-token distributions at same prefixes | Chosen tokenwise divergence; trajectory draw detached in the paper’s estimator | Peer-reviewed ICLR 2024 |
| SeqKD mode approximation | Fixed teacher beam-search sequences | Teacher-generated target tokens | Student likelihood on a fixed teacher dataset | Peer-reviewed EMNLP 2016 |
| ImitKD | Mixed initial/static and student-explored states | Teacher correction, token or distribution variants | Imitation-learning method; may pool older generations | Peer-reviewed EMNLP 2020; “mixed” is GKD’s cross-paper interpretation |
| MiniLLM | Teacher–student sampling mixture in its practical estimator | Teacher/student log-density ratios plus exact local term | Sequence-level reverse KL with a policy-gradient decomposition | Peer-reviewed ICLR 2024; current author manuscript v6 |
| Scalar-reward RL | Often student trajectories | Scalar task/environment reward | No teacher token distribution required | General category; GKD Eq. 5 keeps reward and distillation distinct |
| Thinking Machines 2025 implementation | Student trajectories | Sampled-token teacher and student log probabilities | Reverse-KL-style advantages in an RL importance-sampling update | First-party engineering report, not peer-reviewed |
Read the primary method accounts: SeqKD, EMNLP 2016 OBS-CL-007; ImitKD, EMNLP 2020 OBS-CL-008; and the MiniLLM ICLR 2024 record.
GKD can combine a separate scalar-reward objective with distillation; on-policy state collection does not make every method ordinary scalar-reward RL. OBS-CL-009
08 · Explain both axes
Transfer checks
Classify the source and divergence before opening each analysis. These are prompts for self-explanation, not a mastery score.
OPD-XFER-A — reverse KL on a fixed teacher-generated xA prefix
Answer: the divergence is reverse KL because the student distribution is the numerator. The state is not student on-policy: the teacher-generated path was fixed. Changing the objective label cannot rewrite that provenance.
OPD-XFER-B — student samples B, teacher is queried at xB, then forward KL
Answer: under the canonical GKD provenance meaning, this is on-policy state sampling because the student produced the completion and therefore visited xB. Forward KL says the teacher distribution weights ln(T/S); it does not change the state source.
09 · Carry the questions forward
What survives outside the toy
- Which policy or dataset produced the completion trajectories?
- Are teacher and student evaluated at exactly the same prefixes?
- Which divergence or sequence objective is optimized?
- Is feedback full-vocabulary, sampled-token, or top-k—and under which estimator?
- Does gradient flow through a likelihood-ratio estimator or only through local distributions?
- How are EOS, padding, length, temperature, and tokenizer mismatch handled?
- How fresh is the behavior policy relative to the updated student?
Unresolved outside scope The toy assumes an aligned two-token vocabulary, positive probabilities, a two-position unweighted mean, a fresh tempered behavior policy, and a detached canonical GKD draw. It does not resolve EOS inclusion, variable-length reduction, tokenizer mapping, stale-policy correction, generic top-k semantics, teacher correctness, or real-model performance. OBS-CL-012
Primary reading
- Rishabh Agarwal et al., On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes, ICLR 2024; author manuscript v3.
- Yuxian Gu et al., Knowledge Distillation of Large Language Models, ICLR 2024; current MiniLLM manuscript v6.
- Yoon Kim and Alexander Rush, Sequence-Level Knowledge Distillation, EMNLP 2016.
- Alexander Lin et al., Autoregressive Knowledge Distillation through Imitation Learning, EMNLP 2020.
- Kevin Lu and Thinking Machines Lab, On-Policy Distillation, engineering report, October 2025.