Five Prefix Autopsies

A diagnostic casebook for on-policy distillation

Here is the first training record on the desk:

Is this on-policy distillation?

Not enough information appears in the rendered answer alone. In this record, the missing fact is decisive: a teacher decoded 22 <EOS> earlier, and the sequence was stored before this student update. The student is learning now, but it did not generate the continuation it is learning on.

That distinction is the center of this casebook. We will examine seven invented training records through the same five questions. By the end, you should have a practical way to describe a record without letting one attractive label do five different jobs.

This is a diagnostic reading experience, not a benchmark or learner study. Its experimental premise—that repeated classification of matched near misses may help practical distinction-making—is unproven. Completing the final set does not establish mastery, retention, or real-world performance.

Evidence key

• SOURCE means a claim is supported by the linked primary paper or official record, within the stated scope.
• DERIVED means arithmetic or a transparent consequence of disclosed inputs.
• TOY means an invented prompt, probability, completion, score, or mixture. No item with this label is a real model run.
• INFERRED means a bounded interpretation of a cited objective or algorithm.
• UNRESOLVED means the record lacks information needed for a faithful equality or method claim.

All readable word pieces in the dossiers are illustrative; they do not assert production tokenizer boundaries. Probability-vector cases explicitly assume aligned teacher/student token support.

The autopsy sheet

Every dossier uses CASE-FRAME-001 in the same order:

1. Who generated this prefix? Record the prompt source separately from the continuation source.
2. What exactly did the teacher supply? A hard target, a dense distribution, a subset of probabilities, a sequence-level likelihood signal, or one scalar score?
3. Which loss is computed where? Name the visited state or sequence, the objective, and the KL direction when there is one.
4. Does sampling receive a gradient? Separate no draw, a detached discrete draw, a differentiable local distribution loss, and score-function credit.
5. What changes next? Usually the student parameters—not the recorded token, prefix, or teacher values.

This is not merely a checklist. Removing one field collapses cases that train differently. A dense teacher vector does not reveal who generated its prefix. A reverse KL does not reveal whether its state was fixed. A sampled token does not reveal how its probability enters the estimator.

A faithful GKD-style step can likewise be read as five distinct operations: sample or select a state source, record the visited prefix, evaluate teacher and student at that prefix, construct a loss, then update the student. This decomposition follows Agarwal et al.’s algorithm; it should not be read as a claim about universal wall-clock scheduling. [SOURCE + INFERRED · OPD-CL-004] (author manuscript v3, Algorithm 1)

Dossier 01 — The teacher wrote the answer

CASE-DOS-01 · TOY · CASE-CL-016

Prompt: “Write only the sum: 17 + 5 =”. Before student training, a teacher decoder stores 22 <EOS>. At each stored position, the record retains only the selected target token. It does not retain the teacher’s probability for alternatives.

Diagnosis: basic/mode-approximation SeqKD-style hard-target training on a fixed teacher-decoded sequence. It is not on-policy student-prefix distillation for the student now being updated.

Tempting wrong label: “It is on-policy because the student is updated immediately.” Update timing does not rewrite continuation provenance.

Evidence seal. Kim and Rush’s basic SeqKD construction approximates the teacher sequence distribution with a high-scoring decoded output, then trains the student on the resulting generated dataset. This supports the diagnosis above, but it does not establish that every later method called “sequence KD” uses exactly one beam output. [SOURCE · CASE-CL-001] (Kim & Rush, EMNLP 2016, §3.2, pp. 1319–1320)

Now change one thing: keep the stored path, but ask the teacher for more information.

Dossier 02 — Same prefix, fuller teacher payload

CASE-DOS-02 · TOY · CASE-CL-016

Reuse the arithmetic prompt and fixed teacher-decoded prefixes from Dossier 01. At each same fixed prefix, query the teacher for a full next-token distribution over a declared, aligned toy vocabulary.

Diagnosis: dense full-distribution distillation on fixed prefixes; still not on-policy state sampling.

Tempting wrong label: “Full teacher probabilities make the record on-policy.” Feedback richness and prefix provenance occupy different fields.

In Agarwal et al.’s supervised token-level KD expression, the teacher and student distributions are compared on prefixes from a fixed dataset. The equation can carry dense teacher information without turning those prefixes into student-generated states. [SOURCE + INFERRED · CASE-CL-002] (Agarwal et al., author manuscript v3, Eqs. 2–3)

Controlled pair 01 — Hard versus dense, provenance held fixed

[INFERRED · CASE-PAIR-01] A hard target discards the teacher’s allocation over alternatives; a dense vector retains it. Neither payload makes the fixed state on-policy.

The next record changes provenance for real: the student chooses the branch.

Dossier 03 — The student takes the branch; forward KL follows

CASE-DOS-03 · OPD-TOY-001 · TOY · CASE-CL-016

Begin with fixed prompt x and an aligned toy vocabulary [A, B]. At x, the student assigns [0.75, 0.25] to [A, B]. Supply the uniform value u=0.8. Because the cumulative interval for A ends at 0.75, the categorical draw is B, and the next visited prefix is xB.

At the two visited states, the fixture contains:

Every number is invented. The behavior snapshot that draws B is declared to be the current student for this local record. That declaration matters: text produced by an older student is not automatically on-policy for a later update.

Diagnosis: GKD-style on-policy prefix distillation with a forward-KL objective under this local detached-draw contract.

Tempting wrong label: “Forward KL means off-policy.” It does not. KL direction tells us how distributions are compared, not who generated xB.

Agarwal et al.’s on-policy loss samples a completion from the student and evaluates teacher and student token distributions on its intermediate prefixes; the paper’s Eq. 4 uses forward KL. Its algorithm separately chooses the student-data fraction and the divergence. [SOURCE · OPD-CL-001, OPD-CL-002, OPD-CL-003, CASE-CL-003] (official ICLR 2024 record; author manuscript v3, §3.1 and Eq. 4)

A fixed prompt set is compatible with this: prompt x can come from a fixed collection while the continuation and later prefixes come from the student. [SOURCE · OPD-CL-006] (Agarwal et al., §3 and Eq. 4)

There is one more fixed record in the fixture: xA, from a teacher-demonstration comparison. It is displayed later, but it is not part of this student-visited path and must not enter the canonical mean.

Dossier 04 — Same student branch; reverse the discrepancy

CASE-DOS-04 · OPD-TOY-001 · TOY · CASE-CL-016

Reuse everything from Dossier 03: prompt x, current-student draw B, visited prefix xB, both student vectors, both teacher vectors, detached draw, and unweighted two-position reduction. Change only the local divergence from teacher-to-student to student-to-teacher.

Diagnosis: GKD-style on-policy prefix distillation with reverse KL under the same local detached-draw contract.

Tempting wrong label: “This is on-policy because reverse KL is on-policy.” Dossier 03 is the counterexample: it is equally on-policy with forward KL.

GKD treats divergence choice and student-data fraction as separate controls and includes reverse KL among its divergence variants. That supports this pairing; it does not imply reverse KL is universally preferable. [SOURCE · OPD-CL-003, CASE-CL-004, CASE-CL-017] (Agarwal et al., §3.1 “Choice of Divergence” and §4)

Controlled pair 02 — Forward versus reverse, provenance held fixed

[DERIVED · CASE-PAIR-02] Both records are on-policy on the provenance axis. Only the weighting inside the discrepancy changes.

The notation the cases now require

We have finally earned symbols. Let

• s_t=(x,y_{<t}) be the exact prompt-plus-prefix state before token position t;
• S_θ(v|s_t) be the trainable student probability of vocabulary item v at that state;
• T(v|s_t) be the teacher probability at the same state; and
• V be an explicitly aligned teacher/student vocabulary.

Forward KL is teacher-weighted:

Reverse KL is student-weighted:

[SOURCE · OPD-CL-003, CASE-CL-003, CASE-CL-004] These are the conventional KL definitions used in the GKD paper. They are asymmetric; if the weighting distribution has positive mass where the denominator distribution has zero mass, the divergence is infinite rather than silently clipped. (Agarwal et al., §2)

Rendering contract. The displayed equations are the canonical LaTeX source. The site should render them with semantic MathML, for example through KaTeX’s HTML+MathML output, and retain the nearby prose as the no-script/print interpretation. Do not replace them with raster images. Plain-text fallback names are D_KL(T || S) and D_KL(S || T).

Compute OPD-TOY-001

All inputs in this subsection are TOY. All values produced from them are DERIVED. Natural logarithms make the unit nats. The fixture scores exactly two positions, x and xB, with an unweighted mean. It models no EOS transition.

At x, forward KL is

At xB, reverse KL is

The complete recomputation is:

[DERIVED · OPD-TOY-001, CASE-LAB-001] The different means arise because the two directions weight the same log ratios differently. They do not show that either objective is better for a real model. xA is a useful contrast precisely because it is excluded.

Lab bench — recompute, then inspect membership

CASE-LAB-001 is a standard-library Python companion. It is an arithmetic demonstration, not a language-model training run. The complete runnable source is lab/run.py; the focused excerpt below is CASE-LAB-LIST-002.

def kl_divergence(p, q):
    p_checked = validate_distribution(p, label="p")
    q_checked = validate_distribution(q, label="q")
    if len(p_checked) != len(q_checked):
        raise ValueError("p and q must have the same length")

    terms = []
    for p_i, q_i in zip(p_checked, q_checked):
        if p_i == 0.0:
            continue
        if q_i == 0.0:
            return math.inf
        terms.append(p_i * math.log(p_i / q_i))
    return math.fsum(terms)

Run from the variant folder:

python -B lab/run.py

Deterministic output — captured from the delivered constants and Python standard-library operations:

prefix  source                 reverse KL   forward KL   canonical mean?
x       fixed rollout start      0.130812     0.143841   yes
xB      student-visited          0.311239     0.226289   yes
xA      off-policy comparison    0.582685     0.534111   no

mean(x, xB)                      0.221025     0.185065
draw: u=0.8 -> B -> xB; gradient_through_draw=false (local toy only)
xA is displayed for comparison and excluded from both means.

The lab keeps student_sampling → visited_prefix → teacher_evaluation → local_loss_computation → parameter_update as distinct named stages (CASE-LAB-LIST-004). The last stage is intentionally not executed: this fixture does not invent parameters, gradients, or an optimizer. Its local draw is detached, but that does not prevent differentiation through S_θ(·|x) and S_θ(·|xB) in a selected local loss.

The captured run passed its eight internal contract checks, and the focused suite passed 31 tests under Python 3.12.0. Those are implementation checks, not model-quality or learner-outcome evidence. The lab does not test a real tokenizer, EOS handling, behavior-policy lag, sequence-level policy gradients, reward learning, or whether one KL direction is generally preferable.

Dossier 05 — One batch, two prefix sources

CASE-DOS-05 · TOY · CASE-CL-016

Prompt: “Condense to five words: The prototype battery lasted nine hours during the bench test.” An invented 50/50 source rule selects either a stored fixed prefix, Battery lasts, or a freshly sampled current-student prefix, Nine-hour battery. The teacher supplies a dense next-token vector at whichever prefix is selected. Both branches use the same forward-KL local loss.

Diagnosis: fixed/on-policy mixture—not purely on-policy and not purely fixed.

Tempting wrong label: “A fixed prompt set makes the whole batch offline.” The prompt is fixed; one continuation is freshly generated by the student.

For a disclosed toy weight \lambda_{\mathrm{data}}=0.5, the case is represented as

[SOURCE + TOY · CASE-CL-005] GKD’s λ controls how often student-generated rather than fixed outputs supply training states; 0.5 here is only an invented clarity aid, not a recommendation. (Agarwal et al., §3.1, Algorithm 1 and boxed objective)

Do not confuse this coefficient with generalized-JSD β, which changes a divergence, or ImitKD’s β_i, which controls an initial-data versus student-sequence mixture in that paper. [SOURCE · CASE-CL-006, CASE-CL-007] (Agarwal et al., §2 Eq. 1; Lin et al., EMNLP 2020, §§3.2–3.3)

Dossier 06 — A teacher score is not automatically distribution distillation

CASE-DOS-06 · TOY · CASE-CL-016

Reuse the summary prompt. The current student samples Nine-hour battery bench result. <EOS>. A teacher-like judge returns only the invented scalar 0.8 for the completed sequence. The record contains no hard teacher target, dense teacher distribution, or sampled-token teacher likelihood.

Diagnosis: on-policy scalar-reward RL in this record. Calling it prefix-distribution OPD would be unsupported.

Tempting wrong label: “A teacher produced feedback, so this must be distillation.” Teacher identity does not specify payload or objective.

The bounded claim is source-specific: Agarwal et al.’s combined objective writes expected scalar reward and generalized on-policy distillation as separate summands. This invented record contains the scalar-reward term and no distribution-matching term. RL and distillation can still be combined; this dossier does not declare them mutually exclusive or impose a universal naming rule on all literature. [SOURCE + INFERRED · CASE-CL-008] (Agarwal et al., §3.2 Eq. 5)

One sharp score is not proof of correctness. Nor would a sharp teacher probability vector be proof: it is a teacher-matching signal, and the cited GKD paper explicitly notes a teacher can hallucinate. [SOURCE + INFERRED · CASE-CL-014] (Agarwal et al., §4.1)

Dossier 07 — Sequence-level reverse KL is a different estimator family

CASE-DOS-07 · TOY surface + SOURCE method · CASE-CL-009, CASE-CL-016

Prompt: “Correct OCR: rnodern rneter.” In the toy surface, the current student samples modern meter <EOS>. We now use that surface to inspect MiniLLM’s method family: a sequence-level student-to-teacher reverse-KL objective with policy-gradient semantics.

Diagnosis: MiniLLM-style sequence-level reverse-KL policy-gradient distillation, kept separate from the local detached-rollout GKD toy.

Tempting wrong label: “It is just Dossier 04 summed over more tokens.” The objective level, estimator, feedback requirements, sampler, normalization, and gradient semantics differ.

The sequence-level objective family can be written

MiniLLM derives a Monte Carlo policy-gradient estimator, then introduces an exact single-step term that consumes the dense teacher next-token distribution at sampled prefixes, a long-horizon return that uses teacher likelihood on sampled tokens, a teacher/student mixture behavior distribution, importance weighting, and length normalization. Its objective is student-distribution reverse KL; its practical behavior sampler can be mixed when the paper’s mixture coefficient is nonzero. State the two payloads and the behavior source separately rather than calling every realized prefix strictly student-only. [SOURCE · OPD-CL-007, CASE-CL-009] (official ICLR 2024 record, published as “MiniLLM: Knowledge Distillation of Large Language Models”; current author manuscript v6, retitled “MiniLLM: On-Policy Distillation of Large Language Models,” §§2.1–2.3)

Controlled pair 03 — Local reverse KL versus sequence reverse KL

[SOURCE · OPD-CL-007] Two shared labels do not establish estimator equivalence.

The two-axis field map

The seven records can now be summarized without pretending they form one linear spectrum.

Misconception ledger

The wrong labels are tempting because real method names often bundle several choices. This ledger points back to the field that actually discriminates. It is also the complete static form of CASE-FIG-006.

Blind diagnosis set

Apply the five autopsy questions before reading the key. The prompts appear without category labels, but the explanations remain openly available below. A future interactive treatment may record selections; it must not gate this key or convert a tally into a mastery claim.

OPD-XFER-A

OPD-XFER-B

CASE-XFER-001 — same OCR text, different records

Record A · TOY: For “Correct OCR: c1ear,” a teacher decoder stores clear <EOS>. The student later minimizes hard-target cross-entropy on that fixed sequence.

Record B · TOY: For the same prompt, the current student samples clear <EOS>. A judge returns only scalar 1.0, and the trainer applies a disclosed policy-gradient estimator.

CASE-XFER-002 — incomplete top-k record

Record · TOY: A student snapshot generated a summary prefix, but the record does not identify whether that behavior snapshot matches the student now being updated. The teacher returns its top two token probabilities, but the record does not say whether remaining mass is discarded, renormalized, aggregated, or corrected. The trainer report calls the loss “the same as full forward KL.”

Open answer key

OPD-XFER-A — two-axis explanation

• Objective axis: Yes. The record explicitly says it computes reverse KL.
• Provenance axis: No. xA is fixed and teacher-generated relative to the student being updated; no fresh student draw produced that state.
• Why: Reverse KL changes which distribution weights the log ratio. It does not change the source of the prefix on which the ratio is evaluated. [SOURCE · OPD-CL-003]

OPD-XFER-B — two-axis explanation

• Provenance axis: Yes, under the stated current/fresh student behavior snapshot. The student draw B creates visited prefix xB, even if prompt x came from a fixed set.
• Feedback/objective axis: The teacher is queried at that same xB state and forward KL is computed there, so this is a local on-policy distillation record under the aligned-support assumptions.
• Why: Forward rather than reverse KL changes the discrepancy’s weighting, not who generated xB. [SOURCE · OPD-CL-001, OPD-CL-002, OPD-CL-003, OPD-CL-006]

CASE-XFER-001 — identical text does not imply identical training

Record A: The prompt is fixed and the continuation is teacher-decoded/stored. The teacher supplies hard targets. Hard-target cross-entropy is computed on fixed positions. There is no current-student rollout draw. Student parameters update toward the stored sequence. Diagnosis: basic SeqKD-style fixed hard-target training.

Record B: The prompt is fixed but the completion comes from the current student. The judge supplies one uncalibrated scalar, not token targets or a distribution. A sequence-return policy-gradient estimator is used; sampled tokens receive score-function credit rather than pathwise differentiation. Student parameters update toward higher expected scalar return. Diagnosis: on-policy scalar-reward RL in the stated record.

The rendered completion clear <EOS> matches in both records, but the provenance, payload, objective, and estimator do not. Both strings and the score are TOY, not observed model results.

CASE-XFER-002 — refuse the false equality

Prefix provenance is UNRESOLVED because the record does not supply behavior-policy freshness. The loss identity is also UNRESOLVED. Two teacher probabilities are not a full-vocabulary vector unless the method specifies aligned support, how top-k was selected, what happens to missing mass, whether either distribution is renormalized or corrected, the objective formula, and the position reduction. EOS, truncation, and length/mask handling are also missing.

Therefore the record cannot be called “the same as full forward KL.” Current primary sources distinguish full-vocabulary, sampled-token, and teacher-top-k objectives; the latter two do not become equivalent because they produce one scalar. This boundary is supported by a 2026 preprint and an author engineering report, so empirical conclusions retain those source statuses. [QUALIFIED SOURCE · CASE-CL-010] (Fu et al., arXiv v2, §3.1; Thinking Machines Lab engineering report)

The field card

Before accepting any “on-policy distillation” label in a codebase or paper, record:

1. prompt source;
2. continuation-prefix source and behavior-policy version;
3. freshness, replay, mixture, or importance-correction rule;
4. teacher payload and its support;
5. tokenizer/vocabulary alignment;
6. objective level and direction;
7. discrete-sampling and gradient-estimator semantics;
8. sampling temperature and decoding rule;
9. EOS, truncation, masks, length normalization, and reduction; and
10. the parameters that actually update.

The opening arithmetic record is now straightforward: fixed prompt, fixed teacher-decoded continuation, hard targets, hard-target loss on stored prefixes, no current-student draw, student update. The model name was never the key; the record was.

Where this casebook stops

OPD-TOY-001 assumes a shared two-token vocabulary, natural logs, a supplied draw at temperature implicit in the fixed probabilities, exactly two scored positions, no modeled EOS transition, a fresh behavior snapshot, and a detached local draw. It executes no optimizer step. The summary and OCR records are also invented. None establishes model quality, training stability, teacher correctness, process supervision, causal blame, efficiency, or a preferred divergence.

Cross-tokenizer mapping, stale or asynchronous rollouts, partial-support corrections, tool-using trajectories, teacher ensembles, adaptive teachers, and production serving are outside this treatment. So are universal comparisons among OPD, SFT, fixed distillation, and RL. A stronger or more confident teacher is not guaranteed to help.

Primary source path

• Peer-reviewed ICLR 2024: Agarwal et al., “On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes”, with technical details from author manuscript arXiv v3.
• Peer-reviewed EMNLP 2016: Kim & Rush, “Sequence-Level Knowledge Distillation”.
• Peer-reviewed EMNLP 2020: Lin et al., “Autoregressive Knowledge Distillation through Imitation Learning”.
• Peer-reviewed ICLR 2024 record: Gu et al., “MiniLLM: Knowledge Distillation of Large Language Models”, with detailed equations from the later, retitled author manuscript arXiv v6.
• Author engineering report, not peer reviewed: Thinking Machines Lab, “On-Policy Distillation”.
• 2026 arXiv preprint, work in progress: Fu et al., “Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes,” v2.

A good diagnosis is not a fashionable method name. It is a compact record of provenance, payload, objective, estimator, update, and the assumptions that make each field meaningful.