Methods that share a surface

Student generations and teacher feedback appear in several recipes. Their targets and gradients still differ.

Compare five fields, not one label

A method name is a poor substitute for an estimator. Ask about trajectory source, feedback granularity, objective, gradient treatment, and practical freshness.

Boundary map · primary-source status shown in final column
Method	Trajectory / prefix source	Teacher feedback	Training target	Gradient through source law?	Status
SeqKD	fixed teacher beam output	hard teacher-generated sequence	student NLL on that fixed sequence	no current-student source	EMNLP 2016
ImitKD	sequence-level mixture of fixed and student-generated examples; schedule may anneal	teacher optimal action or full token distribution at visited prefixes	oracle NLL or cross-entropy	sample generation is used as data; practical pools may be a few iterations old	EMNLP 2020
GKD / canonical toy	fixed, mixed, or current-student outputs via λ	full next-token distribution at each selected prefix	declared local divergence, averaged over positions	no; sampled response is detached in canonical estimator	ICLR 2024
MiniLLM	theory: student response distribution; practical behavior is teacher-mixed	teacher/student log ratios plus full-vocabulary single-step term	sequence-distribution reverse KL with policy-gradient decomposition	yes, through score-function terms—not pathwise through a token	ICLR 2024
Scalar-reward RL	often student trajectories, depending on algorithm	external scalar reward	expected reward / surrogate	algorithm-dependent score-function or surrogate	separate term in GKD Eq. 5
Sampled-token engineering recipe	student trajectories	teacher and student log-probability of sampled token	per-token reverse log-ratio as advantage in importance-sampling loss	RL-style loss implementation	Thinking Machines 2025 report
Top-k soft target	separate choice; docs example uses teacher completion	teacher probabilities renormalized over selected top-k tokens	soft-target cross-entropy over k entries	depends on surrounding sampler	current Tinker docs

feedback granularity Full vocabulary, one sampled token, and renormalized top-k probabilities are not equivalent observations. “Token-level” is too coarse a label to establish the estimator.

Fixed teacher sequence is not teacher distribution

SeqKD. Kim and Rush generate a fixed dataset with teacher beam search and train the student on a selected teacher sequence. Their EMNLP 2016 paper, Fig. 1 and the sequence-level method on pp.1318/1322, make the hard-sequence target explicit. This is not full-distribution matching at a student-generated prefix.

ImitKD. Lin et al. mix fixed sequences with student-generated sequences and query an oracle at the resulting prefixes. Their EMNLP 2020 paper, §3.1–3.4 and Algorithm 1, permits either the teacher's optimal action or full-distribution cross-entropy. Its $\beta_i$ schedule and optional sample pooling are not simply a constant GKD $\lambda=0.5$ .

MATH-NOTE-001 — MiniLLM starts from a different objective

bounded advanced note

Keep the equations in separate boxes

MiniLLM's peer-reviewed conference target is reverse KL between complete response distributions:

MINILLM EQ. 1 · sequence distribution

\mathcal L_{\mathrm{MiniLLM}}(\theta)=D_{KL}\!\left(q_\theta(y\mid x)\,\middle\|\,p(y\mid x)\right).

Here the student response distribution is part of the mathematical target, not merely a detached collector.

Its policy-gradient derivation contains a future cumulative log-ratio $R_t$ :

MINILLM EQ. 2 · score-function form

\nabla\mathcal L=-\mathbb E_{x,\,y\sim q_\theta}\sum_{t=1}^{T}(R_t-1)\nabla\log q_\theta(y_t\mid y_{<t},x).

This is a likelihood-ratio gradient. It does not make the categorical token pathwise differentiable.

The practical method then decomposes a full-vocabulary single-step term from a long-horizon score-function term and adds teacher-mixed sampling, approximate importance weighting, length normalization, clipping, and an auxiliary language-modeling loss. See the ICLR 2024 paper, §2.1–2.3 and Appendix A.2.

Boundary: the canonical GKD toy's fixed-state reverse-KL gradient is not a derivation of MiniLLM. Both may visit student-reachable prefixes; their objectives and retained gradient terms differ.

Sampled token, top-k, and full vocabulary

A 2025 Thinking Machines engineering report samples student trajectories, asks teacher and student for the sampled token's log-probability, and uses the negative reverse log-ratio as a per-token advantage in an importance-sampling RL loss. The report explicitly says its experiments did not use top-k logit distillation.

Current Tinker cross-entropy documentation separately exposes a top-k soft-target form, with teacher probabilities renormalized over selected target tokens. Truncating and renormalizing changes the compared distribution; one sampled token changes the available information again.

These are useful engineering choices, not evidence that one feedback granularity is universally better. The monograph records them to stop a notation shortcut from erasing implementation semantics.

What a method label still leaves unresolved

Vocabulary / tokenizer: The full-distribution KL assumes aligned entries. Cross-tokenizer mapping is outside this toy.
EOS and length: The paper writes a position mean; the toy fixes two evaluated positions. EOS inclusion and variable-length batching must be declared in an implementation.
Sampling temperature: GKD reports student temperature 1 during training. This fixture directly supplies probabilities at temperature 1.
Behavior freshness: Fresh per-step samples support the on-policy label. Replay or asynchronous collection may change it relative to current parameters.
Teacher semantics: A probability vector supplies dense supervision. It does not guarantee correctness, causal credit, or process supervision.
Teacher strength: A larger or stronger teacher is not guaranteed to help every student/configuration. No such claim is made here.