Reading companion

Glossary & sources

Definitions, source status, assumptions, and any optional marked additions behind the field guide.

A small vocabulary

Policy: A conditional distribution over the next token in this guide.
Trajectory: A generated response sequence; its prefixes provide the states evaluated during tokenwise distillation.
Prefix: The prompt plus response tokens generated before the current position.
On-policy: Here, the response occurrence was sampled from the student behavior policy used for collection.
Behavior policy: The policy that generated the collected response; it may differ from the current trainable parameters later.
Teacher: The model supplying supervision at the selected prefix; teacher probability is not automatically correctness.
Forward KL: D_KL(teacher || student) at one held-fixed prefix in this notation.
Reverse KL: D_KL(student || teacher) at one held-fixed prefix in this notation.
Nat: The information unit produced by natural logarithms.
Stop-gradient: An estimator choice that treats a dependency as fixed during differentiation; here it applies to the sampled trajectory.

Primary sources and status

SRC-GKD-ICLR-2024 · Peer-reviewed · ICLR 2024 official record
On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes
Canonical publication anchor for GKD. It does not turn this toy into model evidence.
SRC-GKD-ARXIV · Author manuscript v3 · 2024-01-17 · accepted at ICLR 2024
Agarwal et al. author manuscript, arXiv:2306.13649v3
Navigable equations, Algorithm 1, source mixture, divergence choices, and stop-gradient statement.
SRC-MINILLM-ICLR-2024 · Peer-reviewed · ICLR 2024 official record
MiniLLM: Knowledge Distillation of Large Language Models
Sequence-distribution reverse KL and its policy-gradient decomposition; distinct from the canonical local toy.
SRC-OPD-RETHINKING-2026 · Recent preprint evidence · arXiv 2026 · not peer reviewed
Rethinking On-Policy Distillation of Large Language Models
Preprint evidence used only to mark current feedback-granularity questions; no empirical conclusion is imported into the guide.
SRC-SEQKD-EMNLP-2016 · Peer-reviewed · EMNLP 2016
Sequence-Level Knowledge Distillation
Fixed teacher beam outputs used as hard sequence targets.
SRC-IMITKD-EMNLP-2020 · Peer-reviewed · EMNLP 2020
Autoregressive Knowledge Distillation through Imitation Learning
Sequence-source mixture, oracle supervision, schedule, and sample-pooling boundaries.

What this guide does not establish

The probabilities in OPD-TOY-001 are invented teaching inputs. They are not a trained language model, a benchmark, or evidence that one method wins. No learner study or controlled learning outcome is claimed. Cross-tokenizer alignment, EOS inclusion, production length weighting, behavior-policy lag, and teacher correctness remain implementation questions beyond the fixture.

Math layer · extended notation

Extended notation and estimator boundary

These symbols expand the optional derivations without changing the canonical OPD fixture.

Source, divergence, reduction, and gradient symbols
Symbol	Meaning here	Boundary to keep explicit
$x$	Prompt; also the fixture's first evaluated prefix.	A fixed prompt set does not imply fixed responses.
$y,\ T(y)$	Sampled response and number of evaluated positions.	The fixture evaluates two positions and does not model EOS.
$s_t=(x,y_{<t})$	Prefix at response position $t$ .	Provenance belongs to its collection event, not the prefix string alone.
$\rho$	Distribution over prompt–response trajectories.	It selects visited prefixes before a local divergence is evaluated.
$\rho_{\mathrm{fixed}}$	Supplied fixed-output source, represented by `[x,xA]`.	This occurrence is off-policy for the current student.
$\rho_{q_{\bar\theta}}$	Student-output source collected from behavior snapshot $q_{\bar\theta}$ , represented by realized `[x,xB]`.	The categorical path is detached for the local update.
$\lambda$	Student-output coefficient in $(1-\lambda)\rho_{\mathrm{fixed}}+\lambda\rho_{q_{\bar\theta}}$ .	The displayed table mixes two conditioned paths, not every possible student path.
$d_\rho(s)$	Length-normalized prefix occupancy induced by source $\rho$ .	It carries source weighting; it is not a divergence.
$p(\cdot\mid s)$	Teacher's full next-token distribution at held-fixed prefix $s$ .	Teacher probability is not automatically a scalar reward or correctness label.
$q_\theta(\cdot\mid s)$	Current student's differentiable next-token distribution at $s$ .	$\theta$ receives the local update while the selected trajectory stays fixed.
$D_F,\ D_R$	Forward $D_{\mathrm{KL}}(p\\|q)$ and reverse $D_{\mathrm{KL}}(q\\|p)$ .	Both use natural logarithms and report nats; neither chooses the source.
$\widehat L_R$	Unweighted mean of reverse KL over the two realized positions.	Compute full-precision terms, then round the final mean once.
$z,\ \tau$	Student logits and softmax temperature; $\tau=1$ in OPD-TOY-001.	The finite difference check perturbs local logits only.

Random during collectionThe student's categorical token draw that realizes xB.

Held fixed during this updateThe realized route, prefix identity, teacher vector, and behavior snapshot.

Differentiated locallyThe current student logits and parameters evaluated at each held-fixed prefix.

Reproduce the fixture checks

Python lab (local copy)
Python tests (local copy)
Lab manifest (local copy)
Numerical receipt (local copy)
Evidence ledger (local copy)

← Return to contents