Distillation
Chapter 1 of 6
Glossary
Contents
Contents   /   Chapter 1

01

What does “on-policy” actually modify?

“On-policy” tells you where the training states came from. It does not tell you which way the KL points.

What does “on-policy” actually modify?

OPD-CORE-v1 · canonical teaching contract A student generates a response. A teacher is queried along that response. A KL loss compares their next-token distributions. Which part makes the recipe on-policy?

The answer is the first part: the student generated the response trajectory, so the response's prefixes are student-visited states. Forward KL, reverse KL, and other divergences describe a different choice—the local comparison made after a prefix has been selected.

Source-backed: Agarwal et al.'s GKD objective exposes a student-output fraction separately from the divergence choice. See the author manuscript §3.1 and §4 variants.

Write the source outside the local loss

Let ρ\rho name the source of prompt–response pairs. Once a response yy is sampled, its prefixes st=(x,y<t)s_t=(x,y_{<t}) identify where teacher and student will be compared:

LD(θ;ρ)=E(x,y)∼ρ ⁣[1T(y)∑t=1T(y)D ⁣(p(⋅∣st) ∥ qθ(⋅∣st))].L_D(\theta;\rho)=\mathbb E_{(x,y)\sim\rho}\!\left[\frac{1}{T(y)}\sum_{t=1}^{T(y)}D\!\left(p(\cdot\mid s_t)\,\middle\|\,q_\theta(\cdot\mid s_t)\right)\right].

The expectation chooses trajectories and therefore visited prefixes. The inner DD compares two distributions at one already chosen prefix. In other words, prefix provenance and divergence direction are different decisions. Keeping those jobs on separate lines is the central habit of this guide.

Place the recipe on both coordinates

OPD-FIG-001 · sourced relationshipPrefix source × divergence direction
Independent source and divergence axes The horizontal axis moves from fixed outputs through mixed outputs to current-student outputs. The vertical axis moves between forward and reverse KL. A marker at current-student outputs and forward KL shows that an on-policy trajectory can use forward KL. INNER COMPARISON forward KL reverse KL OUTER RESPONSE / PREFIX SOURCE fixed mixed current student on-policy + forward KL valid: source does not choose direction fixed + RKL
Move horizontally to change who generated the response and its prefixes. Move vertically to change the local comparison. Neither coordinate determines the other.

A fixed prompt set does not settle the source axis. GKD can sample prompts from a fixed collection and still sample completions from the current student. Conversely, a fixed teacher-generated completion can be paired with reverse KL without becoming on-policy.

The loss does not choose the prefix. So what does?