What does “on-policy” actually modify?

OPD-CORE-v1 · canonical teaching contract A student generates a response. A teacher is queried along that response. A KL loss compares their next-token distributions. Which part makes the recipe on-policy?

The answer is the first part: the student generated the response trajectory, so the response's prefixes are student-visited states. Forward KL, reverse KL, and other divergences describe a different choice—the local comparison made after a prefix has been selected.

Source-backed: Agarwal et al.'s GKD objective exposes a student-output fraction separately from the divergence choice. See the author manuscript §3.1 and §4 variants.

Write the source outside the local loss

Let $\rho$ name the source of prompt–response pairs. Once a response $y$ is sampled, its prefixes $s_t=(x,y_{<t})$ identify where teacher and student will be compared:

The expectation chooses trajectories and therefore visited prefixes. The inner $D$ compares two distributions at one already chosen prefix. In other words, prefix provenance and divergence direction are different decisions. Keeping those jobs on separate lines is the central habit of this guide.

Place the recipe on both coordinates

A fixed prompt set does not settle the source axis. GKD can sample prompts from a fixed collection and still sample completions from the current student. Conversely, a fixed teacher-generated completion can be paired with reverse KL without becoming on-policy.

The loss does not choose the prefix. So what does?

Who generated the supervised state?

OPD-TOY-001 · invented teaching fixture At prefix x, vocabulary order is [A,B] and the student distribution is [0.75,0.25]. The contract fixes a uniform draw at 0.8. Its inverse-CDF interval selects B, so the next visited prefix is xB.

The same contract also supplies a fixed teacher demonstration whose first token is A, reaching xA. In this comparison, the occurrence of xA is fixed/off-policy and the occurrence of xB is student-generated/on-policy at collection. A prefix string alone cannot reveal who generated it.

Keep collection, supervision, and update distinct

The five labels are a teaching decomposition. Implementations may fuse forward passes, but the logic remains: sampling determines the states; the teacher supplies feedback at those states; the objective turns feedback into a scalar; the update differentiates the declared estimator.

Detach the draw, not the student

The canonical GKD-style toy does not backpropagate through the discrete 0.8 → B draw. During the local update, the realized path [x,xB] is treated as fixed. The student distribution evaluated at x or xB is still differentiable through its logits and parameters.

Boundary. A fixed prompt collection is compatible with on-policy completion sampling. Reusing old responses after the student changes may not be. Record the behavior-policy snapshot instead of inferring freshness from a method name.

Now that xB exists, what exactly does the teacher provide there?

What does the teacher provide at xB?

At the student-visited prefix xB, the student assigns [0.6,0.4] to [A,B]. Querying the teacher at that same prefix returns [0.9,0.1].

The teacher's probability vector is dense token-distribution supervision. It is not automatically a scalar reward, a correctness certificate, or process-level causal feedback.

Source-backed: GKD Eq. 2 evaluates teacher and student next-token distributions conditioned on the identical prefix. See the author manuscript §3.

The local comparison begins after the state is fixed

For a chosen state $s$, abbreviate the teacher probabilities by $p_i=p(i\mid s)$ and the student probabilities by $q_i=q_\theta(i\mid s)$. A declared divergence maps these aligned vectors to one scalar:

The state source is metadata about how this occurrence entered the batch. Changing the divergence does not rewrite that history.

Do not hide feedback granularity

This guide's canonical toy compares the complete two-token vocabulary. Other recipes may expose one sampled token, a hard teacher action, a top-k truncation, or a scalar reward. That objective granularity changes what information is available even when each signal is casually called “teacher feedback.”

Toy assumption. Teacher and student share the aligned vocabulary [A,B]. Cross-tokenizer mapping, EOS handling, and variable-length weighting remain outside the fixture.

With the state held fixed, KL direction finally becomes the question.

Forward KL weights by the teacher

At one held-fixed prefix, forward KL places the teacher distribution first:

Teacher probabilities weight the log-ratios. The value uses natural logarithms, so its unit is nats.

Reverse KL weights by the student

Reverse KL swaps the arguments:

Student probabilities now weight the log-ratios. KL is asymmetric, so forward and reverse values need not match even though the prefix and both vectors are unchanged.

Reduce the two visited positions once

The canonical student path is [x,xB]. Its reverse-KL losses are 0.130812035941… and 0.311238679583… nats (six-decimal displays: 0.130812 and 0.311239). The contract uses an unweighted two-position mean:

Compute at full precision and round the final mean once. Averaging two already rounded display values can change the last displayed digit.

A local gradient is not a gradient through collection

With the prefix and teacher fixed, the student logits can receive a gradient from either KL. That derivative answers how to change the student distribution at this state. It does not differentiate through the earlier categorical token draw in the canonical estimator.

Estimator boundary. Sequence-level reverse-KL policy-gradient methods can retain score-function terms from the sampling law. They are distinct estimators, not a hidden form of this detached local calculation.

The same surface words can therefore describe different training semantics.

Which similarly named recipes differ?

A method name is a starting point, not a complete estimator. Ask what generated the response, what teacher information is visible, how it is reduced, and which dependencies reach the update.

Canonical GKD keeps source and divergence configurable

GKD can mix fixed and student-generated outputs with a coefficient $\lambda$, then choose a token-distribution divergence independently. Its canonical sampling path is stopped for the local update.

Peer-reviewed anchor: Agarwal et al., ICLR 2024. Navigable equations and Algorithm 1: author manuscript v3.

Sequence-level reverse KL retains different terms

MiniLLM begins from reverse KL between complete response distributions and derives likelihood-ratio terms involving sampled tokens. Its practical estimator adds further approximations and stabilizers. That is not the same gradient semantics as freezing a sampled GKD trajectory and differentiating only local token-distribution losses.

Peer-reviewed anchor: Gu et al., MiniLLM, ICLR 2024. See §2 and Appendix A.2 in the linked proceedings paper.

Six details the label still does not answer

1. Prompt source: fixed collection, live environment, or another distribution?
2. Response source: teacher, current student, stale behavior policy, or a mixture?
3. Feedback granularity: full vocabulary, top-k, one sampled token, hard target, or scalar reward?
4. Objective and reduction: which arguments, positions, lengths, and units?
5. Gradient semantics: detached collection, score-function term, importance weighting, or another estimator?
6. Vocabulary and EOS: how are supports aligned and sequence boundaries counted?

Use those questions on two recipes whose axes deliberately disagree.

OPD-XFER-A — Reverse loss, fixed prefix source

OPD-XFER-B — Student prefix source, forward loss

Carry a five-question rule

1. What samples the response?
2. Which prefix occurrence is evaluated?
3. What teacher feedback is visible?
4. What objective and reduction are formed?
5. Which dependence reaches the gradient?

If those answers are explicit, “on-policy” becomes one coordinate again instead of a bag of assumptions.

The field-guide version

The source chooses the visited states. The local divergence compares distributions at a visited state. The gradient estimator decides which parameter dependence survives.

This guide explains a canonical detached GKD-style toy and several boundaries. It does not define every method called On-Policy Distillation, report a real-model benchmark, or establish a measured learner outcome.