What does “on-policy” actually modify?

OPD-CORE-v1 · canonical teaching contract A student generates a response. A teacher is queried along that response. A KL loss compares their next-token distributions. Which part makes the recipe on-policy?

The answer is the first part: the student generated the response trajectory, so the response's prefixes are student-visited states. Forward KL, reverse KL, and other divergences describe a different choice—the local comparison made after a prefix has been selected.

Source-backed: Agarwal et al.'s GKD objective exposes a student-output fraction separately from the divergence choice. See the author manuscript §3.1 and §4 variants.

Write the source outside the local loss

Let $\rho$ name the source of prompt–response pairs. Once a response $y$ is sampled, its prefixes $s_t=(x,y_{<t})$ identify where teacher and student will be compared:

L_D(\theta;\rho)=\mathbb E_{(x,y)\sim\rho}\!\left[\frac{1}{T(y)}\sum_{t=1}^{T(y)}D\!\left(p(\cdot\mid s_t)\,\middle\|\,q_\theta(\cdot\mid s_t)\right)\right].

The expectation chooses trajectories and therefore visited prefixes. The inner $D$ compares two distributions at one already chosen prefix. In other words, prefix provenance and divergence direction are different decisions. Keeping those jobs on separate lines is the central habit of this guide.

Math layer · optional derivation

Result. The trajectory-source expectation changes how often prefixes receive weight. The local divergence still compares teacher and student at one already selected prefix.

Scope. The λ table below mixes two conditioned paths, [x,xA] and [x,xB]. It is not the exact expectation over every trajectory the student could sample.

Derive normalized occupancy and the λ = 0, 0.5, 1 values

For any response source $\rho$ , define its length-normalized occupancy of prefix $s$ by

d_{\rho}(s)=\mathbb E_{(x,y)\sim\rho}\!\left[\frac{1}{T(y)}\sum_{t=1}^{T(y)}\mathbf 1\!\left\{s_t=s\right\}\right],\qquad \sum_s d_{\rho}(s)=1.

Linearity of expectation separates source weighting from the inner comparison:

L_D(\theta;\rho)=\sum_s d_{\rho}(s)\,D\!\left(p(\cdot\mid s)\,\middle\|\,q_\theta(\cdot\mid s)\right).

For fixed outputs and student outputs collected from a detached behavior snapshot $q_{\bar\theta}$ ,

\rho_\lambda=(1-\lambda)\rho_{\mathrm{fixed}}+\lambda\rho_{q_{\bar\theta}}\quad\Longrightarrow\quad d_{\rho_\lambda}=(1-\lambda)d_{\mathrm{fixed}}+\lambda d_{q_{\bar\theta}}.

For either fixed KL direction, the declared two-path contrast is

\widehat L_{D,\lambda}=(1-\lambda)\widehat L_D^{\mathrm{fixed}}+\lambda\widehat L_D^{\mathrm{student}}.

$\lambda=0$ uses only the fixed route, $\lambda=0.5$ gives both routes equal coefficients, and $\lambda=1$ uses only the realized student route.

Conditioned two-path balance sheet; all values are nats
λ	Fixed coefficient	Student coefficient	Reverse KL	Forward KL
0	1	0	0.3567486689921882	0.33897592246809893
0.5	0.5	0.5	0.28888701337714273	0.2620205105868618
1	0	1	0.22102535776209725	0.18506509870562465

This equality is exact for those two conditioned paths. A full student-trajectory expectation would also integrate over all other possible draws and resulting prefixes.

Place the recipe on both coordinates

OPD-FIG-001 · sourced relationshipPrefix source × divergence direction

Move horizontally to change who generated the response and its prefixes. Move vertically to change the local comparison. Neither coordinate determines the other.

A fixed prompt set does not settle the source axis. GKD can sample prompts from a fixed collection and still sample completions from the current student. Conversely, a fixed teacher-generated completion can be paired with reverse KL without becoming on-policy.

The loss does not choose the prefix. So what does?