Two axes, one overloaded phrase

“On-policy” tells you where a trajectory came from. It does not tell you which way the KL points.

A shortcut that breaks the objective

Suppose a student language model generates a response, a teacher is queried along that response, and the loss is reverse KL. Which part makes the update on-policy?

The tempting answer is “reverse KL.” It is also the wrong coordinate. Reverse KL names the order of two distributions inside a local comparison. On-policy names the source of the response trajectory—and therefore the prefixes at which that comparison is evaluated.

This monograph follows that factorization all the way down. By the end, you will be able to point to the source measure, freeze one prefix, choose the local loss, and say exactly which dependence the gradient estimator keeps.

First, make “policy” small enough to inspect

A policy need not arrive with rewards, value functions, or a simulator. Here it is simply a conditional next-token distribution. Given a prompt $x$ and earlier response tokens $y_{<t}$ , the student returns

MATH-EQ-POLICY-001

q_\theta(\cdot\mid x,y_{<t})\in\Delta^{M-1}.

Read it as: at one prefix, the student assigns one probability to each of $M$ vocabulary items. The vector is unitless and sums to one.

A response is built one token at a time, so its probability factors into those conditionals:

MATH-EQ-AR-001 · derived

q_\theta(y\mid x)=\prod_{t=1}^{T}q_\theta(y_t\mid x,y_{<t}).

An early draw does more than add one token. It determines which later prefix is visited.

Call that prefix-state $s_t=(x,y_{<t})$ . Once a trajectory has been generated, it gives us a sequence of states $s_1,s_2,\ldots$ at which a teacher can be queried.

Put source and loss on different coordinates

The inspection plane

OUTER MEASUREWhere did the response and visited prefixes come from?

fixed data ↔ mixed ↔ current student

INNER COMPARISONHow are teacher and student compared at a held-fixed prefix?

forward KL · reverse KL · JSD · another declared divergence

Choose one point from each axis. Neither coordinate determines the other.

A fixed prompt set does not settle the outer coordinate. GKD samples $x$ from a prompt collection $X$ , then can sample the response $y$ from the current student. The prompt is fixed; the completion trajectory is on-policy for the student at collection time.

boundary “At collection time” matters. Reusing trajectories after the student changes can make the behavior policy stale. A method name does not reveal freshness; an implementation must.

Classify before calculating

Take two recipes:

Student response + forward KL. The trajectory is on-policy; the local direction is forward.
Fixed teacher response + reverse KL. The local direction is reverse; the supplied trajectory is off-policy relative to the current student.

Those combinations are not contradictions. They are two cells in a product space. The next chapter writes the outer coordinate as an actual measure rather than an adjective.