06
Carry the factorization with you
Two changed cases, then a five-question inspection rule for the next paper or implementation.
OPD-XFER-A — Reverse loss, fixed state source
A learner uses reverse KL on a fixed teacher-generated xA prefix. Is the divergence reverse KL, and is the state sampling on-policy? Explain both axes separately.
Inspect the reasoning
Local axis: yes, the formula is reverse KL because the student distribution is the first argument and weights the log-ratio.
Source axis: no for this supplied occurrence. The xA prefix came from a fixed teacher-generated trajectory, not the current student. The same textual prefix could be reached on-policy in another collection event; provenance belongs to the event.
OPD-XFER-B — Student state source, forward loss
A learner samples B from the student, queries the teacher at xB, and computes forward KL. Is this on-policy distillation? Explain why divergence direction does not change the state source.
Inspect the reasoning
Source axis: yes, provided the response was sampled from the current behavior policy at collection. xB is a student-visited prefix.
Local axis: forward KL compares teacher to student at that fixed prefix. Changing argument order changes the loss and gradient—not the already established provenance.
Five questions for any claimed OPD recipe
- What samples the response?Fixed data, teacher, current student, a mixture, or a stale behavior policy?
- Which occurrence is evaluated?Record prompt, prefix, and how that occurrence entered the batch.
- What feedback is visible?Full vocabulary, top-k, one sampled token, scalar reward, or something else?
- What objective is formed?Write its arguments and reduction instead of relying on “KL” or “distillation.”
- Which dependence reaches the gradient?Detached collection, score-function term, importance weighting, or another estimator?
If those five answers are explicit, “on-policy” becomes one coordinate again rather than a bag of assumptions.
The measure beneath the loss
The outer measure chooses the states. The inner divergence compares distributions at a state. The gradient estimator decides which parameter dependence survives. Keep those three lines separate and the surrounding terminology becomes much easier to audit.
unproven hypothesis This mathematical form was designed to help a mathematically inclined reader resist one specific conflation. No learner study was run, and completing the transfer prompts does not demonstrate mastery or establish that this variant teaches better than another.
Open notation, evidence, and limitations → Complete script-free reading edition