06
Can the reader diagnose an unfamiliar recipe?
Diagnose an unfamiliar recipe by answering the source and divergence questions separately.
OPD-XFER-A — Reverse loss, fixed prefix source
A learner uses reverse KL on a fixed teacher-generated xA prefix. Is the divergence reverse KL, and is the state sampling on-policy? Explain both axes separately.
Inspect the reasoning
Local axis: yes. The student distribution is the first KL argument and weights the log-ratio.
Source axis: no for this supplied occurrence. The prefix came from a fixed teacher-generated trajectory, not the current student. Another student draw could reach the same text; provenance belongs to the collection event.
OPD-XFER-B — Student prefix source, forward loss
A learner samples B from the student, queries the teacher at xB, and computes forward KL. Is this on-policy distillation? Explain why divergence direction does not change the state source.
Inspect the reasoning
Source axis: yes, provided the response was freshly sampled from the student behavior policy used for collection. xB is a student-visited prefix.
Local axis: forward KL compares teacher to student at that fixed prefix. Swapping the KL order changes the local value and derivative—not the already established provenance.
Carry a five-question rule
- What samples the response?
- Which prefix occurrence is evaluated?
- What teacher feedback is visible?
- What objective and reduction are formed?
- Which dependence reaches the gradient?
If those answers are explicit, “on-policy” becomes one coordinate again instead of a bag of assumptions.
The field-guide version
The source chooses the visited states. The local divergence compares distributions at a visited state. The gradient estimator decides which parameter dependence survives.
This guide explains a canonical detached GKD-style toy and several boundaries. It does not define every method called On-Policy Distillation, report a real-model benchmark, or establish a measured learner outcome.