06

Can the reader diagnose an unfamiliar recipe?

Diagnose an unfamiliar recipe by answering the source and divergence questions separately.

OPD-XFER-A — Reverse loss, fixed prefix source

A learner uses reverse KL on a fixed teacher-generated xA prefix. Is the divergence reverse KL, and is the state sampling on-policy? Explain both axes separately.

Inspect the reasoning

Local axis: yes. The student distribution is the first KL argument and weights the log-ratio.

Source axis: no for this supplied occurrence. The prefix came from a fixed teacher-generated trajectory, not the current student. Another student draw could reach the same text; provenance belongs to the collection event.

OPD-XFER-B — Student prefix source, forward loss

A learner samples B from the student, queries the teacher at xB, and computes forward KL. Is this on-policy distillation? Explain why divergence direction does not change the state source.

Inspect the reasoning

Source axis: yes, provided the response was freshly sampled from the student behavior policy used for collection. xB is a student-visited prefix.

Local axis: forward KL compares teacher to student at that fixed prefix. Swapping the KL order changes the local value and derivative—not the already established provenance.

Carry a five-question rule

  1. What samples the response?
  2. Which prefix occurrence is evaluated?
  3. What teacher feedback is visible?
  4. What objective and reduction are formed?
  5. Which dependence reaches the gradient?

If those answers are explicit, “on-policy” becomes one coordinate again instead of a bag of assumptions.

The field-guide version

The source chooses the visited states. The local divergence compares distributions at a visited state. The gradient estimator decides which parameter dependence survives.

This guide explains a canonical detached GKD-style toy and several boundaries. It does not define every method called On-Policy Distillation, report a real-model benchmark, or establish a measured learner outcome.