Policy Detective
Print edition

An on-policy distillation case file

Follow the prefix, not the label.

A student sampled B, arrived at xB, and received teacher feedback there. A review note says the run is on-policy because it uses reverse KL. Your job is to test that reasoning against the trace.

Open the case file →Read the complete static edition

One deterministic toy · no model download · no training · no score · no claim of mastery

Metaphorical archive desk where one paper trail branches and a lens follows one route; technical evidence appears in the adjacent text.
DETECTIVE-FIG-001Metaphor only: the paper trail suggests provenance. It contains no policy value, equation, or research claim.

Briefing · four labels

Know what each record can answer

We will introduce notation only when the case needs it. For now, keep four nouns separate.

Policy
A rule that assigns probabilities to next-token choices.
Prefix
The prompt plus tokens generated so far—the state for the next prediction.
Behavior policy
The particular student version that actually sampled the stored token.
Divergence
A declared objective comparing two distributions; its direction does not reveal provenance.
DETECTIVE-FIG-002 · derived teaching decomposition

Six records, not one magical “OPD step”

  1. 01SampleStudent distribution + declared draw
  2. 02VisitAppend token; record child prefix
  3. 03EvaluateTeacher at that same prefix
  4. 04Local lossDeclared divergence; aligned events
  5. 05ReduceDeclared positions and length rule
  6. 06Update boundarySeparate optimizer step; omitted here
Arrows encode the trace order. Sampling, teacher feedback, the loss, and a later update are kept distinct. OPD-CL-004