From forecasts to a student

Same table, a new source of probabilities

Imagine a teacher model and a student model looking at the same input. They assign probabilities to the same possible outputs. For a language model, an output here could be the next token.

Keep the two-outcome teaching table: teacher T=[0.75,0.25], student S=[0.5,0.5]. These remain invented numbers. We now call A and B two possible outputs rather than delivery outcomes; the arithmetic is unchanged.

Knowledge distillation can train the student to match the teacher's probability distribution, using soft targets rather than only the most likely answer. The teacher is a reference here; that does not make it infallible.

Soft-target distillation: Hinton, Vinyals & Dean, section 2. Temperature and mixtures with hard labels are outside this small example.

Say the direction explicitly

In this guide, forward KL means teacher first:

D_{\mathrm{KL}}(T\Vert S)=\sum_x T(x)\log_2\frac{T(x)}{S(x)}

The teacher supplies the weights. Our example gives 0.188722 bits. If the teacher is fixed, reducing this KL by changing S is equivalent to reducing cross-entropy, because the teacher's own entropy is constant.

Here, reverse KL means student first:

D_{\mathrm{KL}}(S\Vert T)=\sum_x S(x)\log_2\frac{S(x)}{T(x)}

The student supplies the weights. The same table gives 0.207519 bits. Changing the student now also changes those weights. Always read the written arguments; the words “forward” and “reverse” need a declared reference.

This is one comparison at one fixed input. On-policy distillation adds another question: which inputs or prefixes does the student visit during training? That question is separate from the direction of KL at an individual input.

Continue with the corrected OPD worked-example guide when you want to connect those two decisions.

One sentence you can reconstruct

KL(P‖Q) is the extra average surprise from using Q to score outcomes drawn from P, compared with using P itself.

Read it left to right: P sets the event mix, Q is the alternative scoring forecast. For each outcome, find the extra surprise, weight it by P, then add. You have now built the equation rather than memorized its shape.

A teacher gives one output a 10% chance. A student gives it exactly zero. Which direction is certainly infinite, and why?

Explain the boundary

Forward T‖S is infinite because the first distribution gives positive weight to an output the second calls impossible. That output contributes zero to reverse S‖T because S gives it zero weight. Reverse could still be infinite for a different output; its finiteness depends on the whole table.