One delivery, two possible outcomes

Imagine a delivery service. In this invented example, a delivery arrives on time with probability 75%, and late with probability 25%. Call on time A and late B.

P is our reference event mix: 75% A, 25% B. Q is a forecast we want to judge: it says 50% A, 50% B. A distribution is simply this complete list of probabilities. Each list adds to 100%.

Over 100 deliveries, that reference gives expected counts of 75 on-time and 25 late deliveries. An actual batch could differ. We will use these counts to make an average easy to see.

Q is too pessimistic about A and too optimistic about B. How much does that mismatch cost? We first need a scoring rule for one outcome.

The halving ladder

If you give an outcome a 100% chance, seeing it is unsurprising. If you give it a 1% chance, seeing it contradicts a confident forecast. We want a score that grows as the assigned probability shrinks.

Use this rule: every halving of probability adds one unit of surprise. Start at 100%, with score zero.

Here a bit is the unit of this surprise score. It is not a percentage point. Halving again adds another bit; it does not double the score.

The logarithm extends the ladder to probabilities between its rungs. To score 75%, ask: what exponent on ½ gives 0.75? The answer is about 0.415, because one half raised to 0.415 is about 0.75. Fractional scores are allowed.

The compact way to calculate this is below. r means the probability your chosen forecast assigned to the outcome that happened; “log₂” is the base-two logarithm. The minus sign makes this score positive for probabilities below one.

For example, log₂(½)=−1, so −log₂(½)=1. This is a mathematical score, not a claim about a person's emotions. It is useful for repeated events: for independent events, probabilities multiply, while these surprise scores add.

Definition and additive property: Stanford information theory notes, definitions 1–4.

Let the frequent event count more

P assigns A a surprise of 0.4150375 bits and B a surprise of 2 bits. Averaging those two numbers equally would pretend the events happened equally often.

Instead, imagine the expected 75 A outcomes each contributing 0.4150375, and the 25 B outcomes each contributing 2. Add all the scores, then divide by 100:

Equivalently, multiply by the fractions 0.75 and 0.25. That is a weighted average. The average surprise when P scores its own event mix is called entropy. The name comes after the operation; you do not need it to do the arithmetic.

P's own average is about 0.811 bits. Q's is 1 bit. The extra amount is the idea behind KL.

Two rows, one extra average

We still use P = [0.75, 0.25] and Q = [0.50, 0.50]. Both scoring rules see the same event mix from P. Under Q the average is 1 bit; under P it is 0.811278 bits.

KL divergence is this extra average surprise. Here it is about 0.189 bits per delivery. It is neither an 18.9% error rate nor the surprise of every individual delivery.

You can also subtract first, one outcome at a time:

• A: Q scores it 1 instead of 0.4150375. Extra = 0.5849625. It has P weight 0.75, giving +0.4387219.
• B: Q scores it 1 instead of 2. Extra = −1. It has P weight 0.25, giving −0.25.

Add the weighted contributions: 0.4387219 − 0.25 = 0.1887219 bits. Q is less surprised by B than P is, so B's contribution is negative. A happens more often in this reference mix, and its positive contribution is larger.

Predict what will happen when you swap the two distributions. Then use the calculator. Reset returns to the delivery example.

The difference-of-averages identity and nonnegative total follow from Stanford information theory notes, definitions 1–4, equations 14–23. Numbers here are derived from our invented table.

Now give the arithmetic a short name

Write D with a KL subscript for “KL divergence.” Write P first to say whose event mix we average over, and Q second to say which alternative forecast we use. The two vertical bars separate those roles.

x

One possible outcome: A, then B in our example.

P(x)

The probability of that outcome in the reference mix. This is also its averaging weight.

Q(x)

The probability the alternative forecast gives the same outcome.

log₂(P(x)/Q(x))

The extra surprise for that outcome. Subtracting −log₂ P(x) from −log₂ Q(x) gives this log ratio.

Σ (capital sigma)

“Add the following expression for every outcome.” Here there are only two terms.

The formula is a compressed set of instructions: find the extra score, multiply by how often the outcome occurs under P, and add.

The first argument is not decoration

Swap P and Q. You now average over a 50/50 mix and score it using the 75/25 forecast. The new baseline is the 50/50 mix's own surprise.

0.207519 is different from 0.188722. The event mix changed as well as the scoring rule. Because swapping can change the answer, KL is not an ordinary distance.

For valid probability distributions the total KL is nonnegative, and it is zero exactly when they match. Individual rows can still be negative, as we saw. You cannot beat the reference's own forecast on average under that reference using this scoring rule.

Nonnegativity and equality: Stanford information theory notes, definitions 1–4, p.3. The swap is a direct calculation, not a new dataset.

Bits and nats measure the same mismatch

Our ladder uses log₂ and reports bits. Using the natural logarithm, written ln, reports nats. The natural log uses the mathematical base e, about 2.718, instead of 2. You do not need a new scoring idea: only the scale changes.

Multiply bits by about 0.693147 to get nats. Our 0.1887219 bits becomes 0.1308120 nats. The unit switch changes the numbers, never which forecast is better. The landscape in the next chapter keeps its height axis in bits.

Units: Toronto CSCC11 notes, section 8.1. Conversion is the change-of-base identity.

Read the three directions

Our outcomes have not changed. A is on time; B is late. A two-outcome distribution is also called a Bernoulli distribution. Since its probabilities sum to 1, we only need the probability of A.

• p = P(A): move along the reference direction on the floor. P(B) is automatically 1 − p.
• q = Q(A): move along the forecast direction on the floor. Q(B) is automatically 1 − q.
• Height: KL(P‖Q), the extra average surprise in bits.

Every point on the surface is a different pair of probability tables. The line p=q lies on the floor: if both forecasts agree, there is no extra score. A ring shows the pair currently chosen below.

Predict: keep p at 75%. Should moving q from 50% toward 75% lower or raise the ring?

Change a forecast, follow its height

Start with Equal: the ring sits in the valley. Try Moderately wrong, then Confidently wrong. The latter says A has only a 1% chance even though P says 75%: its extra surprise rises sharply.

The slice beneath the surface holds your current p fixed and varies only q. Its dashed line marks q=p, where the curve reaches zero. Top view helps you locate the pair on the floor; return to 3D surface to see height. Swap moves to a different scoring problem, so the height usually changes.

The mesh draws q from 0.01 to 0.99, with p from 0 to 1. Surface and slice heights are capped at 4 bits, with no log transformation. This keeps the valley visible; KL itself has no upper bound. The displayed calculations use the actual formula, not the mesh approximation. Finite endpoint markers can appear just outside the mesh.

Zero is not “very unlikely”

Choose Impossible forecast. Q(A)=0 while P(A)=0.75. The forecast rules out an outcome that can happen. As q approaches zero, −log₂ q grows without limit; at exact zero the extra average is infinity.

Infinity is not the top of the drawn surface. It has no finite location, so the ring disappears and the numerical answer says ∞. We do not replace zero with a small positive number.

Now consider a different zero: P(x)=0. That outcome never occurs in the mix being averaged, so its contribution is defined as zero. We skip the term before evaluating a logarithm. This includes P(x)=Q(x)=0; we never try to divide 0 by 0.

Certain & equal puts all probability on A in both distributions. A contributes zero because they agree; the unused B row contributes zero too. Total: zero.

Boundary handling follows the extended-value KL formula and the zero-weight limit; see the boundary note in the appendix.

Same table, a new source of probabilities

Imagine a teacher model and a student model looking at the same input. They assign probabilities to the same possible outputs. For a language model, an output here could be the next token.

Keep the two-outcome teaching table: teacher T=[0.75,0.25], student S=[0.5,0.5]. These remain invented numbers. We now call A and B two possible outputs rather than delivery outcomes; the arithmetic is unchanged.

Knowledge distillation can train the student to match the teacher's probability distribution, using soft targets rather than only the most likely answer. The teacher is a reference here; that does not make it infallible.

Soft-target distillation: Hinton, Vinyals & Dean, section 2. Temperature and mixtures with hard labels are outside this small example.

Say the direction explicitly

In this guide, forward KL means teacher first:

The teacher supplies the weights. Our example gives 0.188722 bits. If the teacher is fixed, reducing this KL by changing S is equivalent to reducing cross-entropy, because the teacher's own entropy is constant.

Here, reverse KL means student first:

The student supplies the weights. The same table gives 0.207519 bits. Changing the student now also changes those weights. Always read the written arguments; the words “forward” and “reverse” need a declared reference.

This is one comparison at one fixed input. On-policy distillation adds another question: which inputs or prefixes does the student visit during training? That question is separate from the direction of KL at an individual input.

Continue with the corrected OPD worked-example guide when you want to connect those two decisions.

One sentence you can reconstruct

KL(P‖Q) is the extra average surprise from using Q to score outcomes drawn from P, compared with using P itself.

Read it left to right: P sets the event mix, Q is the alternative scoring forecast. For each outcome, find the extra surprise, weight it by P, then add. You have now built the equation rather than memorized its shape.