The extra amount is KL · KL field guide

Two rows, one extra average

We still use P = [0.75, 0.25] and Q = [0.50, 0.50]. Both scoring rules see the same event mix from P. Under Q the average is 1 bit; under P it is 0.811278 bits.

1-0.8112781245=0.1887218755\ \text{bits}

KL divergence is this extra average surprise. Here it is about 0.189 bits per delivery. It is neither an 18.9% error rate nor the surprise of every individual delivery.

You can also subtract first, one outcome at a time:

A: Q scores it 1 instead of 0.4150375. Extra = 0.5849625. It has P weight 0.75, giving +0.4387219.
B: Q scores it 1 instead of 2. Extra = −1. It has P weight 0.25, giving −0.25.

Add the weighted contributions: 0.4387219 − 0.25 = 0.1887219 bits. Q is less surprised by B than P is, so B's contribution is negative. A happens more often in this reference mix, and its positive contribution is larger.

Predict what will happen when you swap the two distributions. Then use the calculator. Reset returns to the delivery example.

Default: reference 75% on time; forecast 50% on time.

P · reference: A 75% · B 25%

Q · forecast: A 50% · B 50%

Solid length = A (on time). Remaining length = B (late).

Each row uses its P probability as the weight. Scores in bits; decimals rounded.
Event	P weight	Q	Extra score	Weighted contribution
A · on time	0.75	0.50	0.584963	0.438722
B · late	0.25	0.50	-1.000000	-0.250000

1.000000 − 0.811278 = 0.188722 bits

Average using Q − average using P, both over the P event mix.

The difference-of-averages identity and nonnegative total follow from Stanford information theory notes, definitions 1–4, equations 14–23. Numbers here are derived from our invented table.

Now give the arithmetic a short name

Write D with a KL subscript for “KL divergence.” Write P first to say whose event mix we average over, and Q second to say which alternative forecast we use. The two vertical bars separate those roles.

D_{\mathrm{KL}}(P\Vert Q)=\sum_x P(x)\log_2\frac{P(x)}{Q(x)}

x: One possible outcome: A, then B in our example.
P(x): The probability of that outcome in the reference mix. This is also its averaging weight.
Q(x): The probability the alternative forecast gives the same outcome.
log₂(P(x)/Q(x)): The extra surprise for that outcome. Subtracting −log₂ P(x) from −log₂ Q(x) gives this log ratio.
Σ (capital sigma): “Add the following expression for every outcome.” Here there are only two terms.

The formula is a compressed set of instructions: find the extra score, multiply by how often the outcome occurs under P, and add.

Optional: connect the two average names

H(P) is entropy: P's own average surprise. H(P,Q) is cross-entropy: Q's surprise averaged over P. For finite discrete distributions with Q positive wherever P is positive:

D_{\mathrm{KL}}(P\Vert Q)=H(P,Q)-H(P)

The log identity log a − log b = log(a/b) combines the two surprises into the ratio above.

The first argument is not decoration

Swap P and Q. You now average over a 50/50 mix and score it using the 75/25 forecast. The new baseline is the 50/50 mix's own surprise.

D_{\mathrm{KL}}(Q\Vert P)=0.5\log_2\frac{0.5}{0.75}+0.5\log_2\frac{0.5}{0.25}

\approx-0.292481+0.5=0.207519\ \text{bits}

0.207519 is different from 0.188722. The event mix changed as well as the scoring rule. Because swapping can change the answer, KL is not an ordinary distance.

For valid probability distributions the total KL is nonnegative, and it is zero exactly when they match. Individual rows can still be negative, as we saw. You cannot beat the reference's own forecast on average under that reference using this scoring rule.

Nonnegativity and equality: Stanford information theory notes, definitions 1–4, p.3. The swap is a direct calculation, not a new dataset.

Bits and nats measure the same mismatch

Our ladder uses log₂ and reports bits. Using the natural logarithm, written ln, reports nats. The natural log uses the mathematical base e, about 2.718, instead of 2. You do not need a new scoring idea: only the scale changes.

D_{\text{nats}}=D_{\text{bits}}\ln 2

Multiply bits by about 0.693147 to get nats. Our 0.1887219 bits becomes 0.1308120 nats. The unit switch changes the numbers, never which forecast is better. The landscape in the next chapter keeps its height axis in bits.

Units: Toronto CSCC11 notes, section 8.1. Conversion is the change-of-base identity.