01
Start with surprise
A forecast is a set of bets. Give each possible outcome a score before averaging anything.
One delivery, two possible outcomes
Imagine a delivery service. In this invented example, a delivery arrives on time with probability 75%, and late with probability 25%. Call on time A and late B.
P is our reference event mix: 75% A, 25% B. Q is a forecast we want to judge: it says 50% A, 50% B. A distribution is simply this complete list of probabilities. Each list adds to 100%.
| Outcome | P: reference | Q: forecast |
|---|---|---|
| A · on time | 0.75 = 75% | 0.50 = 50% |
| B · late | 0.25 = 25% | 0.50 = 50% |
Over 100 deliveries, that reference gives expected counts of 75 on-time and 25 late deliveries. An actual batch could differ. We will use these counts to make an average easy to see.
Q is too pessimistic about A and too optimistic about B. How much does that mismatch cost? We first need a scoring rule for one outcome.
The halving ladder
If you give an outcome a 100% chance, seeing it is unsurprising. If you give it a 1% chance, seeing it contradicts a confident forecast. We want a score that grows as the assigned probability shrinks.
Use this rule: every halving of probability adds one unit of surprise. Start at 100%, with score zero.
0 bits
1 bit
2 bits
3 bits
Here a bit is the unit of this surprise score. It is not a percentage point. Halving again adds another bit; it does not double the score.
The logarithm extends the ladder to probabilities between its rungs. To score 75%, ask: what exponent on ½ gives 0.75? The answer is about 0.415, because one half raised to 0.415 is about 0.75. Fractional scores are allowed.
The compact way to calculate this is below. r means the probability your chosen forecast assigned to the outcome that happened; “log₂” is the base-two logarithm. The minus sign makes this score positive for probabilities below one.
For example, log₂(½)=−1, so −log₂(½)=1. This is a mathematical score, not a claim about a person's emotions. It is useful for repeated events: for independent events, probabilities multiply, while these surprise scores add.
Definition and additive property: Stanford information theory notes, definitions 1–4.
Let the frequent event count more
P assigns A a surprise of 0.4150375 bits and B a surprise of 2 bits. Averaging those two numbers equally would pretend the events happened equally often.
Instead, imagine the expected 75 A outcomes each contributing 0.4150375, and the 25 B outcomes each contributing 2. Add all the scores, then divide by 100:
Equivalently, multiply by the fractions 0.75 and 0.25. That is a weighted average. The average surprise when P scores its own event mix is called entropy. The name comes after the operation; you do not need it to do the arithmetic.
Before continuing: Q gives both events a score of 1 bit. What is its average score over the same 75/25 mix?
Show the reasoning
0.75 × 1 + 0.25 × 1 = 1 bit. Every event scores 1, so any mix averages to 1.
P's own average is about 0.811 bits. Q's is 1 bit. The extra amount is the idea behind KL.