03
Walk the KL landscape
Two probabilities choose a place on the floor. Extra average surprise sets its height.
Read the three directions
Our outcomes have not changed. A is on time; B is late. A two-outcome distribution is also called a Bernoulli distribution. Since its probabilities sum to 1, we only need the probability of A.
- p = P(A): move along the reference direction on the floor. P(B) is automatically 1 − p.
- q = Q(A): move along the forecast direction on the floor. Q(B) is automatically 1 − q.
- Height: KL(P‖Q), the extra average surprise in bits.
Every point on the surface is a different pair of probability tables. The line p=q lies on the floor: if both forecasts agree, there is no extra score. A ring shows the pair currently chosen below.
Predict: keep p at 75%. Should moving q from 50% toward 75% lower or raise the ring?
Change a forecast, follow its height
Surface height is in bits, capped at 4. Exact result below is uncapped.
| Event | P weight | Q | Extra score | Weighted contribution |
|---|---|---|---|---|
| A · on time | 0.75 | 0.50 | 0.584963 | 0.438722 |
| B · late | 0.25 | 0.50 | -1.000000 | -0.250000 |
1.000000 − 0.811278 = 0.188722 bits
Start with Equal: the ring sits in the valley. Try Moderately wrong, then Confidently wrong. The latter says A has only a 1% chance even though P says 75%: its extra surprise rises sharply.
The slice beneath the surface holds your current p fixed and varies only q. Its dashed line marks q=p, where the curve reaches zero. Top view helps you locate the pair on the floor; return to 3D surface to see height. Swap moves to a different scoring problem, so the height usually changes.
The mesh draws q from 0.01 to 0.99, with p from 0 to 1. Surface and slice heights are capped at 4 bits, with no log transformation. This keeps the valley visible; KL itself has no upper bound. The displayed calculations use the actual formula, not the mesh approximation. Finite endpoint markers can appear just outside the mesh.
Read the landscape without moving it
| Q(A) | Extra bits | What changes |
|---|---|---|
| 0.75 | 0 | Forecast matches: valley floor. |
| 0.50 | 0.188722 | Moderate mismatch: a little above the floor. |
| 0.01 | 4.175239 | Confidently wrong: above the drawn 4-bit ceiling. |
| 0 | ∞ | A can happen but Q calls it impossible. |
If both distributions put all probability on A, KL is zero. Their unused B row contributes zero even though both B probabilities are zero.
Zero is not “very unlikely”
Choose Impossible forecast. Q(A)=0 while P(A)=0.75. The forecast rules out an outcome that can happen. As q approaches zero, −log₂ q grows without limit; at exact zero the extra average is infinity.
Infinity is not the top of the drawn surface. It has no finite location, so the ring disappears and the numerical answer says ∞. We do not replace zero with a small positive number.
Now consider a different zero: P(x)=0. That outcome never occurs in the mix being averaged, so its contribution is defined as zero. We skip the term before evaluating a logarithm. This includes P(x)=Q(x)=0; we never try to divide 0 by 0.
Certain & equal puts all probability on A in both distributions. A contributes zero because they agree; the unused B row contributes zero too. Total: zero.
Try a new case mentally: P=[1,0], Q=[0.5,0.5]. What is KL? What changes when you swap?
Show the answer
Only A occurs under P. Its extra surprise is 1−0=1 bit. After swapping, the first mix gives B positive probability but the second forecast gives B zero. The swapped KL is infinite.
Boundary handling follows the extended-value KL formula and the zero-weight limit; see the boundary note in the appendix.