KL divergence
A four-chapter field guide
Glossary
Contents ☰

Glossary & sources

A small vocabulary

Probability distribution
A complete list of outcome probabilities, adding to one.
P and Q
In KL(P‖Q), P supplies the averaging weights; Q is the alternative forecast. P need not be the true world.
Surprise
The score −log of the probability assigned to an outcome. Smaller probability gives a larger score.
Logarithm
The exponent needed to reach a number from a chosen base. The halving ladder is −log₂ probability.
Entropy
Average surprise when a distribution scores its own outcomes.
Cross-entropy
Average surprise when Q scores outcomes averaged under P.
KL divergence
Cross-entropy minus the reference entropy; extra average surprise.
Bit and nat
Units from base-two and natural logarithms respectively; one bit equals ln(2) nats.
Bernoulli
A two-outcome distribution: probabilities p and 1−p.
Forward / reverse KL
Here forward is T‖S and reverse is S‖T. State the reference convention explicitly.

Sources

  1. Stanford: Entropy, Relative Entropy, and Mutual Information

    Tsachy Weissman, EE376A lecture 3 (2018): definitions 1–4, equations 14–23, pp.1–3. Surprise, averages, KL and nonnegativity.

  2. Toronto: CSCC11 Information Theory

    Aaron Hertzmann and David Fleet (2023), sections 8.1, 8.2 and 8.4: units, zero-weight limit and cross-entropy.

  3. Distilling the Knowledge in a Neural Network

    Geoffrey Hinton, Oriol Vinyals and Jeff Dean (2015), section 2: teacher soft targets.

Boundary and example notes

All probability tables are invented teaching inputs. Calculations use normalized, finite, discrete distributions. P-zero rows are omitted from the expectation before taking logarithms. For positive P, decreasing Q to zero sends the log ratio to infinity. Both-zero rows contribute zero by convention, not by evaluating 0/0.

The landscape is a three-coordinate mathematical mesh projected into SVG. It needs no WebGL. Its finite mesh excludes q=0 and q=1; its height is capped at four bits. Exact labels use the formula directly and preserve infinity. No probability smoothing is applied.

This guide does not model sampling noise, continuous densities, gradient updates or a full distillation pipeline. Implementation checks do not establish improved comprehension.

← Contents