Information Theory

Information theory quantifies uncertainty and “surprise”. It explains why cross-entropy is the standard loss for classification and why KL divergence appears everywhere in generative modeling.

Entropy

Entropy measures the uncertainty of a distribution.

H(P) = - Σₓ P(x) log P(x)

Cross-Entropy

Cross-entropy measures how well a predicted distribution Q matches the true distribution P.

H(P, Q) = - Σₓ P(x) log Q(x)

KL Divergence

KL divergence measures how different Q is from P.

KL(P || Q) = Σₓ P(x) log (P(x) / Q(x))

Mutual Information

Mutual information measures shared information between variables.

I(X; Y) = KL(P(X,Y) || P(X)P(Y))