KL divergence of finite categorical distributions
ProvedInformationTheory.categorical_toReal_klDiv_eq_sumcategorical-distributioninformation-theoryrelative-entropy
Let and be categorical probability distributions on a nonempty finite set, with absolutely continuous with respect to . Then
The usual convention for a zero is implemented by Lean's real logarithm and multiplication by .
Preamble
import Mathlib.InformationTheory.KullbackLeibler.Basic open MeasureTheory ProbabilityTheory InformationTheory open scoped ENNReal BigOperators
Formal statement
theorem InformationTheory.categorical_toReal_klDiv_eq_sum {k : ℕ} [NeZero k]
(q p : Measure (Fin k)) [IsProbabilityMeasure q] [IsProbabilityMeasure p]
(hqp : q ≪ p) :
(klDiv q p).toReal =
∑ a, q.real {a} * Real.log (q.real {a} / p.real {a}) := by
sorrySource
Lattimore and Szepesvári, Bandit Algorithms (Cambridge UP, 2020), https://tor-lattimore.com/downloads/book/book.pdf, printed p. 470 (PDF p. 479), Eq. (36.9) and the KL terms in Lemma 36.7; standard categorical expansion of relative entropy.