KL of a density measure equals the entropy of the density
ProvedEntFunctional.toReal_klDiv_withDensity_eq_integral_mul_logconcentration-inequalitiesentropy-methodinformation-theoryprobability
KL of a density measure equals the entropy of the density. For a probability measure and a measurable nonnegative density with , set (a probability measure with ). Then the Kullback–Leibler divergence of against equals the entropy functional . This is the bridge identity connecting Mathlib's InformationTheory.klDiv to the entropy functional of the entropy method: combined with the mass-normalization it computes as a relative entropy, making KL tensorization (Han's inequality / sub-additivity, BLM Theorem 4.10) usable at the entropy-functional level.
Preamble
import Mathlib.InformationTheory.KullbackLeibler.Basic import Mathlib.MeasureTheory.Measure.Decomposition.RadonNikodym import Mathlib.MeasureTheory.Integral.Bochner.ContinuousLinearMap import Mathlib.Analysis.SpecialFunctions.Log.NegMulLog open Real MeasureTheory open scoped ENNReal NNReal
Formal statement
theorem EntFunctional.toReal_klDiv_withDensity_eq_integral_mul_log
{α : Type*} {mα : MeasurableSpace α} {μ : Measure α}
[IsProbabilityMeasure μ] {g : α → ℝ}
(hg_meas : Measurable g) (hg_nonneg : ∀ x, 0 ≤ g x)
(hg_int : Integrable g μ) (hg_mass : ∫ x, g x ∂μ = 1) :
(InformationTheory.klDiv (μ.withDensity (fun x ↦ ENNReal.ofReal (g x))) μ).toReal
= ∫ x, g x * log (g x) ∂μ := by sorrySource
Boucheron, Lugosi, Massart, *Concentration Inequalities*, OUP 2013, §4.1 (Theorem 4.10 / Theorem 4.13: sub-additivity and the variational/duality characterization of entropy).