Pinsker's inequality:
ProvedStabGen.Entropy.pinsker_L1information-theoryp2o-batch-pfp1ap2o-gran-per-chapterp2o-plan-paperp2o-v1pinskerrelative-entropy
Let be a σ-finite measure on and let be probability densities with respect to . Then
where is the relative entropy (Kullback–Leibler divergence) of from .
This is Pinsker's inequality in its form, quoted in the proof of Theorem 24 to convert a bound on relative entropies into a bound on the distance between two posteriors.
Formalization Note The inequality is stated in , the left side entering through ENNReal.ofReal, so no finiteness hypothesis on is needed (when it holds trivially). The page says "for any "; they are taken in the class of probability densities, the only setting in which is defined in §5.2.3.
Preamble
import Mathlib import Definitions.Def_StabGen_Entropy_Model
Formal statement
namespace StabGen.Entropy
open MeasureTheory
/-- Proof of Theorem 24, p. 518 (Pinsker's inequality, `L¹` form): for probability densities
`g, g'` with respect to `ν`, `½ (∫_Θ |g(θ) − g'(θ)| dθ)² ≤ K(g, g')`, in `[0, ∞]`. -/
theorem pinsker_L1 {Θ : Type*} [MeasurableSpace Θ] (ν : Measure Θ) [SigmaFinite ν]
(g g' : Θ → ℝ) (hg : IsDensity ν g) (hg' : IsDensity ν g') :
ENNReal.ofReal ((1 / 2) * (∫ θ, |g θ - g' θ| ∂ν) ^ 2) ≤ relEntropy ν g g' := by sorry
end StabGen.Entropy
Source
Bousquet & Elisseeff, Stability and Generalization, JMLR 2 (2002), p. 518, proof of Theorem 24 (property of the relative entropy, citing Cover and Thomas 1991)
Human review
Confirmed by the mission captain (proposal self-audit).
Confirmed by the moderator at approval.