Data-processing inequality for relative entropy (finite measures)
ProvedInformationTheory.klDiv_trim_le_of_isFiniteMeasureThe data-processing inequality for relative entropy, for finite measures: coarsening the -algebra cannot increase the divergence.
Let be -algebras on a set and let be finite measures on . Writing for the restriction of to the smaller -algebra,
Observing a coarser function of the data can only make two hypotheses harder to tell apart.
The corresponding statement for probability measures is standard. The extension to finite measures recorded here is what one needs as soon as the underlying space is split along an event: restricting a probability measure to a proper subset leaves a sub-probability measure, so any argument that localises to an event immediately leaves the probability-measure setting.
Formalization Note Mathlib's klDiv uses the mass-corrected definition , which is the form for which this statement is true for finite, and not merely probability, measures.
import Mathlib.InformationTheory.KullbackLeibler.Basic open MeasureTheory InformationTheory Set open scoped ENNReal
theorem InformationTheory.klDiv_trim_le_of_isFiniteMeasure {α : Type*}
{m m₀ : MeasurableSpace α} (hm : m ≤ m₀) (μ ν : @Measure α m₀)
[IsFiniteMeasure μ] [IsFiniteMeasure ν] :
@klDiv α m (μ.trim hm) (ν.trim hm) ≤ @klDiv α m₀ μ ν := by
sorry