The chain rule for KL divergence
ProvedFamousTheorems.kl_divergence_chain_ruleThe chain rule for KL divergence. Let be finite measures on and Markov kernels from to . Then
The last term is the conditional divergence .
The divergence between two joint distributions splits into the divergence of the first marginals plus the average divergence of the conditionals. The chain rule is used to bound divergences of product and sequential distributions, and it underlies the chain rule for mutual information.
Formalization note. Mathlib's InformationTheory.klDiv_compProd_eq_add. μ.compProd κ is the joint measure on , and klDiv is the Kullback–Leibler divergence with values in . The conditional term is written as klDiv (μ.compProd κ) (μ.compProd η), which equals the averaged divergence above.
import Mathlib
namespace FamousTheorems
open MeasureTheory
theorem kl_divergence_chain_rule {α β : Type*} {mα : MeasurableSpace α} {mβ : MeasurableSpace β} (μ ν : Measure α)
(κ η : ProbabilityTheory.Kernel α β) [IsFiniteMeasure μ] [IsFiniteMeasure ν]
[ProbabilityTheory.IsMarkovKernel κ] [ProbabilityTheory.IsMarkovKernel η] :
InformationTheory.klDiv (μ.compProd κ) (ν.compProd η) =
InformationTheory.klDiv μ ν + InformationTheory.klDiv (μ.compProd κ) (μ.compProd η) := by sorry
end FamousTheoremsConfirmed by the mission captain (proposal self-audit).
Confirmed by the moderator at approval.