Equation (24.5): under a distribution P on a finite domain, the true log-loss risk of θ is D_RE[P‖P_θ] + H(P)
ProvedUnderstandingML.log_loss_risk_decompositionentropylog-lossrelative-entropy
Equation (24.5). Assuming that the data is distributed according to a distribution (not necessarily of the parametric form we employ), the true risk of a parameter becomes
Formally: on a finite domain, a probability mass function and positive.
Preamble
import Definitions.Def_UnderstandingML_Generative open MeasureTheory ProbabilityTheory open scoped InnerProductSpace
Formal statement
namespace UnderstandingML
/-- **Equation (24.5)** (p. 345). For data distributed according to a distribution `P` over a
finite domain, the true risk of the parameter `θ` under the log-loss is
`E_x[ℓ(θ, x)] = −∑ₓ P[x] log(P_θ[x]) = D_RE[P‖P_θ] + H(P)`. `P_θ` is positive. -/
theorem log_loss_risk_decomposition {Θ X : Type*} [Fintype X] (P : X → ℝ) (hP : IsPMF P)
(Pθ : Θ → X → ℝ) (θ : Θ) (hpos : ∀ x, 0 < Pθ θ x) :
∑ x, P x * logLoss Pθ θ x = relEntropy P (Pθ θ) + entropy P := by sorry
end UnderstandingML
Source
Shalev-Shwartz and Ben-David, Understanding Machine Learning: From Theory to Algorithms, Cambridge University Press 2014, doi:10.1017/CBO9781107298019, §24.1.2 p. 345, Equation (24.5)
Human review
Confirmed by the mission captain (proposal self-audit).
Confirmed by the moderator at approval.