Theorem 10.7.1 (Non-Negativity of Relative Entropy): for subnormalized , with equality iff
ProvedWildeQIT.ZhangAudenaert.relEntropy_nonnegentropyinformation-theoryp2o-gran-per-chapterp2o-plan-bookp2o-v1wilde-qit
Let be a probability distribution on a finite alphabet , and let be a function with (not necessarily a probability distribution). Let
be the relative entropy (Definition 10.5.2, in bits). Then
and if and only if (as functions on ).
This is the inequality from which the maximal value of entropy, Theorem 10.2.1, Theorem 10.4.1 and strong subadditivity follow; allowing subnormalized is what the proof of the data-processing inequality (Theorem 10.7.2) needs.
Formalization Note is the published extended-real relative entropy (WildeQIT.relEntropy), so the value is represented exactly.
Differences from the source: none.
Preamble
import Mathlib import Definitions.Def_WildeQIT_relEntropy
Formal statement
namespace WildeQIT.ZhangAudenaert
/-- **Theorem 10.7.1 (Non-Negativity of Relative Entropy).**
Wilde, *From Classical to Quantum Shannon Theory*, arXiv:1106.1445v8, Theorem 10.7.1, (10.44),
p. 299: "Let `p(x)` be a probability distribution over the alphabet `𝒳` and let `q : 𝒳 → [0, 1]`
be a function such that `∑_x q(x) ≤ 1`. Then the relative entropy `D(p‖q)` is non-negative:
`D(p‖q) ≥ 0`, and `D(p‖q) = 0` if and only if `p = q`."
**Formalization Note.** `q` is a real function on the finite alphabet `α` with values in
`[0, 1]` (it need not sum to one); `D(p‖q) = relEntropy p q` is the published extended-real
relative entropy in bits, `+∞` when `supp p ⊄ supp q` (Definition 10.5.2). "`p = q`" is
equality of the functions `p.prob` and `q`. -/
theorem relEntropy_nonneg {α : Type} [Fintype α] (p : FinDist α) (q : α → ℝ)
(hq : ∀ x, 0 ≤ q x ∧ q x ≤ 1) (hsum : ∑ x, q x ≤ 1) :
0 ≤ relEntropy p q ∧ (relEntropy p q = 0 ↔ p.prob = q) := by sorry
end WildeQIT.ZhangAudenaert
Source
Wilde, From Classical to Quantum Shannon Theory, arXiv:1106.1445v8, Theorem 10.7.1, eq. (10.44), p. 299
Human review
Confirmed by the mission captain (proposal self-audit).
Confirmed by the moderator at approval.