Conditional mutual information $I(X;Y|Z)$ (Definition 10.6.1)

Definition 10.6.1 (Conditional mutual information). Let $X$, $Y$, and $Z$ be discrete random variables with joint distribution $p_{XYZ}$. The conditional mutual information is

where $H(Y|Z)$ is the conditional entropy (Definition 10.2.1) of $Y$ given $Z$ computed from the marginal $p_{YZ}(y,z)=\sum_x p_{XYZ}(x,y,z)$, and $H(Y|X,Z)$ is the conditional entropy of $Y$ given the pair $(X,Z)$ computed from $p_{XYZ}$ regrouped as the pair distribution of $\bigl(Y,(X,Z)\bigr)$. (Wilde also records the equivalent forms $H(X|Z)-H(X|Y,Z)$ and $H(X|Z)+H(Y|Z)-H(X,Y|Z)$.) Logarithms are base $2$.

The conditional mutual information measures the correlation between $X$ and $Y$ that remains once $Z$ is known; its non-negativity is the strong subadditivity of classical entropy (Theorem 10.6.1), and it drives the chain rule for mutual information and the data-processing inequality.

Formalization Note. WildeQIT.condMutualInfo p, for p : WildeQIT.FinDist (α × β × γ) (components $X,Y,Z$ in this order; note α × β × γ is α × (β × γ)), is condEntropy p.margYZ - condEntropy p.groupY_XZ, where margYZ is the $(Y,Z)$ marginal and groupY_XZ presents $p_{XYZ}$ as the pair distribution of $(Y,(X,Z))$, so that condEntropy (first component given second) yields $H(Y|Z)$ and $H(Y|X,Z)$ respectively. The definition uses the first of Wilde's three displayed forms; the other two are theorems.