Local transition deviates by at most twice the ATV measure of entanglement

Theorem 8, first part (Chen and Peng, p. 40), $N$-agent form on the supremum row of Table 1.

Consider an $N$-agent MDP with joint transition $P^\pi_{1:N}$ and a policy $\pi$, with occupancy measure $\mu$ strictly positive and stationary. Fix an agent $i$, let $P^\pi_i$ be the local (marginalized) transition induced by $P^\pi_{1:N}$ and $\mu$ via Eq. (2), and let $P_i$ attain the measure of Markov entanglement of agent $i$ with respect to the unweighted agent-wise total variation distance.

Then the local transition deviates from that optimal independent approximation by at most twice the entanglement, uniformly over local state-action pairs:

The measure here is the supremum-based one, $\mathcal{E}_i = \inf_{P_i} \sup_{s,a} \tfrac12\sum_t |P_i^{\text{marg}}(t\mid s,a) - P_i(t\mid s_i,a_i)|$. This pairing matters: Table 1 (p. 20) matches the unweighted distances with $\|\cdot\|_\infty$ and the $\mu$-weighted ones with $\|\cdot\|_\mu$, and the uniform conclusion is false if the entanglement is measured in the $\mu$-weighted distance, since a rarely visited local state can contribute almost nothing to a $\mu$-average while its conditional deviation stays of constant order.

Strict positivity of $\mu$ forces the joint space to be nonempty, hence every local state-action pair has positive marginal — which is what licenses passing from the conditional average defining $P^\pi_i$ to a uniform bound.