JAO equation (34): the reward collected in the planted gadget is at most $T - \mathbb{E}_{\mathrm{unif}}[N_p] + \varepsilon D' \mathbb{E}_a[N_\circ^*]$

Throughout, the MDP is the collapsed two-state gadget of JAO Figure 3, presented by its defining equations rather than through an auxiliary definition: state $0$ is $s_\circ$ with reward $0$, state $1$ is $s_p$ with reward $1$, the return probability is $p(s_\circ \mid s_p, b) = \delta$ for every action $b$, and the escape probability is $p(s_p \mid s_\circ, b) = \delta$ for every action other than the planted action $a$, for which it is $\delta + \varepsilon$. The gadget has diameter $D' = 1/\delta$. The reference MDP $M_0$ is the same gadget with no planting.

Statement. For every horizon $T$ and every policy $\pi$, writing $N_p$ for the number of rounds spent in $s_p$ (which, because the reward is the indicator of $s_p$, is exactly the total reward collected) and $N_\circ^*$ for the number of rounds in which the planted action $a$ is played in state $s_\circ$,

Here $\mathbb{E}_a$ is the expectation under the planted gadget and $\mathbb{E}_{\mathrm{unif}}$ that under the reference gadget $M_0$, both run against the same policy $\pi$ from $s_\circ$, and $\tfrac{\varepsilon}{\delta} = \varepsilon D'$.

This is equation (34) of JAO (p. 1583), in the form in which the rest of the argument consumes it. The proof has three steps. First, a one-step conditioning identity: summing $\mathbb{P}_a[s_t = s_p]$ over $t$ and splitting on the previous state gives

whence $\mathbb{E}_a[N_p] \le \mathbb{E}_a[N_\circ] + \varepsilon D' \mathbb{E}_a[N_\circ^*]$. Second, playing the planted action can only reduce the probability of remaining in $s_\circ$, so $\mathbb{E}_a[N_\circ] \le \mathbb{E}_{\mathrm{unif}}[N_\circ]$; this is a coupling between the two gadgets under the common policy. Third, $N_\circ + N_p = T$ identically, so $\mathbb{E}_{\mathrm{unif}}[N_\circ] = T - \mathbb{E}_{\mathrm{unif}}[N_p]$.