Expected reward in matrix form,
DefinitionTPG_Bridgecausal-inferencemarkov-chainnonstationaritypolicy-gradienttreatment-effect
For an environment of TPG_Model and a sequence of treatment probabilities, Env.meanReward q t is the number
where , and . Time is -indexed.
It is the expected reward at time when the action at time is treatment with probability (a separate lemma proves this), written in the matrix form used throughout the source's proofs. It is affine in each with and does not depend on for .
Definition code
import Mathlib
import Definitions.Def_TPG_Model
import Definitions.Def_TPG_Markov
open MeasureTheory ProbabilityTheory
open scoped BigOperators ENNReal
namespace TPG
/-- Expected reward at time `t` when the action at time `s` is treatment with probability `q s`,
in matrix form: `ρᵀ P_{q_0}^0 ⋯ P_{q_{t-1}}^{t-1} r_{q_t}`. -/
noncomputable def Env.meanReward {S : Type} [Fintype S] [DecidableEq S] (E : Env S)
(q : ℕ → ℝ) (t : ℕ) : ℝ :=
dotProduct (E.stateVec q t) (E.rMix (q t))
end TPG
Source
Johari, Peng, Xing, Estimation of Treatment Effects Under Nonstationarity via the Truncated Policy Gradient Estimator, revised manuscript of October 2026 (revision of arXiv:2506.05308v3, https://arxiv.org/abs/2506.05308); numbering follows the revised manuscript. Auxiliary statement, not a numbered result of the paper: it is the matrix expression ρ^⊤ P_θ^{1→t} r_θ of Appendix G.1, p. 42 (proof of Proposition 1), for a time-varying sequence of treatment probabilities.