Expectation of the reward-corrected estimator in matrix form
ProvedTPG.integral_rcEstcausal-inferencemarkov-chainnonstationaritypolicy-gradienttreatment-effect
Setting. is a finite state space, actions are , time is (the source's time ). The environment has transition matrices , initial distribution and reward laws on . For a sequence of treatment probabilities, and denote the law of the trajectory when , the actions are independent with , , and the rewards are conditionally independent with (Env.trajLaw T q); the Bernoulli policy is .
Let with and , and let .
Lemma. If and for all , then
where is the constant sequence with the entry at time replaced by and the entry at time replaced by .
The additional weight replaces the continuation reward by , which appears here as the entry at the reward time.
Preamble
import Mathlib import Definitions.Def_TPG_Model import Definitions.Def_TPG_Estimators import Definitions.Def_TPG_Markov import Definitions.Def_TPG_Bridge open MeasureTheory ProbabilityTheory open scoped BigOperators ENNReal
Formal statement
namespace TPG
theorem integral_rcEst {S : Type} [Fintype S] [DecidableEq S] [MeasurableSpace S] [MeasurableSingletonClass S]
(E : Env S) (T k : ℕ) (θ M : ℝ) (hθ0 : 0 < θ) (hθ1 : θ < 1) (hM : E.RewardBound M) :
∫ ω, rcEst θ T k ω ∂(E.expLaw T θ)
= (1 / (T : ℝ)) * ∑ u : Fin T,
((E.meanReward (Function.update (fun _ => θ) (u : ℕ) 1) (u : ℕ)
- E.meanReward (Function.update (fun _ => θ) (u : ℕ) 0) (u : ℕ))
+ ∑ t : Fin T,
if (u : ℕ) < (t : ℕ) ∧ (t : ℕ) ≤ (u : ℕ) + k then
(E.meanReward
(Function.update (Function.update (fun _ => θ) (u : ℕ) 1) (t : ℕ) (1 - θ)) (t : ℕ)
- E.meanReward
(Function.update (Function.update (fun _ => θ) (u : ℕ) 0) (t : ℕ) (1 - θ)) (t : ℕ))
else 0) := by sorry
end TPGSource
Johari, Peng, Xing, Estimation of Treatment Effects Under Nonstationarity via the Truncated Policy Gradient Estimator, revised manuscript of October 2026 (revision of arXiv:2506.05308v3, https://arxiv.org/abs/2506.05308); numbering follows the revised manuscript. Auxiliary statement, not a numbered result of the paper: it is eq. (20), p. 34 (Appendix E.2), E_θ[τ̂^RC_{θ,k}] = (1/T) Σ_t [μ_θ^t (r_1 − r_0) + A^t_{θ,k} r_{1−θ}], written with one-coordinate changes of the randomization sequence.