Markov property of the trajectory law in product form
ProvedTPG.integral_head_mul_tailSetting. is a finite state space, actions are , time is (the source's time ). The environment has transition matrices , initial distribution and reward laws on . For a sequence of treatment probabilities, and denote the law of the trajectory when , the actions are independent with , , and the rewards are conditionally independent with (Env.trajLaw T q); the Bernoulli policy is .
Consider the horizon . The head of a trajectory is its first time steps , and the tail is its last steps , re-indexed from . For a state , let denote expectation for the process restarted at time from the state : kernels , initial state , treatment probabilities , the same reward laws.
Lemma. Let for all , and let (a function of a trajectory of length ) and (a function of a trajectory of length ) be bounded and measurable. Then
where the first expectation on the right is taken for the horizon .
In words: given the head, the state at time is drawn from the row of , and given that state the tail is an independent copy of the restarted process. Covariance decay, strong mixing and the martingale arguments of the mission all rest on this identity.
import Mathlib import Definitions.Def_TPG_Model import Definitions.Def_TPG_Markov open MeasureTheory ProbabilityTheory open scoped BigOperators ENNReal
namespace TPG
theorem integral_head_mul_tail {S : Type} [Fintype S] [DecidableEq S] [MeasurableSpace S] [MeasurableSingletonClass S]
(E : Env S) (m n : ℕ) (q : ℕ → ℝ) (hq : ∀ t : ℕ, 0 ≤ q t ∧ q t ≤ 1)
(F : Traj S (m + 1) → ℝ) (G : Traj S n → ℝ) (hF : Measurable F) (hG : Measurable G)
(CF CG : ℝ) (hFb : ∀ ω, |F ω| ≤ CF) (hGb : ∀ ω, |G ω| ≤ CG) :
∫ ω, F (headTraj (m + 1) n ω) * G (tailTraj (m + 1) n ω) ∂(E.trajLaw (m + 1 + n) q)
= ∑ a : S,
(∫ ω, F ω * E.P m (Z (Fin.last m) ω) (X (Fin.last m) ω) a ∂(E.trajLaw (m + 1) q))
* ∫ ω, G ω ∂((E.shift (m + 1) a).trajLaw n (fun t => q (m + 1 + t))) := by sorry
end TPG