Markov structure of the trajectory law: paths, head and tail, restarted environment, state distribution under time-varying randomization
DefinitionTPG_MarkovAuxiliary objects for working with the trajectory law of TPG_Model. Time is (the source's ).
-
Path embedding (
embedPath x z y): the trajectory with state path , action path and rewards . -
Head and tail (
headTraj m n,tailTraj m n): for a trajectory of length , its first time steps, and its last time steps re-indexed from . -
Reward law along a path (
Env.rewardLaw x z): the product measure on , i.e. the joint law of the rewards given the state and action paths. -
Restarted environment (
Env.shift m a): the environment whose kernel at its time is , whose initial distribution is the point mass at the state , and whose reward laws are unchanged. Its trajectory law is the law of the original process from time on, started at . -
State distribution under time-varying randomization (
Env.stateVec q t): the row vector
which is the distribution of when the action at time is treatment with probability .
These objects are used to state the mixture representation of the trajectory law, its Markov property, and the matrix formulas for expected rewards.
Formalization Note Env.shift and Env.stateVec need decidable equality on (for the point mass and the identity matrix).
import Mathlib
import Definitions.Def_TPG_Model
open MeasureTheory ProbabilityTheory
open scoped BigOperators ENNReal
namespace TPG
/-- The trajectory whose state path is `x`, whose action path is `z` and whose rewards are `y`. -/
def embedPath {S : Type} {T : ℕ} (x : Fin T → S) (z : Fin T → Bool) (y : Fin T → ℝ) :
Traj S T :=
fun t => (x t, z t, y t)
/-- The first `m` time steps of a trajectory of length `m + n`. -/
def headTraj {S : Type} (m n : ℕ) (ω : Traj S (m + n)) : Traj S m :=
fun i => ω (Fin.castAdd n i)
/-- The last `n` time steps of a trajectory of length `m + n`, re-indexed from `0`. -/
def tailTraj {S : Type} (m n : ℕ) (ω : Traj S (m + n)) : Traj S n :=
fun j => ω (Fin.natAdd m j)
section
variable {S : Type} [Fintype S]
/-- Joint law of the rewards along a fixed state path `x` and action path `z`:
independent coordinates with `Y_t ∼ R(· | x_t, z_t)`. -/
noncomputable def Env.rewardLaw (E : Env S) {T : ℕ} (x : Fin T → S) (z : Fin T → Bool) :
Measure (Fin T → ℝ) :=
Measure.pi (fun t => E.R (x t) (z t))
/-- The environment restarted at time `m` from the state `a`: the kernel used at its time `t`
is the original kernel of time `m + t`, the initial distribution is the point mass at `a`, and
the reward distributions are unchanged. -/
noncomputable def Env.shift [DecidableEq S] (E : Env S) (m : ℕ) (a : S) : Env S where
P := fun t => E.P (m + t)
P_nonneg := fun t => E.P_nonneg (m + t)
P_row_sum := fun t => E.P_row_sum (m + t)
ρ := fun x => if x = a then 1 else 0
ρ_nonneg := fun x => by split_ifs <;> norm_num
ρ_sum := by simp
R := E.R
R_prob := E.R_prob
/-- Row distribution of the state at time `t` when the action at time `s` is treatment with
probability `q s`: `ρᵀ P_{q_0}^0 P_{q_1}^1 ⋯ P_{q_{t-1}}^{t-1}`. -/
noncomputable def Env.stateVec [DecidableEq S] (E : Env S) (q : ℕ → ℝ) (t : ℕ) : S → ℝ :=
Matrix.vecMul E.ρ ((List.range t).map (fun s => E.PMix (q s) s)).prod
end
end TPG