Equation (12) — finite true value parameter and consistency
ProvedLeastSquaresTD.Absorbing.value_consistencyFor an absorbing finite Markov chain with rewards zero between absorbing states, take . Suppose the non-absorbing feature vectors are linearly independent, absorbing feature vectors vanish, and the feature dimension is the number of non-absorbing states. Then the expected-return series converges at every state, and there is a parameter representing its value function such that
This supplies the finite true parameter and the consistency relation used to identify the LS TD limit with it.
Formalization Note At , summability follows from absorption and the zero-reward condition; it is stated explicitly because a divergent real tsum would otherwise have a default value. The parameter is existential, not defined by the limiting LS TD matrix.
import Definitions.Def_LeastSquaresTD_Absorbing_Estimator
namespace LeastSquaresTD.Absorbing
variable {m : ℕ}
/-- Proof of Theorem 1, p. 44, Eq. (12), together with the paper's
claim that `θ*` is finite. The value function is the return series, so
summability at `γ = 1` remains part of this conclusion. -/
theorem value_consistency
{X : Type*} [Fintype X] [DecidableEq X] [Nonempty X]
(C : Chain X) (R : X → X → ℝ) (φ : X → Fin m → ℝ) (γ : ℝ)
(habs : C.IsAbsorbing) (hRabs : ∀ x y, C.P x x = 1 → C.P y y = 1 → R x y = 0)
(hm : m = Fintype.card C.Nonabsorbing)
(hli : LinearIndependent ℝ (fun x : C.Nonabsorbing => φ x.val))
(hφabs : ∀ x, C.P x x = 1 → φ x = 0)
(hγ0 : 0 ≤ γ) (hγ1 : γ ≤ 1) :
∃ θstar : Fin m → ℝ,
(∀ x, Summable (fun k : ℕ => γ ^ k *
(Matrix.mulVec (C.P ^ k) (C.rbar R)) x)) ∧
(∀ x, C.value R γ x = ∑ i, φ x i * θstar i) ∧
C.rbar R = Matrix.mulVec (1 - γ • C.P)
(Matrix.mulVec (Matrix.of φ) θstar) := by sorry
end LeastSquaresTD.Absorbing
Confirmed by the mission captain (proposal self-audit).
Confirmed by the moderator at approval.