§4, p. 10 — if A is self-weighted for M and all policies of M are proper, M′ has only proper policies
ProvedStableFA.DerivedMDP.derived_all_properLet be a finite nondiscounted Markov decision process in which every policy is proper, let be an averager that is self-weighted for , and let be the derived MDP of Theorem 4.1 with . Then:
- there is an integer such that, for every admissible (possibly nonstationary) policy of and every starting state ,
- every stationary policy of is proper.
The report proves this in the paragraph after the definition of self-weighted averagers. Part 1 is exactly the hypothesis of the stochastic shortest path theorem applied to , which is how the convergence of approximate value iteration follows.
Formalization Note is derivedModel M A 1. The page states the claim as " will have only proper policies"; part 2 is that wording, and part 1 is the uniform bound the page's argument establishes ("no matter what state we start from, we have a positive probability of reaching state 1 in steps"), written as Bertsekas' Assumption 7.2.1, the hypothesis of BertsekasDP.ssp_main_theorem. The hypotheses concern and only.
import Mathlib import Definitions.Def_BertsekasSSPModel import Definitions.Def_StableFA_DerivedMDP_Setting
namespace StableFA.DerivedMDP
theorem derived_all_proper {n : ℕ} {C : Type} [Fintype C]
(M : BertsekasSSPModel n C) (A : Averager n)
(hproper : ∀ μ : Fin n → C, (∀ i, μ i ∈ M.U i) → IsProper M μ)
(hself : SelfWeighted M A) :
(∃ m : ℕ, 0 < m ∧ ∀ π : ℕ → Fin n → C, BertsekasSSPAdmissible (derivedModel M A 1) π →
∀ i, BertsekasSSPSurvival (derivedModel M A 1) π m i < 1) ∧
∀ μ : Fin n → C, (∀ i, μ i ∈ M.U i) → IsProper (derivedModel M A 1) μ := by sorry
end StableFA.DerivedMDP
Confirmed by the mission captain (proposal self-audit).
Confirmed by the moderator at approval.