Display (40), p. 21 — the gradient estimator is bounded: ‖h(θ; τ) − h(θ; τ′)‖ ≤ 2L_r/(1−γ) = 2L_q
ProvedStructEstMDP.FiniteTime.display_40p2o-batch-pfp2ap2o-gran-per-chapterp2o-plan-paperp2o-v1reinforcement-learning
Let and let the reward satisfy Assumption 3 with constant . For a trajectory let . Then for every and all trajectories ,
Applied to and this is the bound on the stochastic gradient, which controls the drift of the reward parameter.
Preamble
import Mathlib import Definitions.Def_FoundationsML_ReinforcementLearning_IsTransitionKernel import Definitions.Def_FoundationsML_ReinforcementLearning_IsPolicy import Definitions.Def_StructEstMDP_FiniteTime_Setting open FoundationsML.ReinforcementLearning MeasureTheory
Formal statement
namespace StructEstMDP.FiniteTime
/-- Display (40), p. 21: under Assumption 3 the gradient estimator is bounded,
`‖h(θ; τ) − h(θ; τ')‖ ≤ 2L_r ∑_{t≥0} γᵗ = 2L_r/(1−γ) = 2L_q`, for every `θ` and all
trajectories `τ, τ'`. -/
theorem display_40 {S A : Type*} (γ : ℝ) (hγ0 : 0 ≤ γ) (hγ1 : γ < 1)
{p : ℕ} (r : S → A → EuclideanSpace ℝ (Fin p) → ℝ) (Lr Lg : ℝ)
(h3 : Assumption3 r Lr Lg)
(θ : EuclideanSpace ℝ (Fin p)) (τ τ' : Traj S A) :
‖hGrad γ r θ τ - hGrad γ r θ τ'‖ ≤ 2 * (Lr / (1 - γ)) := by sorry
end StructEstMDP.FiniteTime
Source
Zeng, Hong and Garcia, Structural Estimation of Markov Decision Processes in High-Dimensional State Space with Finite-Time Guarantees, arXiv:2210.01282v3, p. 21, display (40), proof of Theorem 1
Human review
Confirmed by the mission captain (proposal self-audit).
Confirmed by the moderator at approval.