pg_bellman_step
Provedalgebraic-geometryanalysiscalculusmdpopen-problempolicy-gradientreinforcement-learning
The one-step (Bellman) gradient identity for the finite-horizon value function. Differentiating the backward recursion by the product rule (and using that and are -independent),
This is the inductive engine of the policy gradient theorem: a score term plus a discounted expectation of the next-state gradient. Needs pg_value_differentiable to rewrite deriv of the inner value functions.
Preamble
import Definitions.Def_pg_mdp_core import Mathlib.Analysis.Calculus.Deriv.Basic import Mathlib.Analysis.Calculus.Deriv.Add import Mathlib.Analysis.Calculus.Deriv.Mul import Mathlib.Analysis.SpecialFunctions.Log.Deriv import Mathlib.Data.Fintype.BigOperators import Mathlib.Data.Real.Basic
Formal statement
theorem pg_bellman_step {S A : Type} [Fintype S] [Fintype A]
(P : S → A → S → ℝ) (r : S → A → ℝ) (π : ℝ → S → A → ℝ) (θ : ℝ)
(hdiff : ∀ s a, DifferentiableAt ℝ (fun θ' => π θ' s a) θ)
(k : ℕ) (s : S) :
deriv (fun θ' => pgValue P r π (k+1) θ' s) θ
= (∑ a : A, pgQ P r π k θ s a * deriv (fun θ' => π θ' s a) θ)
+ ∑ a : A, π θ s a * ∑ s' : S, P s a s' * deriv (fun θ' => pgValue P r π k θ' s') θ := by sorrySource
R. Sutton, D. McAllester, S. Singh, Y. Mansour. Policy Gradient Methods for Reinforcement Learning with Function Approximation. NeurIPS 1999.