policy_gradient_baseline
Provedcalculusgeometrymdpopen-problempolicy-gradientreinforcement-learning
Baseline-invariance of the policy gradient estimator: for any baseline (independent of the action ),
This is the REINFORCE-with-baseline identity (Williams 1992). The most common choice gives the advantage form. Follows from policy_gradient_log_form and score_zero_mean (the term drops out because ). Requires the policy normalized for all near , not just at .
Preamble
import Definitions.Def_pg_mdp_core import Mathlib.Analysis.Calculus.Deriv.Basic import Mathlib.Analysis.Calculus.Deriv.Add import Mathlib.Analysis.Calculus.Deriv.Mul import Mathlib.Analysis.SpecialFunctions.Log.Deriv import Mathlib.Data.Fintype.BigOperators import Mathlib.Data.Real.Basic
Formal statement
theorem policy_gradient_baseline {S A : Type} [Fintype S] [Fintype A] [DecidableEq S]
(P : S → A → S → ℝ) (r : S → A → ℝ) (π : ℝ → S → A → ℝ) (θ : ℝ)
(hdiff : ∀ s a, DifferentiableAt ℝ (fun θ' => π θ' s a) θ)
(hpos : ∀ s a, 0 < π θ s a)
(hnorm : ∀ θ' s, ∑ a : A, π θ' s a = 1)
(T : ℕ) (s₀ : S) (b : ℕ → S → ℝ) :
deriv (fun θ' => pgValue P r π T θ' s₀) θ
= ∑ t ∈ Finset.range T, ∑ s : S, pgRho P π t s₀ θ s
* ∑ a : A, π θ s a * (pgQ P r π (T - 1 - t) θ s a - b t s)
* deriv (fun θ' => Real.log (π θ' s a)) θ := by sorrySource
R. J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. ML 8, 1992.