(7), §4, p. 13 — gradient of the direct parameterization: ∂V^π(μ)/∂π(a|s) = (1/(1−γ)) d^π_μ(s) Q^π(s,a)
ProvedPolicyGradTheory.ProjGA.direct_gradientLet be a finite discounted MDP with rewards in and , and let be a weighting of the start states. Under the direct parameterization , regard as a function of the vector .
At every policy and for all , ,
This is the policy gradient for the direct parameterization. It is what the projected gradient ascent update (9) uses, and it is the link between the gradient and the advantage in the proof of the gradient domination lemma.
Formalization Note The claim is about the Euclidean gradient (Mathlib's gradient, which presupposes differentiability) of the map on EuclideanSpace ℝ (S × A), evaluated at a policy; its coordinate is the stated value. The value function is the series , which converges on a neighbourhood of the simplex, so the gradient there is the genuine one. The page states (7) for a start distribution ; the identity is linear in and holds for every weighting, so no distribution hypothesis is assumed.
import Mathlib import Definitions.Def_FoundationsML_ReinforcementLearning_QFunction import Definitions.Def_PolicyGradTheory_ProjGA_MDP import Definitions.Def_PolicyGradTheory_ProjGA_Algorithm open FoundationsML.ReinforcementLearning
namespace PolicyGradTheory.ProjGA
/-- (7), §4, arXiv:1908.00261v5, p. 13: for the direct parameterization, at every policy `π`,
`∂V^π(μ)/∂π(a|s) = (1/(1−γ)) d^π_μ(s) Q^π(s,a)`; the Euclidean gradient of `π ↦ V^π(μ)` has
these coordinates. -/
theorem direct_gradient {S A : Type*} [Fintype S] [DecidableEq S] [Fintype A] [DecidableEq A]
(P : S → A → S → ℝ) (r : S → A → ℝ) (γ : ℝ) (hM : IsFiniteMDP P r γ) (μ : S → ℝ) :
∀ π ∈ simplexSet S A, ∀ (s : S) (a : A),
gradient (directValue P r γ μ) π (s, a) =
1 / (1 - γ) * visitation (asPolicy π) P γ μ s * QFunction (asPolicy π) P r γ s a := by sorry
end PolicyGradTheory.ProjGA
Confirmed by the mission captain (proposal self-audit).
Confirmed by the moderator at approval.