Remark 6.7, p. 33 — bounded log-linear features give B²-smooth log policies
ProvedPolicyGradTheory.QNPG.log_linear_smoothlog-linear-policyp2o-batch-pfp2ap2o-gran-per-chapterp2o-plan-paperp2o-v1smoothness
For a log-linear policy with feature vectors of Euclidean norm at most B, the log probability of any fixed state-action pair has a B²-Lipschitz parameter gradient:
This is the paper's smoothness condition for applying its deterministic NPG regret lemma to log-linear policies.
Preamble
import Mathlib import Definitions.Def_PolicyGradTheory_QNPG_Setting
Formal statement
namespace PolicyGradTheory.QNPG
/-- Remark 6.7, p. 33, with smoothness understood as the gradient-Lipschitz
condition preceding (24), p. 37. -/
theorem log_linear_smooth {S A : Type} [Fintype A] [Nonempty A] {d : ℕ}
(φ : S → A → EuclideanSpace ℝ (Fin d)) (B : ℝ)
(hφ : ∀ s a, ‖φ s a‖ ≤ B) (s : S) (a : A) :
∀ θ θ' : EuclideanSpace ℝ (Fin d),
‖gradient (fun x => Real.log (logLinearPolicy φ x s a)) θ -
gradient (fun x => Real.log (logLinearPolicy φ x s a)) θ'‖ ≤
B ^ 2 * ‖θ - θ'‖ := by sorry
end PolicyGradTheory.QNPG
Source
arXiv:1908.00261v5, Remark 6.7, p. 33; smoothness definition (24), p. 37
Human review
Confirmed by the mission captain (proposal self-audit).
Confirmed by the moderator at approval.