Global Convergence of Policy Gradient Methods to (Almost) Locally Optimal Policies: Random-Horizon Policy Gradient with Robbins–Monro Stepsizes Drives the Policy Gradient to Zero Almost SurelyResearch Paper
Policy gradient methods are the main family of reinforcement learning algorithms that search directly over a parameterized class of stochastic policies , , by stochastic gradient ascent on the expected discounted return. They underlie REINFORCE, actor-critic methods and their deep-learning descendants. The return is a nonconvex…