Motivation
Approximate policy iteration and policy gradient methods are the two classical families of reinforcement learning algorithms that work with approximate, sampled information instead of an exact model. Kakade and Langford (ICML 2002) observed that neither family answers three basic questions: is there a performance measure that is guaranteed to improve at every step, how hard is it to verify that an update improves it, and what performance is reached after a reasonable number of updates. Greedy approximate policy iteration can make the policy worse when the value estimates are slightly wrong at a few states, and policy gradient methods can stall on plateaus where estimating the gradient needs an enormous number of samples.
Their answer is conservative policy iteration: instead of jumping to a greedy policy, move only a controlled fraction of the way toward it, with a step size chosen from an estimate of how much the greedy policy helps. The paper proves that this update improves a restart-distribution performance measure monotonically, terminates after a number of iterations that depends only on the reward range and the target accuracy, and stops at a policy that the greedy oracle can no longer improve by much. The idea is the direct ancestor of trust-region and proximal policy optimization methods (TRPO, Schulman et al. 2015; PPO, Schulman et al. 2017), whose improvement bounds are refinements of the paper's Theorem 4.1.
Setting
A finite Markov decision process has a finite nonempty set of states S, a finite nonempty set of actions A, transition probabilities P(s′;s,a) (for each state s and action a, a probability distribution over next states s′), a reward function R:S×A→[0,R] with R>0, and a discount factor 0≤γ<1. A stochastic policy π(a;s) gives, for each state s, a probability distribution over actions.
The normalized value of π from s is Vπ(s)=(1−γ)E[∑t≥0γtR(st,at)∣π,s], where s0=s, at∼π(⋅;st), st+1∼P(⋅;st,at); it lies in [0,R]. The state-action value is Qπ(s,a)=(1−γ)R(s,a)+γEs′∼P(s′;s,a)[Vπ(s′)] and the advantage is Aπ(s,a)=Qπ(s,a)−Vπ(s)∈[−R,R].
For a state distribution μ (a restart distribution), the discounted future state distribution is dπ,μ(s)=(1−γ)∑t≥0γtPr(st=s;π,μ) (eq. (2.1)), and the performance measure is ημ(π)=Es∼μ[Vπ(s)].
The policy advantage of a policy π′ with respect to π and μ is
Aπ,μ(π′)=Es∼dπ,μ[Ea∼π′(a;s)[Aπ(s,a)]],
and OPT(Aπ,μ)=maxπ′Aπ,μ(π′). The conservative update (4.1) is πnew=(1−α)π+απ′ with α∈[0,1]. An ε-greedy policy chooser Gε (Definition 4.3) returns, for every policy π, a policy π′ with Aπ,μ(π′)≥OPT(Aπ,μ)−ε.
Conservative policy iteration (§5) starts from any policy and repeats: call Gε(π,μ) to get π′; form an 3ε-accurate estimate A^ of Aπ,μ(π′) from μ-restarts; if A^<32ε, stop and return π; otherwise apply (4.1) with α=4R(1−γ)(A^−ε/3) and repeat.
Formalization targets
Goal: Theorem 4.4 (p. 5)
With probability at least 1−δ, conservative policy iteration (i) strictly improves ημ with every policy update, (ii) stops after at most 72R2/ε2 policy updates, and (iii) returns a policy π with
OPT(Aπ,μ)<2ε.
The estimation step is represented by its guarantee: each reached loop's estimate fails to be 3ε-accurate with probability at most δ/(N+1), N=⌊72R2/ε2⌋.
Milestones
Lemma 6.1 (p. 6), the performance difference identity:
ημ(π~)−ημ(π)=1−γ1E(a,s)∼π~dπ~,μ[Aπ(s,a)].
Theorem 4.1 (p. 4), with ε=maxs∣Ea∼π′(a;s)[Aπ(s,a)]∣ and all α∈[0,1]:
ημ(πnew)−ημ(π)≥1−γα(A−1−γ(1−α)2αγε).
Corollary 4.2 (p. 5): if A≥0, the step size α=4R(1−γ)A gives
ημ(πnew)−ημ(π)≥8RA2.
Significance
Theorem 4.4 is the first guarantee of its kind for approximate reinforcement learning: the number of iterations is bounded by 72R2/ε2, independent of the number of states and of the restart distribution, and every iteration provably helps. Lemma 6.1 is the standard performance difference lemma, used throughout the analysis of policy optimization, including natural policy gradient and trust-region methods; Theorem 4.1 is the prototype of the "surrogate objective minus a penalty" bound that TRPO refines.
These results are proved in the paper. As far as a search of the platform shows, none is formalized: the platform's finite-horizon performance difference lemma (Foster and Rakhlin's Lemma 13) is a different statement, for episodic problems with non-stationary policies. This mission produces machine-checked versions of the discounted performance difference identity, the conservative improvement bound with its exact constants, and the high-probability termination and quality guarantee of the algorithm, all on top of an explicit infinite-horizon model rather than an assumed Bellman equation.
Difficulty
The obvious argument for the improvement bound expands ημ(πnew) to first order in α; that only gives 1−γαA+O(α2) with an unspecified constant, which cannot fix a step size. The exact bound needs control of how far the state distribution of the mixed policy drifts from that of the old policy, uniformly in time, and the performance difference identity is only useful once the states are weighted by the new policy's distribution. On the formal side, Vπ and dπ,μ are infinite discounted series, so summability, exchanges of sums and the identities ∑sdπ,μ(s)=1 and ∑aπ(a;s)Aπ(s,a)=0 must all be established from the definitions. For Theorem 4.4, the algorithm is a random process whose policies depend on all earlier estimates; the argument has to be made pathwise on the event that every reached loop is accurate, together with a union bound over the loops that can be reached.
Formalization scope
Policies are functions π : S → A → ℝ with π s a the paper's π(a;s), and P s a s' is P(s′;s,a); both are constrained by the published predicates IsPolicy and IsTransitionKernel. Vπ is (1−γ) times the published series PolicyValue, so values are normalized as in the paper. OPT is a real supremum over all stochastic policies; the set is nonempty and bounded, and the maximum is attained. Every theorem carries the standing assumptions of §2: finite nonempty S and A, a transition kernel, rewards in [0,R] with R>0, 0≤γ<1, and a state distribution μ. In Corollary 4.2, R is any upper bound on the rewards rather than necessarily the attained maximum.
In Theorem 4.4 the run is formalized pathwise, driven by arbitrary real random estimates on a probability space; the conclusion bounds the probability of the failure event by δ. Two deviations from the printed statement are disclosed. First, (ii) is stated for policy updates: the proof bounds updates, and the algorithm calls Gε once more than it updates, so "at most 72R2/ε2 calls" is off by one. Second, the per-loop failure budget is δ/(N+1), which covers the N+1 loops that may be reached. The Hoeffding estimate (5.1) is not formalized: as printed it concerns the 6ε-biased target, and its role is taken by the accuracy hypothesis. The step size is clipped at 1, which never binds when the estimate is accurate. No trivializing reading is available: the accuracy hypothesis is satisfied by a perfect estimator and a 0-greedy chooser exists, so the theorem is not vacuous, and strict improvement at every update is required, not merely nonnegative change.
Pages are PDF pages; the paper has no printed page numbers.
A complete development needs summability and algebra of discounted occupation measures, the performance difference identity, and a union bound over the loops of a random process; the first two are reusable for any discounted policy-optimization result. Proofs of the milestones in any order are welcome.
Selected references
- S. Kakade and J. Langford, Approximately Optimal Approximate Reinforcement Learning, Proceedings of the 19th International Conference on Machine Learning (ICML), 2002. https://dl.acm.org/doi/10.5555/645531.656005
- J. Schulman, S. Levine, P. Moritz, M. Jordan, P. Abbeel, Trust Region Policy Optimization, ICML 2015. https://arxiv.org/abs/1502.05477
- J. Schulman, F. Wolski, P. Dhariwal, A. Radford, O. Klimov, Proximal Policy Optimization Algorithms, 2017. https://arxiv.org/abs/1707.06347
- D. J. Foster and A. Rakhlin, Foundations of Reinforcement Learning and Interactive Decision Making, 2023. https://arxiv.org/abs/2312.16730