Motivation
Planning in a Markov decision process with a large or infinite state space cannot visit every state. The trajectory tree method of Kearns, Mansour and Ng (2000) finds a near-best policy in a class Π with a number of generative-model calls independent of the state space, but exponential in the horizon T. Chapter 6 of S. M. Kakade's PhD thesis, On the Sample Complexity of Reinforcement Learning (University College London, 2003), trades that exponential factor for a weaker notion of optimality. The weaker notion is tied to a reset distribution μ that encodes prior knowledge of where good policies spend their time, such as a desired trajectory in robotic control or a known operating regime of a queueing network.
The resulting algorithm, μ-PolicySearch, chooses one decision rule per time step, backward in time, from a hypothesis class Π1, optimizing each against states drawn from μ. The same idea was later published as Policy Search by Dynamic Programming (Bagnell, Kakade, Ng and Schneider, NeurIPS 2003). Conservative policy iteration (Kakade and Langford, ICML 2002) carries the same reset-distribution idea into the discounted setting.
Timeline:
- 2000 — Kearns, Mansour and Ng: trajectory trees; uniform convergence over Π with O(2T) dependence on the horizon.
- 2002 — Kakade and Langford: the performance difference lemma for discounted values and conservative policy iteration under a reset distribution.
- 2003 — Kakade's thesis, Chapter 6: μ-optimality (Theorem 6.3.1) and μ-PolicySearch, with sample complexity polynomial in T (Theorem 6.3.3, an O(⋅) statement not part of this mission).
Setting
A T-epoch MDP has a finite state set S, a finite nonempty action set A, a transition kernel P(s′∣s,a), a deterministic reward r(s,a)∈[0,1] and a horizon T≥1. Decision epochs are t=0,…,T−1. A non-stationary policy π gives a distribution π(⋅∣s,t) over actions at each state and epoch. A decision rule is a map h:S→A. A class Π1 of decision rules induces the class Π=Π1T of deterministic policies that use a rule of Π1 at every epoch.
Values are normalized. The t-value Vπ,t(s) is T1 times the expected reward collected from epoch t to T−1 when starting in s at epoch t, so Vπ,t(s)∈[0,1]. The value is Vπ(s)=Vπ,0(s). The state-action value is Qπ,t(s,a)=T1r(s,a)+Es′∼P(⋅∣s,a)[Vπ,t+1(s′)], and the advantage is Aπ,t(s,a)=Qπ,t(s,a)−Vπ,t(s). The future state-time distribution of π from s0 is dπ,s0(s,t)=T1Pr(st=s∣π,s0) on S×{0,…,T−1}.
A μ-reset distribution μ is a family of distributions μ(⋅∣t) on S, one per epoch t<T. Its joint law, uniform over epochs, is μ(s,t)=μ(s∣t)/T. For a decision rule h the μ-advantage is
Aπ,t(μ,h)=Es∼μ(⋅∣t)[Aπ,t(s,h(s))],
and Qπ,t(μ,h) is defined in the same way. The ℓ1 distance of two functions on S×{0,…,T−1} is ∥p−q∥1=∑s∑t<T∣p(s,t)−q(s,t)∣.
Formalization targets
Goal: μ-optimality (Theorem 6.3.1, p. 75)
If a policy π satisfies Aπ,t(μ,h)≤ε/T for all h∈Π1 and t<T, then for all π′∈Π and all start states s0,
Vπ(s0)≥Vπ′(s0)−ε−T∥dπ′,s0−μ∥1.
Milestones
- Lemma 5.2.1 (undiscounted), p. 59. Vπ′(s0)−Vπ(s0)=TE(s,t)∼dπ′,s0Ea∼π′(⋅∣s,t)[Aπ,t(s,a)] for all policies π,π′.
- Change of measure, p. 76. TEdπ′,s0Eπ′[Aπ,t]≤TEμEπ′[Aπ,t]+T∥dπ′,s0−μ∥1.
- Lemma 5.3.1, p. 63. The policy returned by non-stationary approximate policy iteration (NAPI) has the same Qπ,t as the input policy of update t. Its advantages are bounded by the per-state errors εt(s) of the PolicyChooser.
- Theorem 6.3.2, p. 76. Exact μ-PolicySearch returns π~ with Aπ~,t(μ,h)≤0 for all h∈Π1, t<T.
Significance
Theorem 6.3.1 states what μ-PolicySearch buys. The guarantee holds at every start state, whereas the trajectory tree guarantee holds only at the root it was built for. The policy is guaranteed to compete with each policy of Π whose future state-time distribution is close to μ. The penalty T∥dπ′,s0−μ∥1 makes the role of the reset distribution precise: it is the price of the change from the "training" distribution μ to the "test" distribution dπ′,s0. With Theorem 6.3.2 at ε=0, the exact algorithm attains the bound with no slack. The sample-based analysis (Theorem 6.3.3) reduces to controlling the ε/T condition.
The results are proved in the thesis, and no machine-checked version is known to exist. The mission produces a Lean development of the normalized T-epoch model (non-stationary policies, t-values, advantages, future state-time distributions) together with the undiscounted performance difference lemma. These pieces are the common substrate of Chapters 4–6 of the thesis and of later finite-horizon policy-search analyses.
Difficulty
The performance difference lemma relates two objects defined in opposite directions of time: the state distribution of π′ runs forward from s0, while the t-values of π are defined backward from the horizon. A formal statement must keep the epochs of the two aligned, including the boundary epochs t=0 and t=T−1. The change of measure needs the bound ∣Aπ,t(s,a)∣≤1, which holds only because values are normalized and rewards lie in [0,1]. With unnormalized rewards the constant T in front of the ℓ1 term is wrong.
A first idea is to read the hypothesis as "the advantages are small under dπ′,s0". That reading is not available. The hypothesis controls averages under μ only, and the gap between the two distributions is exactly the penalty term.
Formalization scope
- The state space is finite (
Fintype S), so expectations and the ℓ1 distance are finite sums. The thesis also allows infinite state spaces in this chapter, with integrals in place of sums.
- Epochs are 0-based, t∈{0,…,T−1}, with T≥1.
- Values carry the factor 1/T. Vπ,t is defined by backward recursion and is 0 for t≥T.
- Policies are real functions
π t s a together with a validity predicate. Deterministic policies h=(ht) enter as indicator policies π(a∣s,t)=1[a=ht(s)].
- Π1 is an arbitrary set of deterministic decision rules, not assumed finite. Exact μ-PolicySearch is described as a run: the chosen rule at each update is an exact maximizer over Π1 of Qπ,t(μ,⋅) for the current policy π, and the run is stated by hypotheses rather than by a choice of argmax.
- The policy π in the goal may be stochastic. The competitors π′ range over Π1T.
- The μ-mismatch term must stay in the goal. Dropping it, or replacing dπ′,s0 by μ, gives a different and false statement. An empty class Π1 makes the conclusion vacuous. That case is harmless, since the conclusion only quantifies over π′∈Π1T, and the source has it too.
The T-epoch layer (policies, state distributions, values, advantages, the future state-time distribution) is shared with the other missions of this series (namespaces SampleComplexityRL.PolicyGrad and SampleComplexityRL.Mismeasure); the μ-reset objects are defined in SampleComplexityRL.MuPolicySearch; IsTransitionKernel and IsPolicy are published definitions. Contributions of the performance difference lemma and of general facts about the T-epoch layer (values in [0,1], dπ,s0 sums to one) are welcome.
Selected references
- S. M. Kakade, On the Sample Complexity of Reinforcement Learning, PhD thesis, Gatsby Computational Neuroscience Unit, University College London, 2003. https://discovery.ucl.ac.uk/id/eprint/10100726/
- M. Kearns, Y. Mansour, A. Y. Ng, Approximate planning in large POMDPs via reusable trajectories, NeurIPS 12, 2000. https://papers.nips.cc/paper_files/paper/1999
- S. Kakade, J. Langford, Approximately optimal approximate reinforcement learning, ICML 2002. https://dl.acm.org/doi/10.5555/645531.656005
- J. A. Bagnell, S. Kakade, A. Y. Ng, J. Schneider, Policy search by dynamic programming, NeurIPS 16, 2003. https://papers.nips.cc/paper_files/paper/2003