Motivation
Controlled queueing systems (admission control, routing, service-rate selection) are naturally modelled as Markov decision chains whose state is a buffer content and therefore ranges over a countably infinite set. Linn Sennott's Stochastic Dynamic Programming and the Control of Queueing Systems (Wiley, 1999, DOI 10.1002/9780470317037) develops the dynamic programming theory for exactly this setting: countable state space, finite action sets, nonnegative and possibly unbounded costs, and value functions that are allowed to be infinite. The book's computational method, the approximating sequence method (ASM), replaces the infinite chain by a sequence of finite truncations and asks when optimal values and policies of the truncations converge to those of the original chain.
This mission is the first of a series on the book. It covers Chapter 3, finite horizon optimization, together with the model of Chapter 2 and three results from Appendices A and B that the chapter uses. The finite horizon theory is the entry point: it is where the book's general policy class, its extended-valued cost criteria and its approximating sequences are first used together.
Setting
A Markov decision chain Δ has a countable state space S; for each i∈S a finite nonempty action set Ai; a finite cost C(i,a)≥0; and for each a∈Ai a transition distribution (Pij(a))j∈S. A history at time t is ht=(i0,a0,…,it−1,at−1,it), and a general policy θ chooses the action at time t from a distribution θ(⋅∣ht) on Ait: it may use the whole history and may randomize. Stationary policies f (f(i)∈Ai) and deterministic Markov policies (a stationary policy for each time) are special cases.
Fix a finite terminal cost F≥0 and a discount factor 0<α≤1 (α=1 is the undiscounted case). The n horizon expected discounted cost of θ from initial state i is
vθ,α,n(i)=t=0∑n−1αtEθ[C(Xt,At)∣X0=i]+αnEθ[F(Xn)∣X0=i],
and the value function is vα,n(i)=infθvθ,α,n(i) over all general policies. Both may be +∞. A policy is optimal for the n horizon if it attains vα,n(i) at every i. For n≥1 put uα,n(i,a)=C(i,a)+α∑jPij(a)vα,n−1(j) and let Bi(α,n) be the set of a∈Ai minimizing it.
An approximating sequence (ΔN)N≥N0 has finite nonempty state spaces SN increasing to S, the same actions and costs, and transition distributions Pij(a;N) on SN converging to Pij(a) as N→∞. Its value functions are vα,nN. In an augmentation type approximating sequence, the probability Pir(a) of leaving SN to r is redistributed over SN by an augmentation distribution qj(i,a,r,N). Assumption FH(α, n) requires limsupNvα,nN(i) to be finite and at most vα,n(i) for every i. A stationary policy e is a limit point of stationary policies eN if, along a subsequence, eNr(i)=e(i) eventually for each i.
Formalization targets
Goal: Theorem 3.2.3
For fixed n≥1,
(∀i: N→∞limvα,nN(i)=vα,n(i)<∞)⟺FH(α,n),
and under either condition every limit point en of stationary policies enN with enN(i)∈BiN(α,n) satisfies en(i)∈Bi(α,n) for all i∈S.
Milestones
- Proposition A.1.1: a probability average of u is at least minu, with equality iff the distribution is concentrated on the minimizers.
- Theorem 3.1.2: the finite horizon optimality equation vα,n(i)=minauα,n(i,a), and the characterization of all optimal general policies.
- Corollary 3.1.4: choosing fn−t(i)∈Bi(α,n−t) yields an optimal deterministic Markov policy.
- Proposition 2.5.6: the augmentation (2.19) defines an approximating distribution.
- Lemma 3.2.2: vα,0N→vα,0 and liminfNvα,nN≥vα,n.
- Propositions B.3 and B.5: sequences of stationary policies, for Δ or for (ΔN), have limit points.
- Propositions 3.3.1, 3.3.2 and 3.3.4: three sufficient conditions for FH(α, n), namely bounded costs, an augmentation sending excess probability to a finite set, and the augmentation inequality (3.20).
Significance
Theorem 3.1.2 is the finite horizon dynamic programming equation in the generality the rest of the book needs: the value function is an infimum over history-dependent randomized policies, and the equation holds with infinite values allowed. Its characterization of optimal policies is Bellman's principle of optimality in necessary-and-sufficient form. Corollary 3.1.4 shows that deterministic Markov policies suffice. The discounted chapter builds on these results, since its value function is the limit of finite horizon ones, and so does the value iteration algorithm of the average cost chapters.
Theorem 3.2.3 is the finite horizon case of the approximating sequence method. It says exactly when finite truncations give the right answer, and it reduces the question to Assumption FH, for which Section 3.3 gives checkable conditions. The same structure (a lim inf inequality, a lim sup assumption, a limit point of optimal truncated policies) recurs for the discounted and the average cost criteria in later chapters.
The results are proved in the book. None of them is formalized: the platform has finite horizon dynamic programming only for Markov policies, abstract monotone mappings or finite reward-maximizing MDPs, and nothing on approximating sequences. A formalization contributes a Lean model of Markov decision chains with general policies and extended-valued criteria, which the later missions of the series restate and can merge with this one.
Difficulty
The obvious proof of the optimality equation conditions on the first action and state and then applies the induction hypothesis to the rest of the trajectory. With general policies the rest of the trajectory is governed by a continuation policy that depends on the first state and action, and the decomposition of the path law into a first step and a continuation must be proved from the definition of the process, not assumed. Infinite values also make the "only if" direction delicate: a strict inequality between expected costs becomes an equality once both sides are infinite.
For approximating sequences, the natural idea is to pass to the limit in the optimality equation of ΔN. This fails in general. Example 3.2.1 of the book has limNv1,2N(0)=2>1=v1,2(0), because truncation moves probability onto states of high cost and dominated convergence is not available. Only the lim inf inequality holds for free, through a generalized Fatou lemma for approximating distributions. The lim sup side is exactly what Assumption FH supplies. The limit point argument then needs the compactness statement of Appendix B and the fact that a lim inf can be passed through a minimum over a finite set.
Formalization scope
The state space is a type S with [Countable S], the actions a type Act, and A i : Finset Act is nonempty. Costs are ℝ≥0, transition probabilities ℝ≥0∞ summing to 1 over S, and all values and expectations are in ℝ≥0∞, so infima over policies are lattice infima and +∞ is a genuine value. A history is the list of past state–action pairs, most recent first, with the current state, and a policy gives a distribution on A i for every history. Expectations are sums over histories of the path probabilities ∏θ(as∣hs)Pisis+1(as), which is the book's (2.6) and (2.9), not the dynamic programming recursion. The discount factor satisfies 0<α≤1 in every statement. An approximating sequence is indexed by N∈N with a start level N0; its value functions are set to 0 for the finitely many N at which a given state is not yet in SN, which does not affect limits.
The optimality equation must not be made definitional by defining vθ,α,n or vα,n through the recursion (3.2). The policy class must not be restricted to deterministic Markov policies either, since that would make the characterization in Theorem 3.1.2 a different statement. Theorem 3.1.2(ii)(2) is stated with the guard vα,n(i)<∞; the book omits it, and without it the "only if" direction is false (see the item's note).
A complete development needs the first-step decomposition of the path law under a general policy, the generalized Fatou lemma for approximating distributions (Proposition A.2.5, a milestone of the Appendix A mission of this series), and lim inf / lim sup manipulations in ℝ≥0∞. The model definitions are reusable by every later mission of the series. Contributions are welcome at every milestone, including proofs of the definitional sanity facts (for instance vθ,α,0=F).
Selected references
- Linn I. Sennott, Stochastic Dynamic Programming and the Control of Queueing Systems, Wiley Series in Probability and Statistics, John Wiley & Sons, 1999. https://doi.org/10.1002/9780470317037
- Martin L. Puterman, Markov Decision Processes: Discrete Stochastic Dynamic Programming, Wiley, 1994 (the standard reference for finite horizon dynamic programming with history-dependent randomized policies).
- Richard Bellman, Dynamic Programming, Princeton University Press, 1957.