Motivation
Dynamic programming (DP) solves sequential decision problems by backward recursion: compute the optimal cost of the last stage, then of the last two stages, and so on. For problems with finitely many states and controls and real-valued costs, the recursion obviously gives the optimal cost. Applications are rarely like that. Control spaces are continuous, costs can be unbounded or infinite, the criterion can be multiplicative (risk-sensitive exponential cost) or worst-case (minimax), and the set of policies is an infinite product of function spaces. In this setting the DP recursion can fail to produce the optimal cost.
Bertsekas and Shreve, Stochastic Optimal Control: The Discrete-Time Case (Academic Press 1978; Athena Scientific 1996), Part I, separates the order-theoretic content of DP from the measure theory. It works with an abstract monotone mapping H that covers deterministic, stochastic, multiplicative-cost and minimax problems at once, following Bertsekas, Monotone mappings with application in dynamic programming, SIAM J. Control Optim. 15 (1977). Chapter 3 answers the finite-horizon questions: when does the DP algorithm give the N-stage optimal cost, and when do optimal or nearly optimal policies exist? This mission is the first of a series formalizing the book. Later chapters (contraction models, monotone increase and decrease models, the Borel models of Part II) are built on the model fixed here.
Setting
Let S (states) and C (controls) be sets, and for each x∈S let U(x)⊆C be a nonempty control constraint set. Write R∗=[−∞,∞] and let F be the set of all functions J:S→R∗, ordered pointwise. A mapping H:S×C×F→R∗ is given, subject to the Monotonicity Assumption: J≤J′ implies H(x,u,J)≤H(x,u,J′) for all x∈S, u∈U(x).
A selector is a function μ:S→C with μ(x)∈U(x) for all x. A policy is a sequence π=(μ0,μ1,…) of selectors. Define
Tμ(J)(x)=H[x,μ(x),J],T(J)(x)=u∈U(x)infH(x,u,J),
and let Tk be the k-fold composition of T. A terminal function J0∈F with J0(x)>−∞ for all x is fixed. The N-stage cost of π and the N-stage optimal cost are
JN,π=(Tμ0Tμ1⋯TμN−1)(J0),JN∗(x)=πinfJN,π(x).
A policy is uniformly N-stage optimal if each tail (μi,μi+1,…) is (N−i)-stage optimal, and N-stage ε-optimal if JN,π(x)≤JN∗(x)+ε where JN∗(x)>−∞ and JN,π(x)≤−1/ε where JN∗(x)=−∞.
The three conditions on H used in the chapter are F.1 (continuity of H along nonincreasing sequences Jk with H(x,u,J1)<∞), F.2 (there is α>0 with H(x,u,J)≤H(x,u,J+r)≤H(x,u,J)+αr for all r>0), and F.3 (a quantitative selection property with a constant β>0).
Formalization targets
Goal: Proposition 3.1
Under F.1, if Jk,π(x)<∞ for all x,π and k=1,…,N; or under F.2, if Jk∗(x)>−∞ for all x and k=1,…,N:
JN∗=TN(J0),
and under F.2, for every ε>0 there is πε with JN∗≤JN,πε≤JN∗+ε.
Milestones
- Proposition 3.3: π∗ is uniformly N-stage optimal iff (Tμk∗TN−k−1)(J0)=TN−k(J0) for k<N. Needs monotonicity only.
- Corollary 3.3.1: a uniformly N-stage optimal policy exists iff every infimum Tk+1(J0)(x)=infuH[x,u,Tk(J0)] is attained, and then JN∗=TN(J0).
- Proposition 3.4: if C is Hausdorff and every sublevel set {u∈U(x)∣H[x,u,Tk(J0)]≤λ} is compact, then JN∗=TN(J0) and a uniformly N-stage optimal policy exists.
- Proposition 3.7: the minimax mapping H(x,u,J)=supw∈W(x,u){g+αJ[f]} satisfies F.2 with constant α.
- Proposition 3.6: the multiplicative mapping H(x,u,J)=E{gJ[f]∣x,u} over a countable disturbance set satisfies F.1, and F.2 with constant b when 0≤g≤b.
- Proposition 3.2: under F.3 and the finiteness of Jk,π, JN∗=TN(J0) and, for εn↓0, policies with {εn}-dominated convergence to optimality exist.
- Corollary 3.7.1(a): for minimax control with J0=0 and Jk∗>−∞, the DP algorithm gives JN∗ and N-stage ε-optimal policies exist.
Significance
The identity JN∗=TN(J0) says that an infimum over an infinite-dimensional policy space equals N nested one-dimensional infima. Every numerical use of finite-horizon DP depends on it, and so do the infinite-horizon results of later chapters, which pass to the limit in TN(J0). Corollary 3.3.1 and Proposition 3.4 give the existence of optimal policies, and Propositions 3.6 and 3.7 verify the abstract hypotheses for two models outside standard expected additive cost.
These results are proved in the book; none of them is formalized. Mathlib has no abstract DP model, and the platform's finite-horizon results (Bertsekas, Dynamic Programming and Optimal Control, Prop. 1.3.1 and the minimax DP algorithm) assume finite disturbance and constraint sets and real costs. They are special cases, not this theory. The finite-horizon results of the 1977 paper (Lemma 3.1 here, on compact sublevel sets, and Corollary 3.1.1, the F.1′ case) are already posed on the platform and are not posed again.
Difficulty
The obvious argument interchanges the infimum over policies with the composition of operators: infπTμ0(⋯)=T(infπ′⋯). The inequality TN(J0)≤JN∗ follows from monotonicity alone. The reverse inequality is the content. Taking a near-minimizing selector at each stage requires either passing a limit inside H (F.1) or bounding how errors at later stages propagate through H (F.2, F.3). Both steps break at infinite values. With Jk∗(x)=−∞ there may be no ε-optimal policy at all (Counterexample 4 of the book). Without F.1 or F.2 the identity itself fails (Counterexamples 1–3). A proof must therefore track separately the states where the optimal cost is −∞, which is why F.3 and the definition of ε-optimality have two cases.
Formalization scope
The model is a structure Model S C with fields U, U_nonempty, H : S → C → (S → EReal) → EReal and the monotonicity proof. Policies are ℕ → Selector, with selectors as a subtype of S → C. TN is m.T^[N], and (Tμ0⋯TμN−1)(J) is a recursion that applies TμN−1 first. All values lie in EReal. The book's convention ∞−∞=∞ never arises in Propositions 3.1–3.4, which only add real numbers to extended reals. The minimax and multiplicative mappings implement it explicitly (badd, and an expectation that returns +∞ when the positive part diverges). Every theorem assumes J0>−∞ and N≥1. Assumptions F.1–F.3 are predicates on the model. F.2 is also available with a named constant (F2With) so that Propositions 3.6 and 3.7 can carry the book's constants b and α.
JN∗ is defined as an infimum over policies of the composed operators, never through T, so the goal is not true by definition. A formalization in which JN,π already contains an infimum over controls would make Proposition 3.1 hold by rfl, and this one rules that out.
Proving the goal needs elementary EReal order arithmetic, iterated infima over subtypes, and pointwise selection of near-minimizers via choice. Proposition 3.6 additionally needs monotone and dominated convergence for countable sums in ℝ≥0∞. The model and operator definitions are reusable by the later missions of the series (contraction, monotone increase and decrease models). Proofs of any milestone, and reusable EReal lemmas about shifting by real constants, are welcome.
Selected references
- D. P. Bertsekas and S. E. Shreve, Stochastic Optimal Control: The Discrete-Time Case, Academic Press 1978; Athena Scientific 1996, Chapters 2–3. https://web.mit.edu/dimitrib/www/soc.html
- D. P. Bertsekas, Monotone mappings with application in dynamic programming, SIAM J. Control Optim. 15(3) (1977) 438–464. https://doi.org/10.1137/0315031
- D. P. Bertsekas, Dynamic Programming and Stochastic Control, Academic Press 1976.
- D. P. Bertsekas, Abstract Dynamic Programming, 3rd ed., Athena Scientific 2022. https://web.mit.edu/dimitrib/www/abstractdp_MIT.html