Bellman's Dynamic Programming VIII: Multi-Stage Games, Games of Survival and the Extended Min-Max TheoremTextbook
Why multi-stage games
Chapter X of Richard Bellman's Dynamic Programming (Princeton University Press, 1957; Princeton Landmarks edition 2010, DOI 10.2307/j.ctv1nxcw0f) carries the functional-equation method of the earlier chapters from one-player decision processes to two-player zero-sum games that are played over many stages. At each stage the two players make simultaneous choices, and those choices change the state in which the next stage is played. Games of survival are the standard example. They generalize the gambler's ruin: two players with finite resources play until one of them is ruined. The same framework is the discrete-time form of what Shapley (1953) called stochastic games. It underlies pursuit games, attrition models and the later theory of dynamic games in operations research and economics.
The chapter builds on von Neumann's min-max theorem (1928), which Bellman assumes without proof (§ 3, Eq. (3.4)). It proves three kinds of result: existence and uniqueness for the general multi-stage game equation (§§ 11–16), existence and uniqueness for games of survival with integer payoffs (§ 19), and an extended min-max theorem for ratios of bilinear forms (§ 23). Bellman proposes the ratio as a criterion for non-zero-sum games.
Setting
A matrix game is given by a real matrix . Player chooses row with probability and player chooses column with probability ; and are distribution vectors (points of the standard simplex). The expected return to is
The game has value when , all extrema attained. In the Lean development this is IsMaxMinMinMaxValue, and the bilinear form is bilin.
In the general multi-stage game (§ 12) the state is a pair of vectors , with norms . In state , chooses in a choice domain and chooses in . receives , and play continues from the state , with weight . Mixed strategies , are probability measures on the choice domains. The multi-stage game equation is
A game of survival with integer payoffs (§ 19) has one integer state , the resources of out of a fixed total . is ruined at and wins at . At each stage the players play the game with matrix , whose entries are the transfers to .
Formalization targets
Goal: the extended min-max theorem (Chapter X, Theorem 7)
For real matrices , with for all distribution vectors ,
It is the chapter's final result, and its statement involves only finite matrices.
Milestones
- Eq. (3.4), von Neumann's theorem . It is already on the platform as
AGT.zero_sum_minimax(existence of a saddle point) and enters as a reference. - Lemma 1: the value of a one-stage game moves by at most the largest change of its kernel, .
- Theorem 1: under hypotheses (4a)–(4e), the multi-stage game equation has a unique solution among functions continuous on that vanish at the origin, and it is the uniform limit on bounded regions of the successive approximations.
- Theorem 3: the successive approximations converge from any admissible initial function.
- Theorem 4: stability, .
- Theorem 5: the game of survival equation with boundary values and has a unique solution with values in .
Significance
Theorem 7 gives a value for ratio games, the games in which a player maximizes a return per unit of a resource consumed. Bellman uses it to give a rationale for the play of non-zero-sum games (§ 24) and to derive the approximate equation (22.4) for non-zero-sum games of survival. Chapter XI, Theorem 5 generalizes it to Markovian decision processes. Theorem 1 is the existence and uniqueness result that justifies replacing an infinite game by its functional equation, and Theorems 3 and 4 make that equation usable for computation and perturbation. Theorem 5 is an early uniqueness theorem for a discrete stochastic game with absorbing boundaries.
None of these statements is formalized on the platform. Von Neumann's theorem is (AGT.zero_sum_minimax, and Sion's theorem as FamousTheorems.sion_minimax_theorem). Theorem 7 is a classical result: with a positive denominator the ratio is quasiconcave in and quasiconvex in . This mission asks for a machine-checked proof of the book's statement, by Bellman's route or by any other. Theorems 1, 3 and 4 need a formal theory of games whose mixed strategies are probability measures on moving compact choice domains. No such theory is on the platform yet.
Difficulty
The ratio in Theorem 7 is not bilinear, and in general it is neither concave in nor convex in , so neither von Neumann's theorem nor a concave-convex min-max theorem applies to it directly. For Theorem 1, the contraction argument needs each one-stage game to have a value and the value to depend continuously on the state. That requires a min-max theorem for continuous games on compact sets, together with continuity of the value when the choice domains move. In Theorem 5, existence follows from monotone iteration. The uniqueness step is the hard part: the operator is not a contraction in the sup norm, and a second solution must be ruled out even at states where the optimal mixed strategies are degenerate.
Formalization scope
- Distribution vectors are points of
stdSimplex ℝ ιfor nonempty finite types. "Max-min equals min-max" always asserts attainment (IsGreatest/IsLeast) and never usessSup/sInf, which return on empty or unbounded sets. Theorem 7 must not assume a saddle point; its only hypothesis is the bound on the denominator. - States are
Fin n → ℝwith the norm of Eq. (12.1). Choice domains are nonempty compact sets (NonemptyCompacts), and mixed strategies are probability measures of full mass on them. "Vary continuously" in (4b), which the book does not define, is read as continuity in Mathlib's topology on nonempty compact sets (for a metric space, the topology of the Hausdorff distance). - The maxima of (4d) and of Theorem 4 are expressed through majorants, which is equivalent to the book's condition. is stated explicitly. In Lemma 1 the kernels are assumed measurable and bounded on so that the integrals exist, and each game having a value is a hypothesis, as in Bellman's footnote 4.
- Theorem 3's "converges" is stated as uniform convergence on bounded regions, the mode of Theorem 1, whose proof Theorem 3 repeats. Theorem 4's unspecified is any . The book's p. 301 prints the first min-max of Theorem 4 with the subscripts , interchanged; the equation used is that of Theorem 1.
- Theorem 5 adds the implicit : for the boundary conditions contradict each other. The state is an integer and .
- Taking the choice domains constant would trivialize Theorem 1: condition (4d) would then force on them. The choice domains therefore depend on the state.
- Not included: Theorem 2 (optimal strategies of the infinite game, which needs a model of the game's plays), Theorem 6 (non-zero-sum survival; the boundary conditions (21.3) do not cover all states, see the moderation notes).
Contributions of reusable infrastructure are welcome: the min-max theorem for continuous games on compact sets (Eq. (4.2)), continuity of the value in the choice domains, and value iteration for contracting game operators.
Selected references
- R. Bellman, Dynamic Programming, Princeton University Press, 1957; Princeton Landmarks in Mathematics, 2010. DOI 10.2307/j.ctv1nxcw0f
- J. von Neumann, Zur Theorie der Gesellschaftsspiele, Mathematische Annalen 100, 1928. DOI 10.1007/BF01448847
- L. S. Shapley, Stochastic games, Proceedings of the National Academy of Sciences 39, 1953. DOI 10.1073/pnas.39.10.1095
- M. Sion, On general minimax theorems, Pacific Journal of Mathematics 8, 1958. DOI 10.2140/pjm.1958.8.171
- I. L. Glicksberg, A further generalization of the Kakutani fixed point theorem, with application to Nash equilibrium points, Proceedings of the AMS 3, 1952. DOI 10.1090/S0002-9939-1952-0046638-5