Operations Research

Pooling as a design principle

A firm that holds inventory in five warehouses needs more safety stock than one that holds the same inventory in one warehouse, because the demands of five regions do not all run high at once. Eppen (1979) made this precise for a multi-location newsvendor and gave it its name, the risk-pooling effect. Chapter 7 of Snyder and Shen's Fundamentals of Supply Chain Theory (2019) follows the same idea through three settings in which pooling happens without physical consolidation: two retailers who ship stock to each other after seeing demand (transshipments, after Tagaras 1989), and plants that can each make more than one product (process flexibility, after Jordan and Graves 1995). The chapter's capstone is the theorem of Simchi-Levi and Wei (2012) that, among designs in which every plant makes two products and every product is made at two plants, a single long chain through all of them is best. This mission formalizes the chapter's numbered results, with that theorem as its goal.

Setting

Risk pooling. $N$ distribution centers face normally distributed per-period demands $D_i \sim N(\mu_i, \sigma_i^2)$ with correlation coefficients $\rho_{ij}$, and each runs a base-stock policy with holding cost $h$ and backorder cost $p$ per unit per period, so its optimal expected cost is the optimal newsvendor cost optNvCost h p D, the infimum over base-stock levels $S$ of $\mathbb{E}[h(S - D)^+ + p(D - S)^+]$. Merging the centers gives one facing the total demand, normal with mean $\sum_i \mu_i$ and variance $\sigma_0^2 = \sum_i \sum_j \sigma_i \sigma_j \rho_{ij}$ (pooledVariance).

Transshipments. Two retailers $i, j$ with base-stock levels $S_i, S_j$ face independent demands. After demand is observed, under complete pooling the retailer with a surplus sends the retailer with a shortage $Y_{ji} = \min\{S_j - D_j,\ D_i - S_i\}$ units (transship), and nothing moves otherwise. The type-1 service level is the probability of no stockout, $\alpha^0_i = \Pr[D_i \le S_i]$ without and $\alpha_i = \Pr[D_i - S_i \le Y_{ji}]$ with transshipments; the type-2 service level is the fill rate, one minus expected unmet demand over expected demand, $\beta^0_i$ and $\beta_i$ likewise.

Process flexibility. A flexibility design on $n$ products and $n$ plants is a set $E$ of (product, plant) pairs, an edge $(i, j)$ meaning plant $j$ can make product $i$. Given a demand realization $d$ and a common plant capacity $C$, the performance $P(d, E)$ (perf) is the maximum sales obtainable by assigning production along the edges of $E$ without exceeding any capacity or demand, the linear program (7.22) to (7.26). A balanced system (BalancedSystem) has equal capacities and an exchangeable demand vector, one whose joint law is invariant under permutations of the products, and $[E] = \mathbb{E}[P(D, E)]$ is the expected performance (expPerf). The named designs are the dedicated design $D_n = \{(i, i)\}$, the long chain $C_n$ in which plant $j$ also makes product $j + 1$ (and plant $n$ makes product $1$), the open chain $L_k$ obtained from $C_k$ by deleting the edge $(1, k)$, and $L^n_k$, the open chain on the first $k$ pairs together with the dedicated edges of the rest. A 2-flexibility design (TwoFlex) is one in which every product has exactly two plants and every plant exactly two products; $C_n$ is one, and so is any union of disjoint shorter chains.

Formalization targets

Goal: Theorem 7.9

For a balanced system of size $n \ge 2$ with exchangeable demand,

$$C_n \in \arg\max_{A \in \mathcal{F}_2} [A],$$

that is, $C_n$ is a 2-flexibility design and $[A] \le [C_n]$ for every 2-flexibility design $A$. This is long_chain_optimal.

Supporting targets

The chapter's route to the goal: Lemma 7.5, supermodularity of sales in the flexible edges of the long chain for every realization, $P(d, E) + P(d, E \setminus \{\alpha, \beta\}) \ge P(d, E \setminus \{\alpha\}) + P(d, E \setminus \{\beta\})$ for $E \subseteq C_n$; Corollary 7.6, the same in expectation; Lemma 7.7, the increments $[L^n_{k+1}] - [L^n_k]$ are nondecreasing in $k$, ending with $[C_n] - [L^n_n]$; and Lemma 7.8, $[C_n] = n([L_n] - [L_{n-1}])$.

Risk pooling, Theorem 7.1: $g^*_C \le g^*_D$, the optimal cost of the merged center is at most the sum of the optimal costs of the separate ones, with the covariance inequality $\sqrt{\sum_i\sum_j \sigma_i\sigma_j\rho_{ij}} \le \sum_i \sigma_i$ as a separate lemma.

Transshipments, Theorems 7.2 to 7.4: $\alpha_i = \alpha^0_i + |\partial\mathbb{E}[Y_{ji}]/\partial S_i|$, $\beta_i = \beta^0_i + \mathbb{E}[Y_{ji}]/\mathbb{E}[D_i]$, and all four post-transshipment service levels are nondecreasing in $S_i$.

Significance

Theorem 7.9 is the analytical answer to a question that had been settled only by simulation: Jordan and Graves reported that one chain through all plants achieves nearly twice the sales benefit of three short chains with the same number of edges, and Simchi-Levi and Wei proved that no arrangement of the same edge budget does better. It is the justification for the chaining guideline used in automotive and semiconductor capacity planning, and Lemma 7.8, which expresses the long chain through open chains, is what makes the long chain's performance computable by a greedy pass. Theorem 7.1 is the quantitative basis for consolidation decisions and for postponement, since a generic product is pooled inventory. Theorems 7.2 to 7.4 quantify what transshipments buy in service, which is the argument for allowing them despite their cost.

None of these results has a machine-checked proof. The book proves Lemma 7.7, Lemma 7.8 and Theorem 7.9 in full given Lemma 7.5, which it cites to Simchi-Levi and Wei, and omits the proofs of Theorems 7.3 and 7.4 and the identity (7.30) behind Lemma 7.8. Formalizing Lemma 7.5 and (7.30) means formalizing the structure of maximum flows on a cycle, which is reusable for the later results of Simchi-Levi and Wei on the long chain's performance relative to full flexibility and for the multi-echelon flexibility models the chapter cites.

Difficulty

The obvious approach to Theorem 7.9 is to compare $C_n$ with an arbitrary 2-flexibility design directly. Nothing in the definitions supports that: the two designs share no structure beyond their degree sequences. The book's argument instead routes everything through the long chain's own edges. Lemma 7.5 gives supermodularity only for subsets of $C_n$, and the decomposition of an arbitrary 2-flexibility design into disjoint cycles, each a relabeled long chain on a subsystem, is what allows the comparison. A solver must therefore prove that a 2-regular bipartite graph is a disjoint union of even cycles, that exchangeability makes every relabeling of a cycle worth the same as $C_{n_j}$ on its subsystem, and that the performance of a disjoint union is the sum of the performances of its parts.

Lemma 7.5 itself is where the combinatorics lives. It says that on the cycle $C_n$ the maximum flow is supermodular in the flexible edges, and the proof in Simchi-Levi and Wei goes through the structure of augmenting paths on a cycle. The natural first idea, that supermodularity follows from some general property of maximum flows, is false: maximum flow is not supermodular in arbitrary edge sets, and the lemma is specific to subsets of a single cycle.

Lemma 7.7 is where exchangeability is used, and it is used in a way that is easy to state and tedious to formalize: removing the edge $(2, 1)$ from $L^n_{k+1}$ leaves a design that is $L^n_k$ only after the pair $1$ is moved to the end, so the argument needs the invariance of $[E]$ under relabeling the products and plants by a common permutation. The book notes that Lemma 7.7, unlike Lemma 7.5, is false realization by realization.

For the transshipment theorems, the book differentiates a density formula by Leibniz's rule. Under the weaker hypothesis stated here, laws without atoms and with finite means, the derivative of $\mathbb{E}[Y_{ji}]$ in $S_i$ has to be obtained by dominated convergence from the pointwise derivative of a piecewise-linear function whose kinks lie on null sets.

Formalization scope

perf is a supremum over a set of reals, nonempty because $y = 0$ is feasible when $d \ge 0$ and $C \ge 0$, and bounded by $\sum_i d_i$; the demand is nonnegative for every outcome and the capacity nonnegative in BalancedSystem, and Lemma 7.5 carries these as hypotheses. The supremum is attained, but the definition does not assert it. Expected performance is a Lebesgue integral; the demand is integrable by assumption and $P(d, E)$ is $1$-Lipschitz in $d$, so the integrand is integrable, and a solver must prove this measurability rather than assume it.

Exchangeability is the equality of the laws of $(D_{\sigma(i)})_i$ and $(D_i)_i$ for every permutation $\sigma$. Designs are finite sets of pairs of Fin n; the chains are defined with finRotate, so indices wrap modulo $n$ and the closing edge of $C_n$ is $(1, n)$ in the book's numbering, which is the edge its proofs and Figure 7.3(c) use. Lemma 7.8 involves open chains on subsystems of sizes $n$ and $n - 1$; these are designs on Fin k evaluated on the first $k$ coordinates of the demand (subDemand, subPerf).

Theorem 7.1 states the optimal costs as infima of the newsvendor cost over all base-stock levels, on Mathlib's gaussianReal; a nonpositive pooled variance gives a degenerate law, for which the inequality still holds, so the statement is not trivialized by that convention. The transshipment theorems take the two demand laws as probability measures on $\mathbb{R}$ with no atoms (Theorem 7.2) and finite, positive means; the quantity $Y_{ji}$ is defined for all outcomes and the service levels are probabilities and expectations under the product law.

The definition module is shared by all eleven items. Beyond the milestones, formalizing the identity (7.30) as its own lemma and the disjoint-union additivity of perf would be natural contributions.

Selected references

• L. V. Snyder and Z.-J. M. Shen, Fundamentals of Supply Chain Theory, 2nd ed., Wiley, 2019, Chapter 7. https://doi.org/10.1002/9781119584445
• G. D. Eppen, Effects of centralization on expected costs in a multi-location newsboy problem, Management Science 25(5), 1979. https://doi.org/10.1287/mnsc.25.5.498
• G. Tagaras, Effects of pooling on the optimization and service levels of two-location inventory systems, IIE Transactions 21(3), 1989. https://doi.org/10.1080/07408178908966208
• W. C. Jordan and S. C. Graves, Principles on the benefits of manufacturing process flexibility, Management Science 41(4), 1995. https://doi.org/10.1287/mnsc.41.4.577
• D. Simchi-Levi and Y. Wei, Understanding the performance of the long chain and sparse designs in process flexibility, Operations Research 60(5), 2012. https://doi.org/10.1287/opre.1120.1082

Why orders swing more than sales

Procter & Gamble observed in the 1990s that the orders its distributors placed for diapers were far more variable than the retail sales of diapers, and that its own orders to suppliers were more variable still, although the end demand for diapers is about as stable as demand gets. The phenomenon, a growing amplification of variability as one moves upstream in a supply chain, is the bullwhip effect. Lee, Padmanabhan and Whang (1997) argued that it is not a symptom of irrational behaviour: four rational responses of an inventory manager to their own environment each produce it. Chapter 13 of Snyder and Shen's Fundamentals of Supply Chain Theory (2019) makes three of the four quantitative, following Chen, Drezner, Ryan and Simchi-Levi (2000) for demand signal processing, Lee et al. for the rationing game, and Cachon (1999) for order batching. This mission formalizes those three models and the theorems the chapter proves about them.

Setting

Demand signal processing. A retailer faces a demand process $D_t$, $t \in \mathbb{Z}$, that follows the stationary first-order autoregressive model

$$D_t = d + \rho D_{t-1} + \epsilon_t,$$

with a constant $d \ge 0$, a correlation constant $-1 < \rho < 1$, and errors $\epsilon_t$ that are independent $N(0, \sigma^2)$ variables, each independent of the demands before period $t$. In steady state every $D_t$ has the law $N\big(d/(1-\rho),\ \sigma^2/(1-\rho^2)\big)$. The retailer replenishes with a lead time of $L$ periods under a base-stock policy but does not know the demand parameters, so it estimates the lead-time demand from a moving average of the previous $m \ge 1$ demands:

$$\hat\mu^L_t = \frac{L}{m}\sum_{i=1}^m D_{t-i}, \qquad \hat\sigma^L_{et} = C\sqrt{\frac{1}{m}\sum_{i=1}^m e_{t-i}^2}, \qquad e_t = D_t - \hat\mu^1_t,$$

and sets the base-stock level $S_t = \hat\mu^L_t + z_\alpha \hat\sigma^L_{et}$, where $z_\alpha$ is a safety factor. The book writes the constant in $\hat\sigma^L_{et}$ as $C_{L\rho}$ and does not give its form; here it is a free parameter $C$. Each period the retailer orders $Q_t = S_t - S_{t-1} + D_{t-1}$, which may be negative. In Lean the process is the structure AR1Demand, whose fields are the parameters, the errors, the demands, the recursion, the independence properties and the stationary law; muHat, err, sigmaHat, baseStock and order are the five quantities above.

Order batching. $N$ retailers face independent $N(\mu, \sigma^2)$ demands in every period and each orders once every $R \ge 1$ periods, the order being its demand over the previous $R$ periods. The supplier's order in a given period is the total ordered by the retailers whose ordering day falls in that period. Three patterns are compared: random ordering, in which each retailer's day is uniform over the $R$ days, so the number $X$ of retailers ordering on a given day is binomial$(N, 1/R)$; positively correlated ordering, in which all retailers order on the same day, so $X = N$ with probability $1/R$ and $0$ otherwise; and balanced ordering, in which the retailers are spread as evenly as possible, so with $N = MR + k$, $0 \le k < R$, $X$ is $M+1$ with probability $k/R$ and $M$ otherwise. The structure BatchOrders P N R mu sigma carries the demands, the ordering count $X$ independent of them, and supplierOrder, the sum of the last $R$ demands of retailers $1, \dots, X$; each pattern enters a theorem as a hypothesis on the law of $X$.

Rationing game. Two identical retailers face single-period demand with distribution function $F$, holding cost $h$ and stockout penalty $p$, so the newsvendor quantity $Q^*$ satisfies $F(Q^*) = p/(h+p)$. With probability $r$ the supplier can deliver only $A_1 < 2Q^*$ units in total and allocates them pro rata to the orders, retailer 1 receiving $A_1 Q_1/(Q_1 + Q_2)$; with probability $1 - r$ supply is unlimited. Retailer 1's expected cost when the retailers order $Q_1$ and $Q_2$ is

$$g_1(Q_1) = (1-r)\,\mathrm{nv}(Q_1) + r\,\mathrm{nv}\!\Big(\frac{A_1 Q_1}{Q_1 + Q_2}\Big),$$

with $\mathrm{nv}$ the newsvendor cost; this is rationingCost.

Formalization targets

Goal: Theorem 13.2, demand signal processing

$$\frac{\mathrm{Var}[Q_t]}{\mathrm{Var}[D_t]} \;\ge\; 1 + \Big(\frac{2L}{m} + \frac{2L^2}{m^2}\Big)(1 - \rho^m),$$

with equality when $z_\alpha = 0$. This is bullwhip_signal_processing. The bound exceeds $1$ whenever $L > 0$, whatever the value of $\rho$: a lead time and a moving-average forecast are enough to produce the effect.

Supporting targets

The chapter's own route to the goal, each a milestone: the steady-state moments (13.2) to (13.4), $\mathbb{E}[D_t] = d/(1-\rho)$, $\mathrm{Var}[D_t] = \sigma^2/(1-\rho^2)$ and $\mathrm{Cov}[D_t, D_{t-k}] = \rho^k \mathrm{Var}[D_t]$; the identity $Q_t = (1 + L/m) D_{t-1} - (L/m) D_{t-m-1} + z_\alpha(\hat\sigma^L_{et} - \hat\sigma^L_{e,t-1})$; Lemma 13.1, $\mathrm{Cov}[D_{t-i}, \hat\sigma^L_{et}] = 0$ for $1 \le i \le m$; the vanishing of the cross term (13.12); and the variance of the demand part, $\big(1 + (2L/m + 2L^2/m^2)(1 - \rho^m)\big)\mathrm{Var}[D_t]$.

Order batching, Theorem 13.4: under the three patterns the supplier's order has mean $N\mu$ and

$$\mathrm{Var}[Q^c_t] \ge \mathrm{Var}[Q^r_t] \ge \mathrm{Var}[Q^b_t] \ge N\sigma^2,$$

through the three variance formulas $N\sigma^2 + \mu^2 N(R-1)$, $N\sigma^2 + \mu^2 N^2 (R-1)$ and $N\sigma^2 + \mu^2 k(R-k)$.

The rationing game, Theorem 13.3: if $Q > 0$ is a symmetric Nash equilibrium, that is, $Q$ minimizes $g_1$ over positive order quantities when the other retailer orders $Q$, then $Q > Q^*$.

Significance

The three theorems are the quantitative core of the chapter. Theorem 13.2 is the single-stage building block that Theorems 13.6 and 13.7 later iterate along a serial chain, giving the product-form and the exponential lower bounds on the amplification at stage $k$; its comparative statics, the bound decreasing in $m$ and increasing in $L$, are the basis of the remedies the chapter recommends (shorter lead times, smoother forecasts, sharing point-of-sale data). Theorem 13.4 ranks the ordering patterns and justifies the advice to balance ordering days when batching cannot be avoided. Theorem 13.3 shows that pro-rata rationing alone inflates orders; the book is careful to note that inflated orders are not by themselves inflated variances, and that the variance statement for this model is due to Rong, Shen and Snyder (2017).

None of these results has a machine-checked proof. The book's proofs of Theorems 13.2 and 13.4 are complete but informal, and the proof of Lemma 13.1 is omitted with a citation to Ryan's 1997 thesis; formalizing it requires a self-contained argument. The variance decomposition of $Q_t$ and the conditioning argument for Theorem 13.4 are reusable for the multistage results of Sect. 13.2.5, which are natural follow-up missions on the same definitions.

Difficulty

The obvious computation of $\mathrm{Var}[Q_t]$ expands the order into its demand part and its safety-stock part and hopes the cross term disappears. It does, but not for a reason visible in the formulas: $\hat\sigma^L_{et}$ is a square root of a sum of squares of forecast errors, a nonlinear function of $m + m$ demands, and its covariance with a single demand is zero only because the errors are jointly Gaussian with mean zero and $\hat\sigma$ is an even function of them, so the covariance is the expectation of an odd function of a centred Gaussian vector. That is Lemma 13.1, and the vanishing of the cross term needs two further covariances, $\mathrm{Cov}[D_{t-1}, \hat\sigma^L_{e,t-1}]$ and $\mathrm{Cov}[D_{t-m-1}, \hat\sigma^L_{et}]$, which the book reduces to the lemma through the recursion (the second reduction divides by $\rho$) but which hold for every $\rho$ by the same symmetry. A solver must set up the joint Gaussian structure of the demand vector and prove the odd-function argument; nothing in Mathlib does this directly.

The second obstacle is that the moments (13.2) to (13.4) are not assumed but derived: the structure carries the stationary law of each $D_t$ and the independence of $\epsilon_t$ from the past, and the autocovariance $\rho^k \mathrm{Var}[D_t]$ has to be obtained from the recursion by induction on the lag, with integrability supplied by the Gaussian laws.

For Theorem 13.4 the work is the conditioning on $X$: given $X = x$ the supplier's order is a sum of $xR$ independent normals, so its conditional mean is $xR\mu$ and conditional variance $xR\sigma^2$, and the total variance is $\mathbb{E}[\mathrm{Var}[Q \mid X]] + \mathrm{Var}[\mathbb{E}[Q \mid X]]$. The order is defined by a sum over retailers $i < X$, so the independence of $X$ from the demands has to be used through the indicator structure rather than through a conditional-expectation library result.

For Theorem 13.3 the argument is a first-order condition. It requires that the newsvendor cost be differentiable with derivative $(h+p)F(y) - p$, which holds when $F$ is continuous, and that the symmetric equilibrium be an interior minimizer, which is why $Q > 0$ and the minimization over $Q_1 > 0$ are hypotheses.

Formalization scope

Time is indexed by $\mathbb{Z}$ so that $D_{t-m-1}$ exists for every $t$. AR1Demand asserts the recursion for every outcome, the independence of the whole error family, the independence of $\epsilon_t$ from $(D_s)_{s < t}$, and the stationary law of every $D_t$; these are the "steady-state" assumptions the book makes in words. The structure is satisfiable: the stationary Gaussian AR(1) process on a full-measure set of error sequences has all these properties. The constant $C_{L\rho}$ is a free real parameter $C$; no theorem depends on its value.

The goal divides by $\mathrm{Var}[D_t]$, which is $\sigma^2/(1-\rho^2) > 0$ under the structure's hypotheses $\sigma > 0$ and $|\rho| < 1$, so the ratio is a genuine quotient. Mathlib's ProbabilityTheory.variance and covariance are used; both are the ordinary real quantities for square-integrable variables, which every variable here is, $\hat\sigma^L_{et}$ included.

In BatchOrders the demands are indexed by Fin N × Fin R, the count $X$ is a natural-valued random variable bounded by $N$ and independent of the demand family, and supplierOrder sums the $R$ demands of retailers $1, \dots, X$, the book's "without loss of generality" choice. The laws of $X$ are hypotheses on point probabilities P.real {ω | X ω = j}; with $R \ge 1$ each of the three families of hypotheses is satisfiable by a structure with the corresponding law. The subtractions $R - 1$ and $R - k$ are real.

In the rationing game the demand law is a probability measure on $\mathbb{R}$ whose distribution function is continuous and strictly increasing on $[0, \infty)$; the newsvendor loss is assumed integrable at every order quantity. The pro-rata allocation uses Lean's total division, which is never at $0$ in the theorem since $Q_1 + Q_2 > 0$.

Beyond the ten milestones, the multistage Theorems 13.6 and 13.7 and the centralized-information bound of Theorem 13.5 are welcome as extensions on the same AR1Demand.

Selected references

• L. V. Snyder and Z.-J. M. Shen, Fundamentals of Supply Chain Theory, 2nd ed., Wiley, 2019, Chapter 13. https://doi.org/10.1002/9781119584445
• H. L. Lee, V. Padmanabhan and S. Whang, Information distortion in a supply chain: the bullwhip effect, Management Science 43(4), 1997. https://doi.org/10.1287/mnsc.43.4.546
• F. Chen, Z. Drezner, J. K. Ryan and D. Simchi-Levi, Quantifying the bullwhip effect in a simple supply chain: the impact of forecasting, lead times, and information, Management Science 46(3), 2000. https://doi.org/10.1287/mnsc.46.3.436.12069
• G. P. Cachon, Managing supply chain demand variability with scheduled ordering policies, Management Science 45(6), 1999. https://doi.org/10.1287/mnsc.45.6.843
• Y. Rong, Z.-J. M. Shen and L. V. Snyder, The impact of ordering behavior on order-quantity variability: a study of forward and reverse bullwhip effects, Naval Research Logistics 64(1), 2017. https://doi.org/10.1002/nav.21757

Motivation

When a system is controlled indefinitely and decisions are frequent — a router admitting packets, a queue accepting jobs, a machine being maintained — discounting future rewards is often unjustified, and what matters is the long-run average reward per period. Puterman's Chapter 8 (doi:10.1002/9780470316887) develops the theory of this criterion, and its central object is a single equation, the average reward optimality equation $0=\max_{a\in A_s}\{r(s,a)-g+\sum_jp(j\mid s,a)h(j)-h(s)\}$, whose unknowns are a scalar gain $g$ and a bias function $h$. For unichain models, in which every stationary policy generates a Markov chain with one recurrent class, this equation determines the optimal gain and an optimal stationary policy. The results go back to Howard (Dynamic Programming and Markov Processes, MIT Press, 1960) for the recurrent case and to Blackwell (Discrete dynamic programming, Annals of Mathematical Statistics 33, 1962, doi:10.1214/aoms/1177704593) and Derman for the general finite case; Puterman's Section 8.4 proves them through the discounted theory of mission II, by letting the discount factor tend to one.

Setting

The model is stationary (Assumption 8.0.1): a finite set $S$ of states, for each $s$ a finite nonempty set $A_s$ of actions, a reward $r(s,a)$ and transition probabilities $p(j\mid s,a)$, none depending on the decision epoch. A policy $\pi\in\Pi^{HR}$ may randomize and may depend on the whole history; the deterministic stationary policy $d^\infty$ applies the decision rule $d:S\to A$ at every epoch. Its transition matrix is $P_d(i,j)=p(j\mid i,d(i))$.

For a policy $\pi$, $v^\pi_{N+1}(s)=\mathbb E^\pi_s[\sum_{t=1}^Nr(X_t,Y_t)]$ is the expected reward over $N$ epochs. Since the limit of $N^{-1}v^\pi_{N+1}(s)$ need not exist (Example 8.1.1), the chapter works with the lim sup and lim inf average rewards $g^\pi_+(s)$ and $g^\pi_-(s)$, and with $g^*_\pm(s)=\sup_{\pi}g^\pi_\pm(s)$. A policy $\pi^*$ is average optimal when $g^{\pi^*}_-(s)\ge g^\pi_+(s)$ for all $s$ and $\pi$, the strongest of the three criteria of Section 8.1.2.

The optimality residual is $B(g,h)(s)=\max_{a\in A_s}\{r(s,a)-g+\sum_jp(j\mid s,a)h(j)-h(s)\}$, and the optimality equation is $B(g,h)=0$. A decision rule is $h$-improving when it attains $\max_{a\in A_s}\{r(s,a)+\sum_jp(j\mid s,a)h(j)\}$ at every state. A transition matrix is unichain when it consists of a single recurrent class plus a possibly empty set of transient states, and the MDP is unichain when $P_d$ is unichain for every deterministic decision rule.

Formalization targets

Goal — Theorem 8.4.5 (printed p. 361)

For a finite unichain model: (a) some deterministic stationary policy is average optimal; (b) the optimality equation $B(g^*,h^*)=0$ has a solution, and (d) its scalar satisfies $g^*_+(s)=g^*_-(s)=g^*$ for every $s$; (c) for every solution, every $h^*$-improving decision rule gives an average optimal stationary policy.

Theorem 8.4.1 (printed p. 356)

If $B(g,h)\le 0$ then $g\ge g^*_+$; if $B(g,h)\ge 0$ then $g\le\sup_{d}g^{d^\infty}_-\le g^*_-$; if $B(g,h)=0$ then $g^*_+=g^*_-=g$.

Theorem 8.4.3 (printed p. 358)

In a finite unichain model $B(g,h)=0$ has a solution, and every solution has the same $g$.

Theorem 8.4.4 (printed p. 361)

If $B(g^*,h^*)=0$ and $d^*$ is $h^*$-improving, then $(d^*)^\infty$ is average optimal.

Significance

Theorem 8.4.1(c) is what the source calls "one of the most important results for average reward models": a solution of the optimality equation with constant $g$ pins down the optimal gain under every criterion at once, so that in finite unichain models the three optimality criteria of Section 8.1.2 coincide. Theorem 8.4.3 guarantees such a solution exists, and Theorem 8.4.4 reads an optimal policy off it. Together, Theorem 8.4.5 reduces the infinite-horizon average reward problem over all history-dependent randomized policies to a finite system of equations in $(g,h)$, which is what policy iteration, value iteration and linear programming solve in Sections 8.5 to 8.8.

The results are classical and proved. Formalizing them fixes the chain-structure hypothesis in a checkable form and pins down which criterion "average optimal" means, two places where the literature is loose. The platform's MarkovDecisionProcesses series has the finite-horizon (mission I) and discounted (mission II) models; this mission adds the undiscounted stationary model, the gains, and the unichain classification, on which Chapter 9's multichain optimality equations and Chapter 10's sensitive discount optimality can be built.

Difficulty

The obvious argument for Theorem 8.4.3 is to take the discounted optimal value $v^*_\lambda$ of mission II and let $\lambda\uparrow 1$. It fails as stated because $v^*_\lambda$ blows up like $(1-\lambda)^{-1}$; what converges is the Laurent expansion $v^{d^\infty}_\lambda=(1-\lambda)^{-1}ge+h+o(1)$ of the value of a fixed stationary policy, Corollary 8.2.4, and that expansion needs the limiting matrix $P_d^*$ and the deviation matrix $H_{P_d}$ of a unichain chain. So the proof must first develop the Markov chain theory of Section 8.2 and Appendix A, choose a subsequence of discount factors along which one policy is discount optimal (possible because $D^{MD}$ is finite), and only then pass to the limit in the discounted optimality equation.

Theorem 8.4.1 looks elementary and hides the analytic step: iterating $ge\ge r_d+(P_d-I)h$ along an arbitrary history-dependent policy and dividing by $N$ requires the telescoping term $N^{-1}(P^\pi_N-I)h$ to vanish, which uses boundedness of $h$, and requires the reduction from history-dependent randomized to Markov randomized policies (Theorem 8.1.2). For Theorem 8.4.4 the step is Corollary 8.2.7, that $r_d-ge+(P_d-I)h=0$ forces the gain of $d^\infty$ to be $g$, which is the multiplication by $P_d^*$ that annihilates $(P_d-I)$.

The traps are in the definitions. Recurrence and the unichain property must be stated so that the source's Example 8.4.3 comes out as the book says — the policy using $a_{1,1}$ has the absorbing state $s_2$ as its single recurrent class — and the optimality residual must use the lim sup / lim inf gains, since a definition through a limit that need not exist would be a junk value on the policies of Example 8.1.1.

Formalization scope

State and action spaces are Fintypes and admissible actions are nonempty Finsets, as in missions I and II; the stationary model is a new structure because mission II's DiscountedMDP bundles a discount factor, and carries the same data otherwise. Policies are history-dependent and randomized, so "average optimal" has its full strength; a stationary policy is the deterministic one built from a decision rule. Expected total reward is defined by the policy evaluation recursion, as in the earlier missions, rather than through a measure on trajectories.

Gains are Filter.limsup and Filter.liminf of $N^{-1}v^\pi_{N+1}(s)$ on $\mathbb R$; these are the source's because the sequence is bounded by $\max|r|$, and the suprema $g^*_\pm$ over the nonempty family of policies are genuine real suprema for the same reason. The residual $B(g,h)$ is a Finset.sup' over the admissible actions. Recurrence is "every state reachable from $i$ reaches $i$" and unichain is "any two recurrent states communicate", the definitions of Appendix A for finite chains, applied to $P_d$ for every admissible deterministic decision rule.

Restrictions relative to the printed text, all noted in the items: Theorem 8.4.1 is stated for finite $S$ where the source says countable, since the chapter's standing assumption and the model are finite; the gain $g^{d^\infty}$ of a stationary policy in (8.4.5) is written as its lim inf gain, which equals it; and the chain $ge=g^*=g^*_+=g^*_-$ of (8.4.6) is stated through $g^*_+$ and $g^*_-$, since $g^*$ presupposes existing limits. Nothing is trivialized: the existential in Theorem 8.4.5(a) has to produce a decision rule, and $B(g,h)=0$ with a junk maximum is impossible since every $A_s$ is nonempty. Welcome contributions beyond the milestones: Theorem 8.1.2 (reduction to Markov policies), Corollary 8.2.7 (the gain of a stationary policy from the evaluation equations), and the equivalence of the three optimality criteria in finite models.

Selected references

• Martin L. Puterman, Markov Decision Processes: Discrete Stochastic Dynamic Programming, Wiley, 1994, Chapter 8. doi:10.1002/9780470316887
• Ronald A. Howard, Dynamic Programming and Markov Processes, MIT Press, 1960.
• David Blackwell, Discrete dynamic programming, Annals of Mathematical Statistics 33 (1962). doi:10.1214/aoms/1177704593
• Cyrus Derman, Finite State Markovian Decision Processes, Academic Press, 1970.
• Paul J. Schweitzer and Awi Federgruen, The functional equations of undiscounted Markov renewal programming, Mathematics of Operations Research 3 (1978). doi:10.1287/moor.3.4.308

Motivation

Every data-driven decision problem faces the same trap. A decision-maker estimates a risk functional $R(P,\ell) = \mathbb{E}_P[\ell(\xi)]$ from a nominal distribution $\hat P_N$ built from $N$ training samples, then optimizes a loss function $\ell$ against $\hat P_N$ instead of the unknown true distribution $P$. Because the optimizer adapts to the noise in $\hat P_N$, the in-sample risk of the optimizer systematically understates its true, out-of-sample risk — a phenomenon Smith and Winkler named the optimizer's curse (Smith & Winkler, Management Science, 2006). The remedy explored here is to hedge against a whole neighborhood of plausible distributions around $\hat P_N$, rather than trusting the point estimate. Kuhn, Mohajerin Esfahani, Nguyen and Shafieezadeh-Abadeh's INFORMS TutORials chapter (2019) develops this neighborhood using the Wasserstein distance, and the present mission formalizes its foundational duality theory: the machinery every later result in the chapter (finite-sample guarantees, elliptical tractability, regularization) builds on.

Setting

Fix a norm $\|\cdot\|$ on a finite-dimensional real vector space $E$ (representing $\mathbb{R}^m$). For $p \in [1,\infty)$, the type-$p$ Wasserstein distance between two Borel probability measures $Q, Q'$ on $E$ is

$$W_p(Q,Q') = \left(\inf_{\pi \in \Pi(Q,Q')} \int_{E\times E} \|\xi-\xi'\|^p\, \pi(d\xi,d\xi')\right)^{1/p},$$

where $\Pi(Q,Q')$ is the set of couplings of $Q$ and $Q'$ — joint probability measures on $E \times E$ whose marginals are $Q$ and $Q'$. The optimal $\pi$ can be read as a transportation plan moving one pile of dirt ($Q$) into another ($Q'$) at minimum cost, which is why $W_p$ is also called the earth mover's distance; the underlying linear program was formalized by Kantorovich (1942) after Monge's 1781 original.

Given $N$ training samples $\hat\xi_1,\dots,\hat\xi_N$, the empirical distribution is $\hat P_N = \frac1N\sum_{i=1}^N \delta_{\hat\xi_i}$. Centered at $\hat P_N$, the Wasserstein ambiguity set of radius $\varepsilon \ge 0$ is

$$B_{\varepsilon,p}(\hat P_N) = \{Q \in \mathcal{P}(\Xi) : W_p(Q,\hat P_N) \le \varepsilon\},$$

where $\Xi \subseteq E$ is a closed set known to contain the support of the true distribution. The worst-case risk of a loss function $\ell$ is

$$R_{\varepsilon,p}(\hat P_N,\ell) = \sup_{Q \in B_{\varepsilon,p}(\hat P_N)} \mathbb{E}_Q[\ell(\xi)],$$

and minimizing it over a class of admissible loss functions $\mathcal{L}$ is a distributionally robust optimization problem. $\varepsilon$ measures the estimation error one insures against; a larger ambiguity set gives a more conservative (and more expensive) guarantee.

Formalization targets

Goal — Theorem 7, strong duality

$$R_{\varepsilon,p}(\hat P_N,\ell) = \inf_{\gamma \ge 0}\ \mathbb{E}_{\hat P_N}[\ell_\gamma(\xi)] + \gamma\varepsilon^p,\qquad \ell_\gamma(\xi) = \sup_{z\in\Xi} \ell(z) - \gamma\|z-\xi\|^p.$$

This is the Lagrangian dual of the worst-case risk evaluation problem, with $\gamma$ the multiplier of the Wasserstein constraint $W_p(Q,\hat P_N)\le\varepsilon$: it converts a supremum over an infinite-dimensional space of measures into a one-dimensional minimization of the Moreau-Yosida regularization $\ell_\gamma$. Every tractability result later in the chapter (finite convex reformulations, SDP relaxations) specializes this duality by choosing a loss class for which $\ell_\gamma$ is computable.

Supporting dual representations of $W_p$ — Theorems 1 and 2

$$W_p^p(Q,Q') = \sup\left\{\int \psi\,dQ' - \int \varphi\,dQ : \varphi,\psi \text{ bounded continuous},\ \psi(\xi)-\varphi(\xi') \le \|\xi-\xi'\|^p\right\}$$ $$W_1(Q,Q') = \sup_{\mathrm{Lip}(\varphi)\le 1} \int \varphi\,dQ - \int \varphi\,dQ'$$

These identify $W_p$ as a linear program's strong dual (Theorem 1) and, for $p=1$, specialize it to the Kantorovich-Rubinstein form (Theorem 2), which is what lets the worst-case-risk analysis reason about Lipschitz loss functions directly.

Upper and lower bounds — Theorems 5 and 6

$$R_{\varepsilon,p}(\hat P_N,\ell) \le R(\hat P_N,\ell) + \varepsilon\cdot\mathrm{Lip}(\ell)$$ $$R_{\varepsilon,p}(\hat P_N,\ell) \ge \sup\left\{\tfrac1N\textstyle\sum_i \ell(\hat\xi_i+\theta_i) : \hat\xi_i+\theta_i\in\Xi,\ \tfrac1N\textstyle\sum_i\|\theta_i\|^p\le\varepsilon^p\right\}$$

These are the tractable, easily-computed bracket that Theorems 7 and 10 later show is tight in important special cases.

Exact case — Theorem 10

$$\Xi = \mathbb{R}^m,\ \ell \text{ convex},\ p=1 \implies R_{\varepsilon,1}(\hat P_N,\ell) = R(\hat P_N,\ell) + \varepsilon\,\mathrm{Lip}(\ell)$$

Theorem 5's inequality becomes exact under convexity — the cleanest closing corollary of the duality theory, obtained from Theorem 7 by evaluating the Moreau-Yosida regularization of a convex function explicitly.

Significance

Theorem 7 is the hinge on which the entire computational program of Wasserstein distributionally robust optimization turns: every tractable reformulation in the source chapter (piecewise-concave losses via conic duality, quadratic losses via semidefinite programming, the shrinkage-estimator connection) is obtained by substituting a specific loss class into the right-hand side of Theorem 7 and showing the resulting Moreau-Yosida regularization is computable. Kuhn et al. themselves derive it as a corollary of Blanchet & Murthy (2019) and Gao & Kleywegt (2016) for the empirical case, generalized to Polish spaces by Blanchet & Murthy and Gao & Kleywegt independently — the paper cites [12] and [37] for the general statement. Formalizing it is what makes every later, more computational result in the chapter — the ones a solver is more likely to reach for next — rest on a mechanically verified foundation rather than a citation chain.

Status. The mathematical result is well established (multiple independent published proofs cited above); nothing here is open research. What this mission contributes is the first machine-checked formal statement of the duality theorem and its supporting dual representations (Theorems 1, 2, 5, 6, 10) on the Prove2Me platform — none of Wp's dual representation, the Wasserstein ambiguity set, or the worst-case risk functional exist there prior to this mission (see Formalization scope).

Difficulty

The obvious proof strategy — write down the Lagrangian of the semi-infinite program (6), swap the order of the outer supremum over $Q$ and the inner minimization over the multiplier $\gamma$, and invoke ordinary Lagrangian strong duality — fails because (6) is an infinite- dimensional linear program over measures, not a finite convex program: there is no compact feasible set or Slater point in a form that ordinary finite-dimensional duality applies to directly. The actual proof goes through the dual representation of the Wasserstein distance itself (Theorem 1, which is why it is a prerequisite milestone), reformulating the constraint $W_p(Q,\hat P_N)\le\varepsilon$ via its own dual variables and swapping the resulting sup-inf using minimax theorems for semi-infinite programs, not ordinary Lagrangian duality for finite programs.

Formalization scope

$E$ is a generic finite-dimensional real normed space (NormedAddCommGroup, NormedSpace ℝ, Borel-measurable), representing $\mathbb{R}^m$ with the paper's arbitrary fixed norm as a parameter rather than fixing the Euclidean norm. A coupling is formalized directly via MeasureTheory.Measure.map: π.map Prod.fst = Q ∧ π.map Prod.snd = Q'. Constrained infima/suprema (over couplings, over the ambiguity set, over Lipschitz test functions, over perturbation matrices) use Mathlib's guarded-binder idiom ⨅ x (_ : P x), f x, which correctly returns $\top$ (resp. $\bot$) outside the feasible set rather than a finite junk value.

Two deliberate, disclosed conventions keep the extremal-value definitions faithful without extended-real integration machinery, both recorded in MODERATION_NOTES.md:

1. worstCaseRisk and the dual representations (Theorems 1, 2) are valued in EReal, not ℝ, so an unbounded supremum is recorded as $+\infty$ rather than collapsed to Mathlib's real-valued junk value 0 on an unbounded family.
2. The goal theorem (7) and its Moreau-Yosida regularization restrict the loss function to bounded continuous $\ell$ (BoundedContinuousFunction E ℝ), narrower than the paper's general upper-semicontinuous, $\hat P_N$-integrable loss class $\mathcal{L}$ (Assumption 1). This keeps $\ell_\gamma(\xi) = \sup_{z\in\Xi}\ell(z)-\gamma\|z-\xi\|^p$ a finite real number for every nonempty $\Xi$, so the right-hand side's Bochner integral is well-posed; the milestones (Theorems 5, 6, 10) keep the more general real-valued (not necessarily bounded) loss class, since their statements do not require evaluating a pointwise supremum over $\Xi$.
3. Ξ is required closed in Theorems 5, 6 and 7, matching the paper's own standing assumption (p. 6: "we let $\Xi\subseteq\mathbb{R}^m$ be a closed set that is known to contain the support of $P$") for the whole worst-case-risk framework, which is used silently in the paper wherever a theorem takes $\Xi$ as an argument but was not carried into these theorems' own hypothesis lists in an earlier draft.
4. The goal theorem (7) additionally requires $\hat P_N$ itself supported on $\Xi$ ($\hat P_N(\Xi^c)=0$, the same "supported on $\Xi$" convention ambiguitySet uses for $Q\in\mathcal P(\Xi)$), which the paper's framework presupposes for the nominal distribution throughout §2. Combined with $\ell$ bounded, this makes $\ell_\gamma$ bounded on the full-measure set $\Xi$ (above by $\sup\ell$ unconditionally, below by $\ell(\xi)$ itself via $z=\xi$ for $\xi\in\Xi$), which is what makes the right-hand side's integral genuinely well-posed rather than liable to Mathlib's non-integrable junk value $0$.

There is no trivializing formalization risk from a vacuous hypothesis: Ξ.Nonempty and 0 < N are both required exactly where the paper's own indexing and support assumptions require them, and every extremal value uses the extended-real convention above rather than a convention that would make an inequality vacuously true.

No definition in this mission exists on the platform prior to this series (GET /theorems?q=Wasserstein, q=Kantorovich, q=optimal transport, q=coupling return only unrelated discrete/finite-type constructions); all seven definitions and six theorems are drafted fresh. WassersteinDRO.Duality.wassersteinDistance, .ambiguitySet and .worstCaseRisk are the substrate every later mission in this five-part series (Gelbrich tractability, finite-sample guarantees, regularization, shrinkage estimation) either imports directly or redefines locally per the series' reuse rule.

Selected references

• Kuhn, D., Mohajerin Esfahani, P., Nguyen, V. A., & Shafieezadeh-Abadeh, S. (2019). Wasserstein Distributionally Robust Optimization: Theory and Applications in Machine Learning. INFORMS TutORials in Operations Research, 130–166. https://doi.org/10.1287/educ.2019.0198
• Villani, C. (2009). Optimal Transport: Old and New. Springer. (Cited as [108] for Theorems 1 and 2.)
• Smith, J. E., & Winkler, R. L. (2006). The optimizer's curse: Skepticism and postdecision surprise in decision analysis. Management Science, 52(3), 311–322. https://doi.org/10.1287/mnsc.1050.0451
• Gao, R., & Kleywegt, A. J. (2016). Distributionally Robust Stochastic Optimization with Wasserstein Distance. arXiv:1604.02199.
• Blanchet, J., & Murthy, K. (2019). Quantifying Distributional Model Risk via Optimal Transport. Mathematics of Operations Research, 44(2), 565–600. https://doi.org/10.1287/moor.2018.0936

Motivation

Reinforcement learning formalizes a scenario supervised learning cannot: an agent that actively interacts with an environment, choosing actions that change both the state it observes next and the reward it receives, rather than passively receiving an i.i.d. labeled sample. Every practical treatment of this scenario — from classical dynamic programming to modern deep reinforcement learning — is built on the Markov decision process (MDP), a model in which the effect of an action depends only on the current state, not on the full history that led to it. Two questions define the theory this mission covers: given a fixed way of acting (a policy), what value does it obtain, and how is that value actually computed rather than merely characterized as the solution of a fixed-point equation? Mohri, Rostamizadeh and Talwalkar's chapter 17 answers both for the stationary, infinite-horizon discounted case, and this mission targets its two central results: that a fixed policy's value is not just characterized but uniquely determined by a linear system with an explicit closed-form solution (Theorem 17.10), and that the optimal value function — obtained instead by choosing the best action at every state — can be computed by an iterative algorithm guaranteed to converge regardless of where it starts (Theorem 17.11).

Setting

A (finite) Markov decision process consists of a finite set of states $S$, a finite set of actions $A$, a transition kernel $P[s'\mid s,a]$ giving the distribution over the next state $s'$ after taking action $a$ at state $s$, and an expected reward $\mathbb E[r(s,a)]$ for that transition. A (stationary) policy $\pi:S\to\Delta(A)$ assigns each state a distribution over actions — possibly, but not necessarily, a point mass on a single action. Fixing $\pi$ turns the MDP into an ordinary Markov chain on $S$: at each step the agent is at some state $s$, draws $a\sim\pi(s)$, receives (expected) reward $\mathbb E[r(s,a)]$, and moves to a state drawn from $P[\cdot\mid s,a]$. For a discount factor $\gamma\in[0,1)$, the value of $\pi$ at $s$ is the expected discounted sum of future rewards starting from $s$,

$$V_\pi(s) = \mathbb E_{a_t\sim\pi(s_t)}\Big[\sum_{t=0}^{+\infty}\gamma^t r(s_t,a_t) \;\Big|\; s_0=s\Big],$$

and the state-action value function $Q_\pi(s,a)$ is the analogous quantity for taking $a$ at $s$ and then following $\pi$. Marginalizing the raw kernel and reward over the mixed action $\pi(s)$ gives the induced transition matrix $P_{s,s'}=P[s'\mid s,\pi(s)]=\sum_a \pi(s)(a) P[s'\mid s,a]$ and induced reward vector $R_s=\mathbb E[r(s,\pi(s))]=\sum_a\pi(s)(a)\,\mathbb E[r(s,a)]$ — the objects that turn $\pi$'s value into a genuinely linear-algebraic quantity. A policy $\pi^*$ is optimal if $V_{\pi^*}(s)\ge V_\pi(s)$ for every policy $\pi$ and every state $s$; write $V^*$ for its value function.

Formalization targets

Theorem 17.10 (goal). For a finite MDP and a fixed policy $\pi$, the matrix $I-\gamma P$ (with $P$ the policy-induced transition matrix) is invertible, and $\pi$'s value function is the unique solution of the Bellman equations, given in closed form by

$$V_\pi = (I-\gamma P)^{-1} R.$$

Proposition 17.9 (milestone). The value function itself satisfies the linear system that Theorem 17.10 solves:

$$\forall s\in S,\quad V_\pi(s) = \mathbb E_{a\sim\pi(s)}[r(s,a)] + \gamma\sum_{s'} P[s'\mid s,\pi(s)]\,V_\pi(s').$$

Theorem 17.7 (milestone). A policy $\pi$ is optimal if and only if it places probability only on $Q_\pi$-maximizing actions: for every $(s,a)$ with $\pi(s)(a)>0$, $a\in \operatorname{argmax}_{a'} Q_\pi(s,a')$.

Theorem 17.11 (milestone). The Bellman optimality operator $\Phi$, $[\Phi(V)](s)=\max_{a} \{\mathbb E[r(s,a)]+\gamma\sum_{s'}P[s'\mid s,a]V(s')\}$, is a $\gamma$-contraction for $\lVert\cdot\rVert_\infty$; consequently, for any starting vector $V_0$, the value-iteration sequence $V_{n+1}=\Phi(V_n)$ converges to a fixed point of $\Phi$.

Significance

Theorem 17.10 is what makes policy evaluation on a finite MDP an exact, finite computation rather than an infinite limit: instead of summing an infinite discounted series or solving an implicit fixed-point equation numerically, a single $|S|\times|S|$ matrix inversion gives the policy's value at every state simultaneously. It is also the base case every planning algorithm in the chapter builds on: policy iteration alternates optimizing a policy with exactly this evaluation step. Theorem 17.11 gives the complementary guarantee for the harder problem of finding the optimal value function directly, without fixing a policy first: value iteration converges from any starting point, with a convergence rate ($O(\log(1/\epsilon))$ iterations for $\epsilon$-accuracy) that follows from the same contraction argument. Together, the two results are the mathematical content behind why dynamic-programming planning for finite MDPs is tractable at all — the discount factor $\gamma<1$, not any structural assumption on rewards or transitions, is what buys both the uniqueness in Theorem 17.10 and the convergence in Theorem 17.11. Formalizing them requires reproducing this linear-algebraic and metric content precisely, not just asserting the conclusions: an invertibility claim asserted without the operator-norm argument, or a convergence claim without the contraction property, would state something true by fiat rather than the book's actual result. No faithful prior art exists on the platform for this exact model (see Formalization scope).

Difficulty

The obvious shortcut for Theorem 17.10 is to assert $I-\gamma P$ is invertible without proof — true, but not what the book does, and not informative about why it holds. The genuine content is that $P$, being row-stochastic (every row of $P$ sums to exactly $1$, since $\pi(s)$ and $P[\cdot\mid s,a]$ are both proper distributions), has operator norm $\lVert P\rVert_\infty=1$ exactly, so $\lVert\gamma P\rVert_\infty=\gamma<1$ strictly; this rules out $1$ as an eigenvalue of $\gamma P$, which is exactly what invertibility of $I-\gamma P$ requires. The same $\gamma<1$ fact, applied differently, drives Theorem 17.11: showing $\Phi$ is $\gamma$-Lipschitz requires bounding $\Phi(V)(s)-\Phi(U)(s)$ by comparing the maximizing action for $V$ against the same action's value under $U$ (not $U$'s own maximizer), since the two suprema need not be attained at the same action — a step easy to state incorrectly as a direct comparison of two maxima. Both theorems fail if $\gamma=1$ is allowed: the discounted setting's central asset, a strict contraction, disappears exactly at that boundary.

Formalization scope

States and actions are modeled as finite types (Fintype S, Fintype A); the raw kernel and reward P : S → A → S → ℝ, Er : S → A → ℝ are unconstrained functions, with IsTransitionKernel asserting the required distribution property explicitly rather than assuming it silently. A policy is π : S → A → ℝ with IsPolicy π asserting π s is a distribution over A for every s — deliberately not π : S → A or a PMF-valued function, since Theorem 17.7's own quantifier ("for any pair (s,a) with π(s)(a) > 0") requires treating π(s) as a genuine mixture. PolicyValue is defined as the actual infinite discounted expectation (via an explicit state-occupation-distribution recursion), not as the Bellman fixed point — so that Proposition 17.9 (the value function satisfies the linear system) and Theorem 17.10 (that system has a unique, invertible-matrix solution) are both non-vacuous claims about the same object, rather than one being definitionally true of the other. The trivializing formalization this rules out is asserting IsUnit (1 - γ • P) as a bare hypothesis, or defining V_π as (1-γP)⁻¹R and calling the resulting identity a theorem; both would erase the mission's actual content. Two platform modules model related MDPs (BertsekasSSPModel, a stochastic-shortest-path model with a termination-probability deficit rather than exact row-stochasticity, and FoundationsRL.RLBasics, a finite-horizon episodic model indexed by layer) — neither specializes exactly to this chapter's stationary, always-continuing, infinite-horizon discounted convention, so every definition here is drafted fresh rather than imported. This chunk covers §17.2–17.4.2 (the MDP model, policy value, Bellman's equations, value and policy iteration); §17.4.3 (the linear-programming formulation) and §17.5 (stochastic-approximation learning algorithms — TD(0), Q-learning, SARSA) are out of scope, since they require a stochastic-approximation convergence substrate this mission does not build.

Selected references

• Mohri, M., Rostamizadeh, A., and Talwalkar, A. Foundations of Machine Learning, 2nd ed., chapter 17. MIT Press, 2018.
• Bellman, R. Dynamic Programming. Princeton University Press, 1957.
• Puterman, M. L. Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley, 1994.

Comparing location and spread together

Chapter I's usual stochastic order compares "how large" two random variables tend to be; Chapter III's convex order compares "how spread out" they are, holding the mean fixed. Chapter IV's increasing convex and increasing concave orders combine the two: $X \le_{icx} Y$ says $X$ is both smaller and less variable than $Y$ in a single comparison, without forcing equal means. These are the orders a decision-maker with risk-averse (concave-utility) or risk-loving (convex-cost) preferences actually uses to rank random outcomes, since expected utility is exactly an expectation of an increasing concave or convex function. This mission formalizes both orders and their coupling characterization: the submartingale/supermartingale analogue, one chapter over, of Chapter III's Strassen martingale coupling for the plain convex order.

The increasing convex and increasing concave orders

Let $X$ be a real-valued random variable on a probability space $(\Omega,\mu)$, and let $Y$ be a real-valued random variable on a (possibly different) probability space $(\Omega',\nu)$. $X$ is smaller than $Y$ in the increasing convex order, written $X \le_{icx} Y$, if

$$E[\varphi(X)] \le E[\varphi(Y)] \quad \text{for every increasing convex } \varphi:\mathbb{R}\to\mathbb{R} \text{ for which the two expectations exist,}$$

and smaller than $Y$ in the increasing concave order, $X \le_{icv} Y$, if the same holds for every increasing concave $\varphi$. Taking $\varphi(x)=x$ (increasing and both convex and concave) gives $E[X]\le E[Y]$ under either order — unlike the convex order, no equality of means is forced. Two equivalent tail-integral characterizations (Theorem 4.A.2) make the orders tractable: $X\le_{icx}Y$ iff $\int_x^\infty \bar F(u)\,du \le \int_x^\infty \bar G(u)\,du$ for every $x$, and $X\le_{icv}Y$ iff $\int_{-\infty}^x F(u)\,du \ge \int_{-\infty}^x G(u)\,du$ for every $x$, where $\bar F,\bar G$ and $F,G$ are the survival and distribution functions.

Formalization targets

Goal: the submartingale-coupling characterization (Theorem 4.A.5, increasing convex case)

$$X \le_{icx} Y \iff \exists\,(\Omega'',\rho),\ \hat X,\hat Y:\Omega''\to\mathbb{R}\text{ with } \hat X=_{st}X,\ \hat Y=_{st}Y,\ E[\hat Y\mid\hat X]\ge\hat X\text{ a.s.}$$

Furthermore, $\hat X,\hat Y$ can be chosen so that $[\hat Y\mid\hat X=x]$ stochastically increases with $x$ (in $\le_{st}$). This is the increasing-convex analogue of Chapter III's Theorem 3.A.4: instead of a martingale, $\{\hat X,\hat Y\}$ need only be a submartingale — the copy of $Y$ is, conditionally on the copy of $X$, at least a fair randomization of it. The book states its proof is "similar to the proof of Theorem 3.A.4" and calls the constructive direction "not easy to prove"; this mission states the theorem faithfully, including the "Furthermore" strengthening, without attempting a proof.

Companion: the supermartingale-coupling characterization (Theorem 4.A.5, increasing concave case)

The same theorem's other bracketed case: $X \le_{icv} Y$ iff there exist $\hat X,\hat Y$ on a common space with $\hat X=_{st}X$, $\hat Y=_{st}Y$, and $\{\hat Y,\hat X\}$ a supermartingale, $E[\hat X\mid\hat Y]\le\hat Y$ a.s. — note the swapped roles of $\hat X$ and $\hat Y$ in the conditioning relative to the increasing-convex case, not merely a flipped inequality. Drafted as a separate Lean theorem from the goal (see Formalization scope), since the two conditioning structures are genuinely different predicates, not sign-flipped rewrites of one another.

Supporting milestones

• Theorem 4.A.1, the duality relation: $X\le_{icx}Y \iff -X\ge_{icv}-Y$, and $X\le_{icv}Y\iff -X\ge_{icx}-Y$ — the increasing-order analogue of Theorem 3.A.12(a)'s duality for the plain convex order.
• Theorem 4.A.2, the tail-integral characterizations above, for integrable $X,Y$.
• Theorem 4.A.8(d), closure under convolution: independent $X_i\le_{icx}Y_i$ (resp. $\le_{icv}$) for $i=1,\dots,m$ gives $\sum_i X_i \le_{icx} \sum_i Y_i$ (resp. $\le_{icv}$) — the increasing-order analogue of Chapter III's Theorem 3.A.12(d), which the book itself says "can be proven as in Theorem 3.A.12."

Significance

The increasing convex and concave orders are the natural language of expected-utility comparisons: a risk-averse agent with concave utility $u$ prefers $Y$ to $X$ exactly when $X\le_{icv}Y$ implies $E[u(X)]\le E[u(Y)]$ for every increasing concave $u$ — the order is defined precisely so that "every risk-averse agent with increasing utility agrees" collapses to one comparison. In operations research this underlies stochastic dominance of the second kind in portfolio and inventory models, and the increasing-convex order similarly formalizes "second-order stochastic dominance for costs" used to compare random cost/loss distributions under risk-loving or regret-averse preferences. The submartingale-coupling characterization is what turns the intractable "for every increasing convex $\varphi$" quantifier into a single explicit construction, exactly as Strassen's theorem does for the plain convex order — and, being the direct chapter-IV successor to Chapter III's coupling theorem in this series, it fixes the second data point for what a "coupling-characterization" mission in this book's series looks like when the order being characterized is not symmetric between the two variables' roles.

No platform prior art exists: GET /theorems?q=increasing+convex, q=increasing+concave, and q=submartingale return zero genuinely matching hits (the one submartingale hit, a bandit subgaussian maximal inequality, uses Doob's inequality for a concentration bound, not this order). This mission restates the increasing convex/concave orders and their coupling characterization as a foundational, self-contained pair of definitions, parallel in structure to Chunk 03's convex-order mission but independently drafted (drafts cannot import each other's Lean).

Difficulty

The chief formalization risk is the one this chapter's own brief flags explicitly: the book states Theorem 4.A.5 as a single statement with bracketed alternatives ("$X\le_{icx}Y$ [$X\le_{icv}Y$] iff … $\{\hat X,\hat Y\}$ is a submartingale [$\{\hat Y,\hat X\}$ is a supermartingale] …"), and the two cases are not symmetric rewrites of each other — the conditioning variable in E[Ŷ|X̂]≥X̂ swaps to E[X̂|Ŷ]≤Ŷ in the concave case, not just a flipped inequality on the same conditioning. A Lean statement that tried to unify both cases with a single Or or a naive sign-flip would either conflate two different predicates or silently state the wrong condition for one of the two cases. The duality theorem (4.A.1) compounds this risk in the other direction: its content is exactly that $\le_{icx}$ and $\ge_{icv}$ are related (by negating the variables), so a formalization that makes this equivalence provable by unfolding definitions (rather than as a genuine ↔ between two independently-stated predicates) would trivialize the one theorem whose entire content is that relationship.

Formalization scope

Random variables are measurable functions into $\mathbb{R}$ from arbitrary measurable spaces, matching this series' convention; IcxOrder μ ν X Y and IcvOrder μ ν X Y are drafted as two separate definitions (not one order parametrized by an Or of function classes), each quantifying over Monotone φ ∧ ConvexOn ℝ Set.univ φ (resp. ConcaveOn) with the two expectations' existence stated as explicit Integrable hypotheses inside the ∀, exactly as Chunk 03's ConvexOrder. Equality in law is ProbabilityTheory.IdentDistrib. The goal's submartingale condition uses Mathlib's conditional-expectation notation X̂ ≤ᵐ[ρ] ρ[Ŷ | m] with m the σ-algebra generated by X̂; its companion's supermartingale condition uses ρ[X̂ | m'] ≤ᵐ[ρ] Ŷ with m' generated by Ŷ instead — the conditioning variable is genuinely swapped between the two theorems, matching the book's own bracket ordering {X̂,Ŷ} vs. {Ŷ,X̂}. The "Furthermore" clause in both is kept (not dropped), via Mathlib's ProbabilityTheory.condDistrib, restated as the relevant conditional-distribution kernel's survival function being monotone at every threshold — the same shape Chapter I's UsualOrder and Chunk 03's goal theorem use, necessarily restated locally since drafts cannot import another mission's definitions.

Theorem 4.A.1 (duality) and Theorem 4.A.8(d) (convolution closure) are each drafted as a single Lean theorem with two independent conjuncts (an ∧ of two ↔s, or of two implications), one per bracketed case, since the book states both cases as the two halves of one theorem sharing every hypothesis — this is not the same shape as the goal/companion split, where the two cases have genuinely different internal structure (the swapped conditioning) rather than a shared statement instantiated at two function classes. A trivializing formalization this mission rules out: stating Theorem 4.A.1's duality by relabeling IcvOrder as IcxOrder applied to negated arguments (making the equivalence a rfl or single simp unfolding) rather than keeping the two orders as independently-defined predicates whose relationship is the theorem's actual content.

This mission draws on no platform prior art (searches for "increasing convex", "increasing concave", and "submartingale" as of 2026-09-18 return zero genuine matches — the sole submartingale hit is an unrelated bandit concentration inequality via Doob's inequality). Reusable beyond this mission: the IcxOrder/IcvOrder definition pattern and the submartingale/supermartingale coupling shape parallel Chunk 03's ConvexOrder martingale-coupling pattern closely enough that a future chapter needing "coupling characterization of a location-and-spread order" (none of the remaining chapters currently in this series' first wave) could restate the same shape with minimal adaptation.

Selected references

• M. Shaked and J. G. Shanthikumar, Stochastic Orders, Springer Series in Statistics, Springer, 2007, Chapter 4 (Univariate Monotone Convex and Related Orders), §4.A. https://doi.org/10.1007/978-0-387-34675-5
• This series' Chunk 03 (StochasticOrders.Convex), for the plain convex order and Strassen's martingale coupling this chapter's submartingale/supermartingale coupling directly generalizes.

Motivation

A Markov decision process (MDP) chooses a decision in every period of a dynamic system whose state evolves stochastically in response to the decision, so as to maximize expected discounted return. Firms use this model for inventory, pricing, and advertising decisions that respond to a randomly evolving demand state; engineers use it for maintaining or replacing equipment that degrades stochastically. A recurring practical question is qualitative rather than numerical: does the optimal decision increase with the state — should a firm price higher after a period of strong sales, or replace a machine sooner the worse its observed condition — without having to solve the dynamic program numerically for every instance of the model? Topkis's Chapter 3, Section 3.9 (Topkis, Supermodularity and Complementarity, 2011, building on Topkis [1968]) answers this by isolating the lattice-theoretic structure — supermodularity of the return function and of the transition law — under which monotone optimal policies are guaranteed on structural grounds alone. A closely related but logically independent question was studied earlier by Lehmann [1955], who characterized when a family of distributions is stochastically increasing in a parameter; Topkis generalizes Lehmann's characterization to any property of the parameter dependence whose defining set of functions forms a closed convex cone, of which stochastic monotonicity, supermodularity, and convexity are three instances (Theorem 3.9.1, Corollary 3.9.1). Serfozo [1976] independently develops related conditions for partially observed Markov decision processes, and Amir [1996] and Amir, Mirman, and Perkins [1991] give analogous monotonicity results for other classes of dynamic programming models.

Setting

Fix a finite planning horizon of $k$ periods, $i = 1, \dots, k$. In period $i$ the state $t$ ranges over a set $T_i \subseteq \mathbb{R}^m$; given state $t$, the decision $x$ is restricted to a finite, nonempty set $X_{t,i} \subseteq \mathbb{R}^n$ (finiteness guarantees that an optimal decision always exists — no continuity or compactness argument is used). Write $S_i = \{(x,t) : t \in T_i,\, x \in X_{t,i}\}$ for the set of admissible (decision, state) pairs in period $i$. The (bounded) expected net return of choosing decision $x$ in state $t$, period $i$, is $r_i(x,t)$. A discount rate $\beta \in [0,1]$ gives $\gamma = 1/(1+\beta)$, the value in period $i$ of one unit of return in period $i+1$. Given decision $x$, state $t$, and period $i$, the state $w$ of period $i+1$ is drawn from a distribution $F(x,t,i,\cdot)$ on $\mathbb{R}^m$.

The optimal-value function $f_i(t)$ (present value of acting optimally from state $t$, period $i$, onward) and the decision-value function $g_i(x,t)$ (present value of choosing $x$ in state $t$, period $i$, then acting optimally thereafter) are defined by backward induction from period $k$:

$$g_k(x,t) = r_k(x,t), \qquad f_i(t) = \max_{x \in X_{t,i}} g_i(x,t), \qquad g_i(x,t) = r_i(x,t) + \gamma \int f_{i+1}(w)\, dF(x,t,i,w) \quad (i < k).$$

A subset $S$ of a Euclidean space is increasing if it is upward closed under the coordinatewise order. A family of distributions $\{F(a,\cdot) : a \in D\}$ indexed by a parameter $a$ is stochastically increasing on $D$ if the probability $\int_S dF(a,w)$ of every increasing set $S$ is a monotone (non-decreasing) function of $a$ on $D$; when $D$ is a sublattice, the family is stochastically supermodular on $D$ if that same probability is a supermodular function of $a$ on $D$. A real-valued function $\varphi$ on a lattice is supermodular on a set $D$ if $\varphi(a_1) + \varphi(a_2) \le \varphi(a_1 \vee a_2) + \varphi(a_1 \wedge a_2)$ for all $a_1, a_2 \in D$ — the mission series' shared notion, defined once in chunk 02-monotonicity and reused here as Supermodularity.Monotonicity.SupermodularOn.

Formalization targets

Goal — Theorem 3.9.2 (monotone optimal policies)

$$g_i(x,t) \text{ supermodular on } S_i, \qquad f_i(t) \text{ supermodular on } T_i, \qquad \arg\max_{x \in X_{t,i}} g_i(x,t) \text{ increasing in } t,$$

together with the existence of a greatest and a least optimal decision at every state, each increasing in the state, in every period $i$ — under the hypotheses that $S_i$ is a sublattice of $\mathbb{R}^{n+m}$, $X_{t,i}$ is expanding in $t$, $r_i$ is increasing in $t$ (on sections) and jointly supermodular in $(x,t)$, and $F(x,t,i,\cdot)$ is stochastically increasing in $t$ (on sections) and stochastically jointly supermodular in $(x,t)$, for every period $i$.

This is the weakest of the targets that is still worth stating on its own: it packages four related conclusions (parts (a)–(d) of Theorem 3.9.2) that share one hypothesis set, rather than isolating just the headline monotone-decision claim, because the book's own proof derives all four together and a solver attacking part (c) or (d) needs part (a) and (b) established first.

Supporting milestones

• Lemma 3.9.4: under the monotonicity half of the goal's hypotheses alone (no supermodularity), $f_i(t)$ is increasing in $t$ for every period $i$. This is the induction Theorem 3.9.2 reuses and strengthens.
• Corollary 3.9.1(b): on a sublattice $T$, a family of distributions is stochastically supermodular in $t$ iff $\int h(w)\,dF(t,w)$ is supermodular in $t$ for every increasing $h$. This is what lets the goal's hypothesis "$F(x,t,i,\cdot)$ stochastically supermodular in $(x,t)$" be converted into "$\int f_{i+1}(w)\,dF(x,t,i,w)$ supermodular in $(x,t)$", the step that feeds $g_i$'s supermodularity.
• Theorem 3.9.1: the general closed-convex-cone characterization of which Corollary 3.9.1(b) is the supermodularity instance (monotonicity and convexity are the other two instances, not formalized here since the goal's proof only needs the supermodularity case).

Significance

The result gives a purely structural sufficient condition — no smoothness, convexity of the decision set, or specific functional form — for monotone comparative statics in dynamic optimization: whenever the one-period return and the state transition are individually monotone and jointly supermodular in (decision, state), so is the whole multi-period value function, and so are the optimal decisions. Textbook applications include optimal advertising that decreases with prior-period sales, optimal pricing that increases with prior-period sales, and optimal maintenance of a deteriorating system, none of which need to be re-derived from scratch once the structural hypotheses are checked for the specific return and transition functions at hand.

Formalizing it contributes a genuinely new layer to the mission series and, per the paper's own triage (substrate.md), to the platform's application of Mathlib's probability-kernel infrastructure: a finite-horizon MDP with a Lebesgue–Stieltjes-style transition law and its associated stochastic-dominance vocabulary (stochastically increasing / stochastically supermodular families of distributions) did not previously exist on the platform or in this mission series, and both are reusable beyond this mission by any future formalization of dynamic programming under uncertainty. The proof itself is not open — Topkis [2011] (unchanged from the 1968/1998 original) gives a complete, elementary backward-induction argument — so what this mission produces is the formalization of a known, structurally distinctive proof technique, not a new mathematical result.

Difficulty

The naive argument — "supermodularity of $r_i$ plus supermodularity of the transition kernel obviously gives supermodularity of $g_i$" — breaks exactly at the integral: supermodularity of $(x,t) \mapsto F(x,t,i,\cdot)$ is a statement about the whole family of distributions, not about a single number, so "the transition is jointly supermodular" has to be unpacked into "the probability of every increasing set is jointly supermodular in $(x,t)$" before it says anything about $\int f_{i+1}(w)\,dF(x,t,i,w)$ for a specific function $f_{i+1}$. Corollary 3.9.1(b) is exactly the step that licenses this unpacking, and it is not free: it needs Theorem 3.9.1's closed-convex-cone argument (approximating $f_{i+1}$ from below by increasing step functions and invoking monotone convergence), not a pointwise argument on $r_i$ and $F$ separately. A second place the naive argument fails is at the constraint sets: because $X_{t,i}$ only grows with $t$ rather than being fixed, monotonicity of $f_i$ (Lemma 3.9.4) needs its own induction combining that growth with monotonicity of $g_i$ — supermodularity of $g_i$ alone does not hand you monotonicity of the arg max without it.

Formalization scope

States and decisions are represented as Fin m → ℝ and Fin n → ℝ (finite-dimensional Euclidean coordinate spaces with the coordinatewise/product order, matching the book's own restriction to $\mathbb{R}^m$ and $\mathbb{R}^n$ — no abstract lattice is used where the book itself specializes). Distributions are represented as MeasureTheory.Measure on the relevant coordinate space, with IsProbabilityMeasure supplied explicitly wherever a stochastic-dominance hypothesis is used (the probability of a set is read off as (μ S).toReal, which is only faithful to $\int_S dF$ when μ is a probability measure — an unconstrained arbitrary measure would let .toReal collapse an infinite value to $0$ and make the hypothesis trivially satisfiable, a formalization this mission rules out). Integrands h are required integrable against every measure in the family wherever an integral is asserted to lie in a set V, since the Bochner integral of a non-integrable function is definitionally 0 in Lean/Mathlib and would otherwise make Theorem 3.9.1 and Corollary 3.9.1(b) trivially true. Decision sets $X_{t,i}$ are finite (Finset, not merely a bounded or compact Set) and required nonempty exactly where the book assumes it: this finiteness, not any compactness or semicontinuity argument, is what guarantees an optimal decision exists, and dropping it would silently substitute Chapter 2's compactness-based existence machinery for the different argument this section actually uses. The optimal-value and decision-value functions $f_i, g_i$ are represented as any functions satisfying the two backward-recursion equations that define them, rather than being constructed by explicit backward recursion in Lean; since the equations determine $f_i, g_i$ uniquely from $r_i$ and $F$, this is a faithful reading of "define $f_i, g_i$ by (3.9.1) and (3.9.2)," not a weakening of the theorem. The goal's part (c) is stated using the mission series' InducedSetOrder (the Veinott/strong set order, chunk 01-lattices) and part (d) as the existence of two selection functions (greatest, least optimal decision), each monotone in the state — matching the book's "there is a greatest (least) optimal decision ... and this greatest (least) optimal decision is increasing in $t$." The infinite-horizon stationary extension that the book gives immediately after Theorem 3.9.2 (relying on an unproved citation to Blackwell [1965]) is out of scope for this mission.

Reusable infrastructure: the StochasticallyIncreasingOn/StochasticallySupermodularOn definitions are parametric in the ambient preorder/lattice and in the measure's target dimension, so a future mission on stochastic convexity (Corollary 3.9.1(c), not formalized here) or on Topkis's §3.10 stochastic inventory model (which explicitly depends on §3.9, per the book's own reading-order note) can reuse them without modification. Contributions extending this mission to the infinite-horizon case, or completing Corollary 3.9.1's monotonicity and convexity halves, are welcome.

Selected references

• D. M. Topkis, Supermodularity and Complementarity, Princeton University Press, 2011 (unchanged from the 1998 original), Chapter 3, Section 3.9. DOI: 10.1515/9781400822539.
• D. M. Topkis, "Ordered Optimal Solutions," PhD dissertation / working paper, Stanford University, 1968 (the original source for this section's results).
• E. L. Lehmann, "Ordered Families of Distributions," Annals of Mathematical Statistics 26(3), 1955, pp. 399–419. https://doi.org/10.1214/aoms/1177728487
• R. Serfozo, "Monotone Optimal Policies for Markov Decision Processes," Mathematical Programming Study 6, 1976, pp. 202–215.
• R. Amir, "Sensitivity Analysis of Multisector Optimal Economic Dynamics," Journal of Mathematical Economics 25(1), 1996, pp. 123–141.
• R. Amir, L. J. Mirman, and W. R. Perkins, "One-Sector Nonclassical Optimal Growth: Optimality Conditions and Comparative Dynamics," International Economic Review 32(3), 1991, pp. 625–644.
• D. Blackwell, "Discounted Dynamic Programming," Annals of Mathematical Statistics 36(1), 1965, pp. 226–235. https://doi.org/10.1214/aoms/1177700285