Computational Issues in an Infinite-Horizon, Multiechelon Inventory Model 2: The Decomposition Policy Is Average-Cost OptimalResearch Paper
Motivation
Many supply chains move stock through a central warehouse to the retail locations that face customer demand. Deciding how much the warehouse should order from outside, and how much it should ship to each retailer and when, is a stochastic dynamic program whose state contains every stock level and every outstanding order. Exact solution is out of reach except for the smallest systems, so structural results that reduce such a problem to single-location problems matter in practice.
Timeline.
- Clark and Scarf (Management Science 1960) showed that the finite-horizon, discounted problem of a serial system decomposes: an optimal policy is obtained by solving the most downstream location alone, charging its shortfalls to the upstream location through an induced penalty cost, and then solving the upstream location as a single-location problem with that penalty.
- Iglehart (Management Science 1963, and a 1963 chapter in Multistage Inventory Models and Techniques) established the infinite-horizon theory of the single-location problem with a fixed order cost: optimality of stationary policies under discounted and average costs, and the convergence of value iteration.
- Federgruen and Zipkin (Operations Research 1984) carried the decomposition to the infinite horizon for a depot and one retail outlet, under discounted costs (Theorem 1) and under the average-cost criterion (Theorem 2). This mission is about the average-cost case, §3 of that paper.
Setting
Time is divided into periods. A depot orders from an outside supplier with lead time and supplies a retail outlet with shipment lead time . The demand in each period is a nonnegative random variable with law and finite mean ; demands in different periods are independent and identically distributed. Unmet demand at the outlet is backordered.
The state is :
- lists the orders placed periods ago;
- is the depot's echelon inventory, its own stock plus the outlet's inventory position;
- is the outlet's inventory position, its stock plus shipments in transit.
In each period the decision is an order and a shipment with , where is the order arriving now. The state then moves to .
Costs are a fixed order cost , proportional order and shipment rates and , a holding rate on system inventory, an extra holding rate at the outlet and a backorder penalty rate . After the paper's accounting transformation, the one-period cost is
with for and , , and, at ,
where is the demand over periods. The critical number is a minimizer of . The stationary induced penalty is for and otherwise.
For a policy and initial state , is the expected cost of the first periods and is the average cost. Problem IH asks for a policy minimizing from every state. The depot problem IH has states , orders and one-period cost . Its minimal average cost is . The outlet problem has states , shipments and one-period cost . The policy orders by an optimal stationary policy of IH and ships : up to the critical number if the depot has the stock, otherwise as much as it has.
Formalization targets
Goal: Theorem 2 (p. 828)
With and , the policy is measurable and feasible from every physical state, and for every such state and every measurable feasible policy ,
Milestones
- Property (f) (p. 824): for the outlet program .
- Eq. (4) (p. 823), for : .
- §3 claims (p. 828): with , is the critical number of every period, for , , and .
- §3 display (p. 828): .
- Lemma 5 (p. 828): .
- Proof of Theorem 2 (p. 828): for every measurable feasible .
Significance
The result. Theorem 2 reduces an average-cost problem with a multidimensional state to two problems with smaller states: a single-location -type problem for the depot with a known convex penalty , and a myopic critical-number rule for the outlet. The optimal system cost is the sum of their optimal costs. The paper uses this to compute optimal policies with standard single-location software, and its §5 builds heuristics for several outlets on the same decomposition.
Formalizing it. The result is proved in the paper; nothing here is open. To our knowledge none of it has been machine-checked. A formal proof has to make precise what the paper leaves to "standard arguments":
- the class of measurable history-dependent policies;
- the expected costs of policies with unbounded one-period costs;
- the passage from history-dependent to Markov policies;
- the transient of when the outlet starts above its critical number.
Difficulty
The obvious argument would identify the average-cost optimal value through an average-cost optimality equation on the full state space and verify that attains it. No such equation is available here. The state space is unbounded, the one-period costs are unbounded both above and below in the state, and the depot's fixed cost makes its value functions -convex rather than convex.
The paper's route avoids that equation but needs three separate facts:
- value iteration for the whole system, divided by , converges to , which rests on Iglehart's convergence for the depot and on the stationarity of the penalties when ;
- the finite-horizon value bounds the cost of every history-dependent policy, not only of Markov ones;
- achieves from every state, including states with , where it does not ship at all until demand has brought the outlet below its critical number.
Formalization scope
- Representation. A state is a triple in . For the order placed now arrives at once. Time runs forward in Lean; the paper numbers periods backward. Finite-horizon value functions keep the paper's index (periods remaining). Each "min" of programs (1), (2), (3), (5) is a real infimum over the constraint set.
- Policies and costs. Policies are deterministic, history-dependent and measurable, and they must be feasible along every demand realization. is an extended real (expected positive part minus expected negative part of each period's cost). is a in the extended reals, and the optimal average costs are infima in the extended reals.
- Standing assumptions (p. 821):
- ;
- demands i.i.d., nonnegative, without atoms ("for convenience we shall assume is continuous") and with finite mean.
- Added hypotheses.
- States are restricted to the physical ones, and .
- . The paper reduces to this case "without loss of generality", on the grounds that average proportional costs equal and "under all interesting policies" (p. 827). That class is never specified, and the proofs are written for . The general-cost version is the paper's informal reduction and is not part of the goal.
- Eq. (4) is stated for with and (so that it covers §3's case ), and with the relation , which the paper names on p. 827; it holds automatically at .
- Ruling out trivial readings.
- is built from a depot rule assumed optimal for IH from every depot state. Its existence is Iglehart's theorem, cited and not formalized; no form is required.
- The goal quantifies over all measurable feasible policies, and 's own feasibility is a conclusion, so a vacuous policy class cannot satisfy it.
- A sorry-free check in the workspace exhibits an instance (exponential demand) meeting every standing hypothesis other than the optimality of , including the existence of .
- Reusable infrastructure. The definitions of history-dependent policies and of extended-real expected and average costs for controlled processes driven by i.i.d. real noise are generic, and could be reused for other inventory and queueing models. Contributions are welcome on any milestone, and especially on a formal version of the Markov reduction (Dynkin–Yushkevich III.1) for this setting and on Iglehart's convergence of .
Selected references
- A. Federgruen and P. Zipkin, Computational Issues in an Infinite-Horizon, Multiechelon Inventory Model, Operations Research 32(4):818–836, 1984. https://doi.org/10.1287/opre.32.4.818
- A. J. Clark and H. Scarf, Optimal Policies for a Multi-Echelon Inventory Problem, Management Science 6(4):475–490, 1960. https://doi.org/10.1287/mnsc.6.4.475
- D. L. Iglehart, Optimality of (s, S) Policies in the Infinite Horizon Dynamic Inventory Problem, Management Science 9(2):259–267, 1963. https://doi.org/10.1287/mnsc.9.2.259
- D. L. Iglehart, Dynamic Programming and Stationary Analyses of Inventory Problems, Chapter 1 in H. Scarf, D. Gilford and M. Shelly (eds.), Multistage Inventory Models and Techniques, Stanford University Press, 1963.
- E. B. Dynkin and A. A. Yushkevich, Controlled Markov Processes, Springer, 1979.
- D. P. Bertsekas and S. E. Shreve, Stochastic Optimal Control: The Discrete-Time Case, Academic Press, 1978.