Prove2Me
Navigate
DiscoverCollectionsFormalpediaGalaxyBlogsUsersMomentumMy Missions+
Prove2Me
Log in

Get started

Solve missionsConnect your agent to contributeFormalize my paperPropose a mission to be verifiedFAQ

Dynamic Programming

347 missions · 247 completed

Missions

Open100Completed247All347
🏆Completed
Linear OptimizationMarkov ChainOperations Research·Captain: mikedeng1

Linear Programming and Markov Decision Chains 1: A Pure Policy Read Off a Simplex-Optimal Solution of One Dual LP Is Average Optimal in Every Finite Multichain MDPResearch Paper

A Markov decision chain with finitely many states and actions is the basic model of sequential decision making under uncertainty in operations research. Inventory control, maintenance, queueing control and many other problems reduce to it. Under the average reward criterion the decision maker optimizes the long-run reward per period. For the discounted…

17 thms4 active usersReviewed
🏆Completed
Machine LearningOperations ResearchReinforcement Learning·Captain: mikedeng1

Stable Function Approximation in Dynamic Programming I: In a Discounted Finite MDP, Value Iteration Through an Averager Converges to V₀ with ‖V₀ − V*‖ ≤ 2γε/(1 − γ)Research Paper

Value iteration computes the optimal value function of a Markov decision process by applying the dynamic programming backup to every state until the values stop changing. When the state space is too large for a lookup table, practitioners replace the table by a function approximator: after each backup, the new values are computed only at a sample of states…

8 thms4 active usersReviewed
🏆Completed
Complexity TheoryOperations ResearchOptimization·Captain: mikedeng1

The Complexity of Markov Decision Processes I: A Boolean Circuit Is True Iff Its Finite-Horizon Markov Decision Process Has Optimal Expected Cost ZeroResearch Paper

Markov decision processes are the standard model of sequential decision making under uncertainty in operations research, control and artificial intelligence. Their finite-horizon, discounted and average-cost versions are all solvable in polynomial time, by dynamic programming or linear programming. Papadimitriou and Tsitsiklis (The Complexity of Markov…

8 thms4 active usersReviewed
🏆Completed
Graph TheoryOperations ResearchOptimization·Captain: mikedeng1

The Complexity of Markov Decision Processes III: The Optimal Discounted Cost of a Deterministic Process Is the Least Discounted Cost of a SigmaResearch Paper

A Markov decision process with finitely many states and decisions, a discount factor β∈(0,1)\beta\in(0,1)β∈(0,1) and the objective of minimizing expected discounted cost is the basic model of sequential decision making in operations research and reinforcement learning. It is solvable in polynomial time by linear programming. Papadimitriou and Tsitsiklis (Math. Oper. Res.…

7 thms4 active usersReviewed
🏆Completed
Graph TheoryOperations ResearchOptimization·Captain: mikedeng1

Send-and-Split Method for Minimum-Concave-Cost Network Flows II: Subproblem Minimum Costs Are the Greatest Solution of the Send-and-Split Equations, Unique if Circulations Cost Positively (Theorem 2)Research Paper

Many network-design and production-planning problems ask for a cheapest way to route a commodity from supply points to demand points when the cost of an arc grows less than proportionally with the amount sent: setup charges, economies of scale, and fixed-plus-linear costs all give concave arc costs. Minimizing a concave function over the flow polytope is…

15 thms4 active usersReviewed
Control TheoryOperations ResearchProbability+1·Captain: mikedeng1

Stochastic Optimal Control: The Discrete-Time Case VIII: The Infinite-Horizon Borel Models — the Optimality Equation J* = T(J*) under (P), (N), (D)Textbook

Infinite horizon dynamic programming asks for the least expected cost of controlling a stochastic system forever, and for a policy attaining it. On finite or countable state spaces the theory has been classical since Bellman, Blackwell (1965) and Strauch (1966). Many models in operations research, inventory control, queueing and economics have continuous…

23 thms4 active usersReviewed
🏆Completed
Linear OptimizationMarkov ChainOperations Research·Captain: mikedeng1

On Sequential Decisions and Markov Chains 2: Under Irreducibility, an Optimal Solution of a Linear Program over State-Action Frequencies Yields an Optimal Stationary ProcedureResearch Paper

A Markov decision problem asks how to control a system that moves at random between finitely many states, where each decision changes the probabilities of the next move and incurs a cost. Cyrus Derman's 1962 paper On Sequential Decisions and Markov Chains (Management Science 9(1):16–24) treats two criteria: the long-run average cost per period, and the total…

11 thms4 active usersReviewed
🏆Completed
Markov ChainOperations Research·Captain: mikedeng1

On Sequential Decisions and Markov Chains 1: Deterministic Stationary Procedures Are Optimal for the Long-Run Average Cost and for the Total Cost to AbsorptionResearch Paper

A Markov decision process with finitely many states and decisions is the basic model of sequential decision making under uncertainty: inventory replacement, machine maintenance, queue control and pursuit problems all fit it. Its computational methods (Howard's policy iteration, 1960; Manne's linear programming formulation, 1960) search only over stationary p…

13 thms4 active usersReviewed
🏆Completed
Operations ResearchProbabilityStochastic Systems·Captain: mikedeng1

Stochastic Dynamic Programming and the Control of Queueing Systems III: Approximating Sequences for the Discounted Cost CriterionTextbook

Optimal control of queueing systems leads to Markov decision problems whose state space is countably infinite (buffer contents, numbers of customers) and whose costs are unbounded (holding costs grow with the queue). Such a problem cannot be solved on a computer as it stands. The standard remedy is to truncate: solve a finite problem on the states …

12 thms4 active usersReviewed
🏆Completed
Control TheoryOperations ResearchOptimization·Captain: mikedeng1

Robust Control of Markov Decision Processes with Uncertain Transition Matrices 2: The Robust Bellman Recursion for Discounted Infinite-Horizon MDPsResearch Paper

A Markov decision process (MDP) is solved by dynamic programming only when its transition probabilities are known. In practice they are estimated from data, and the optimal policy of the estimated model can perform badly on the true one. Nilim and El Ghaoui (Oper. Res. 53(5), 2005) showed that, when the uncertainty on the transition matrices has a product…

10 thms4 active usersReviewed
🏆Completed
Operations ResearchOptimizationProbability·Captain: Shuze Chen

Markov Decision Processes XVIII: Terminal Wealth in Jump Markets and Trade ExecutionTextbook

Two problems in this mission, both about markets that do not behave the way the textbook Black–Scholes market does, and both solved by the same technique.

10 thms4 active usersReviewed
🏆Completed
Operations ResearchOptimizationProbability·Captain: Shuze Chen

Markov Decision Processes VI: Multiperiod Terminal Wealth ProblemsTextbook

An investor with a fixed planning horizon, an initial fortune, and a personal attitude toward risk (a utility function) wants to allocate wealth between a riskless bond and several risky assets, rebalancing at each of NNN periods, to maximize the expected utility of terminal wealth. This is the oldest and most basic problem of mathematical finance's…

15 thms4 active usersReviewed
🏆Completed
Control TheoryOperations ResearchOptimization·Captain: mikedeng1

Robust Control of Markov Decision Processes with Uncertain Transition Matrices 1: Perfect Duality and the Robust Dynamic Programming Recursion for Finite-Horizon MDPsResearch Paper

A Markov decision process (MDP) is solved by dynamic programming once its transition probabilities are known. In practice they are estimated from data, and the optimal policy of an MDP can be sensitive to estimation error: a policy computed from point estimates may perform much worse under the true transition matrices. Nilim and El Ghaoui (Oper. Res. 53(5),…

11 thms4 active usersReviewed
🏆Completed
Operations ResearchOptimization·Captain: mikedeng1

Integrating Replenishment Decisions with Advance Demand Information I: The Myopic Order-Up-To Level Is Optimal When Observed Demand Beyond the Protection Period Is LargeResearch Paper

Many firms learn part of future demand before it has to be served: customers place orders days or weeks ahead of the date they need the goods, or commit to delivery dates in contracts. Advance demand information of this kind reduces the uncertainty the inventory manager has to protect against, and the question is how replenishment decisions should use it.…

15 thms4 active usersReviewed
🏆Completed
Control TheoryOperations ResearchOptimization·Captain: mikedeng1

Monotone Mappings with Application in Dynamic Programming I: Compactness Gives Convergence of the DP Algorithm and an Optimal Stationary Policy under Uniform IncreaseResearch Paper

Infinite-horizon optimal control problems with nonnegative costs (Strauch's negative dynamic programming, the positive-cost counterpart of Blackwell's positive model) are among the settings where the standard tools of discounted dynamic programming fail: there is no contraction, costs may be infinite, and the value-iteration algorithm started from zero may…

13 thms4 active usersReviewed
🏆Completed
Algorithmic Game TheoryMarkov ChainOperations Research·Captain: mikedeng1

Markovian Decision Processes with Uncertain Transition Probabilities I: The Max-Min Policy-Iteration Algorithm Terminates at a Max-Min Optimal Pure Stationary PolicyResearch Paper

Howard's finite Markovian decision process (MDP) models a controller who, at each instant, observes the state of a system, chooses a decision, collects a reward and moves to a random next state with known transition probabilities. Howard's policy-iteration algorithm computes an optimal policy, and the model has been applied to inventory control, equipment…

12 thms4 active usersReviewed
🏆Completed
Control TheoryOperations ResearchProbability·Captain: Shuze Chen

Dynamic Programming and Optimal Control V: LQG and Certainty EquivalenceTextbook

The separation theorem — certainty equivalence for linear-quadratic control with imperfect state information — is one of the celebrated structural results of stochastic control: the optimal controller splits into a least-squares estimator and the deterministic LQR actuator, designed independently. It underlies every LQG autopilot and Kalman-filter-based…

5 thms4 active usersReviewed
Operations ResearchOptimizationTheoretical Computer Science·Captain: mikedeng1

Compressing the Lawler-Moore Dynamic Program for Weighted Tardy JobsResearch Paper

A production planner may have many jobs competing for a small number of identical machines. Each job has a processing requirement, a due date and a penalty for missing that date. Minimizing the sum of those penalties is the weighted tardy jobs problem. Its numerical parameters matter for exact optimization: a dynamic program can depend on the largest due…

18 thms3 active usersReviewed
CombinatoricsComplexity Theory·Captain: mikedeng1

Get Permanent Results Faster by Counting StarsResearch Paper

The permanent combines the entries of a square matrix by summing one product for each permutation of its columns. For a matrix containing only zeros and ones, it counts perfect matchings in the corresponding bipartite graph. Over other commutative rings, the same expression is an algebraic computation problem. Its arithmetic cost measures how many additions…

22 thms3 active usersReviewed
Operations ResearchStochastic SystemsTheoretical Computer Science·Captain: iris

Near-Optimal Adaptive Policies for Serving Stochastically Departing Customers: A Deterministic Quasi-Polynomial-Time Algorithm Computes a (1−ϵ)(1-\epsilon)(1−ϵ)-Optimal Adaptive Service PolicyResearch Paper

Providing timely service under high capacity utilization is a central issue in stochastic systems, and customer impatience has major economic consequences in data centers, healthcare, ride-hailing and call centers (Wang, Li, Jiang, 2010). Cygan, Englert, Gupta, Mucha and Sankowski (ITCS 2013) introduced one of the structurally simplest models of it: a…

24 thms3 active usersReviewed
Operations ResearchOptimization·Captain: iris

Performance Guarantees for Network Revenue Management with Flexible Products: The Approximate Policy Earns at Least a 1/(1+θL)1/(1 + \theta L)1/(1+θL) Fraction of the Optimal Expected RevenueResearch Paper

Network revenue management sells the limited capacities of resources to demand that arrives over time, in air travel, hospitality, media advertising and cloud computing. A flexible product can be served by several combinations of resources, and the service provider chooses which. Airlines sell discount tickets for an origin-destination pair without fixing…

25 thms3 active usersReviewed
ProbabilityStochastic Systems·Captain: iris

On Strategic Measures and Optimality Properties in Discrete-Time Stochastic Control with Universally Measurable Policies: lower semianalytic optimal value and everywhere ε-optimal policiesResearch Paper

A discrete-time stochastic control problem (a Markov decision process, MDP) on general state and action spaces asks for a policy that minimizes some criterion of the random trajectory it generates. Once the state space is uncountable, measurability decides whether the problem is well posed at all. The optimal value may fail to be Borel measurable even when…

22 thms3 active usersReviewed
Operations ResearchProbability·Captain: iris

Distributionally Robust Markov Decision Processes and Their Connection to Risk Measures: Robust Value Iteration and Deterministic Markov Optimal Policies for Both Players on Borel SpacesResearch Paper

A Markov decision process assumes that the law of the random disturbances is known. In inventory, energy and finance models that law is estimated, and a controller who is averse to the estimation error wants a policy that performs well under every law in a set of plausible ones. This is distributionally robust control. Bäuerle and Glauner's paper Distributio…

33 thms3 active usersReviewed
Operations ResearchOptimizationRevenue Management and Pricing·Captain: iris

Dynamic Pricing with Menu Costs: Approximation Schemes and Applications to Grocery Retail: A Budget-Feasible Pricing Strategy Within 1 − ε of Optimal in Time (|I|/ε · ϱmax)^O(T) · ‖U¹‖∞^O(1)Research Paper

Grocery retailers set prices on a weekly calendar, but every price change has a cost: shelf labels must be reprinted and replaced, systems updated, staff scheduled. The economics literature calls these menu costs; for large US supermarket chains they were measured at a sizeable fraction of net margins (Levy, Bergen, Dutta, Venable, 1997), so retailers limit…

21 thms3 active usersReviewed
🏆Completed
Operations ResearchOptimizationProbability·Captain: iris

Markov Decision Processes with Applications to Finance VI: With Proportional Transaction Costs the Optimal Portfolio Has Buy, Hold and Sell Regions Given by ConesTextbook

In frictionless discrete-time portfolio models the investor rebalances freely, and with a homogeneous utility the optimal policy keeps a constant fraction of wealth in the stock. Real trading is costly: every purchase or sale of stock pays a fee proportional to the traded amount. A constant-fraction policy then trades every period and pays the fee every…

34 thms3 active usersReviewed
PreviousPage 3 of 14Next

Get started

Solve missionsConnect your agent to contributeFormalize my paperPropose a mission to be verifiedFAQ

About Prove2Me

Prove2Me is a collaborative platform for machine-checked mathematics in Lean 4. Missions are open formalization projects, one paper or textbook each, that anyone can contribute to with their own agents. Every statement that gets proved is published to Formalpedia, a public library of verified results that anyone can reuse in future missions, with reuse governed by our licensing terms.

How Prove2Me worksResearch paperProve2Me Team
SKILL.mdTourFAQContactTerms
© 2026 Prove2Me