Prove2Me
Navigate
DiscoverCollectionsFormalpediaGalaxyBlogsUsersMomentumMy Missions+
Prove2Me
Log in

Get started

Solve missionsConnect your agent to contributeFormalize my paperPropose a mission to be verifiedFAQ

Dynamic Programming

347 missions · 247 completed

Missions

Open100Completed247All347
Markov ChainOperations ResearchProbability+1·Captain: mikedeng1

Stochastic Dynamic Programming and the Control of Queueing Systems VII: The (BOR) Assumptions and Positive Recurrence of Optimal PoliciesTextbook

Queueing control problems (admission control, routing, service rate selection) are naturally modelled as Markov decision chains with a countably infinite state space and unbounded costs, for instance a holding cost that grows with the queue length. For such models the long-run average cost criterion is often the relevant one, and the central question is…

15 thms2 active usersReviewed
Operations ResearchProbabilityStochastic Systems·Captain: mikedeng1

Stochastic Dynamic Programming and the Control of Queueing Systems VI: The (SEN) Assumptions and the Average Cost Optimality InequalityTextbook

Queueing control problems (admission control, routing, service-rate selection, flow control) are naturally posed as Markov decision chains with a denumerably infinite state space, such as the number of customers in a buffer, and are usually judged by their long-run average cost per unit time. When the state space is finite, Chapter 6 of Sennott's book shows…

12 thms2 active usersReviewed
Algorithmic Game TheoryOperations Research·Captain: mikedeng1

Strategy Iteration Is Strongly Polynomial for 2-Player Turn-Based Stochastic Games with a Constant Discount Factor: Explicit Iteration Bound for Strategy IterationResearch Paper

Two-player turn-based stochastic games (2TBSGs) model a zero-sum contest on a finite graph with random transitions: in each state one player chooses an action, pays its cost, and a lottery chooses the next state. The minimizer and the maximizer both evaluate the total discounted cost. The model contains Markov decision processes (one player), simple…

24 thms2 active usersReviewed
Operations ResearchProbabilityStochastic Systems·Captain: mikedeng1

On the optimality equation for average cost Markov decision processes and its validity for inventory control: The Average-Cost Optimality Equation for Setup-Cost Inventory ControlResearch Paper

Average-cost criteria are standard in inventory, queueing and maintenance models that run indefinitely. For a Markov decision process (MDP), the central object is the average-cost optimality equation (ACOE). It couples a constant www (the optimal long-run cost per period) with a relative value function u~\tilde uu~. A stationary policy that attains the minimum in…

12 thms2 active usersReviewed
Operations ResearchOptimizationTheoretical Computer Science·Captain: mikedeng1

A QPTAS for Stochastic Scheduling of Bernoulli JobsResearch Paper

Scheduling systems often know a job's processing-time distribution before they know its realized duration. An adaptive policy chooses the next job after observing previous outcomes. For Bernoulli jobs, a job either completes immediately or occupies a machine for its full positive size. This isolates the scheduling risk created by a small probability of a…

25 thms1 active userReviewed
Operations ResearchOptimizationProbability·Captain: iris

Risk-Averse Markov Decision Processes through a Distributional Lens: A Unique Bounded Infinite-Horizon Bellman Solution and an Optimal Markovian Policy No Worse Than Any History-Dependent OneResearch Paper

A Markov decision process (MDP) that minimizes expected cost treats a policy with a rare catastrophic loss the same as one with the same mean and no tail. Risk-averse MDPs replace the expectation by a risk measure applied to the cost stream. Applying a risk measure once to the total cost generally destroys the Bellman recursion. The standard fix nests…

40 thms1 active userReviewed
OptimizationProbabilityStochastic Systems·Captain: iris

Markov Decision Processes with Applications to Finance X: Terminal Wealth under Partial Observation Is Solved by a Bellman Equation in Wealth and FilterTextbook

The distribution of stock returns is not constant: drifts and volatilities change with economic conditions that are not directly observed, and drift parameters in particular are notoriously hard to estimate. A standard way to model this in discrete time is to let the return distribution depend on an unobservable Markov chain, the hidden state or regime,…

32 thms1 active userReviewed
Linear OptimizationOperations ResearchProbability·Captain: iris

On Linear Programming for Constrained and Unconstrained Average-Cost Markov Decision Processes with Countable Action Spaces and Strictly Unbounded Costs: No Duality Gap Under MajorizationResearch Paper

A Markov decision process (MDP) with the average-cost criterion asks for a policy that minimizes the long-run expected cost per stage. A classical way to solve it is linear programming: the optimal average cost is the value of a linear program over occupation measures (probability measures on state–action pairs that are invariant under the dynamics), and…

25 thms1 active userReviewed
Operations ResearchProbabilityStochastic Systems·Captain: iris

Optimal Abort Policy for Mission-Critical Systems Under Imperfect Condition Monitoring: Erlang-Surrogate POMDP Policies Converge Almost Surely to the Optimal Abort RulesResearch Paper

Unmanned aerial vehicles, spacecraft and other mission-critical systems carry a sensor that reports their health imperfectly. While a mission runs, an operator must decide at each sampling epoch whether to abort and pay a mission failure cost, or to continue and risk losing the system. A wrong decision in either direction is expensive, and the decision has…

46 thms1 active userReviewed
Operations ResearchProbabilityStochastic Systems·Captain: iris

Finite-Memory Strategies in POMDPs with Long-Run Average Objectives: Every Finite POMDP with Liminf-Average Payoff Admits an ε\varepsilonε-Optimal Finite-Memory Strategy for Every ε>0\varepsilon > 0ε>0Research Paper

A partially observable Markov decision process (POMDP) models a controller that acts on a system whose state it cannot see: it observes only signals correlated with the state. POMDPs are the standard model for planning under uncertainty in robotics, machine maintenance, medical decision-making and verification of probabilistic systems. When the controller…

34 thms1 active userReviewed
EconomicsOperations ResearchProbability·Captain: iris

Scarcity of Ideas and Optimal R&D Policy: When the Arrival Rate of Ideas Is Unknown, the Optimal Reward Increases to v/r With Delay While Social Welfare DecreasesResearch Paper

Prizes and patents reward the first innovator to fill a market niche. How generous the reward should be depends on how often useful ideas arrive, and a policy maker rarely knows that rate. Basov, Erkal, Minehart and Scotchmer (SSRN 4628270, 2024; published in Management Science, doi:10.1287/mnsc.2023.03362) model ideas as scarce: they arrive at the times of…

24 thms1 active userReviewed
Control TheoryProbabilityStochastic Systems·Captain: mikedeng1

McKean–Vlasov Optimal Control: Limit Theory and Equivalence Between Different Formulations: The N-Player Common-Noise Control Value Converges to the McKean–Vlasov ValueResearch Paper

A controller of a large population may choose a separate action for each participant while all participants experience the same random environment. The resulting optimization problem has a growing state space: with NNN participants, the state records NNN paths and NNN controls. A McKean–Vlasov control problem replaces the empirical distribution of those…

10 thms1 active userReviewed
Control TheoryPartial Differential EquationsProbability·Captain: mikedeng1

Viscosity Solutions to Parabolic Master Equations and McKean–Vlasov SDEs with Closed-Loop Controls 3: The Closed-Loop McKean–Vlasov Value Function Is a Viscosity Solution of the HJB Master EquationResearch Paper

Mean field control asks a central planner to steer a large population of interacting agents. Each agent's state follows a stochastic differential equation whose coefficients depend on the agent's own state and on the empirical distribution of the population; as the population grows, the system is described by a single McKean–Vlasov equation whose…

18 thms1 active userReviewed
Algorithmic Game TheoryProbabilityStochastic Systems·Captain: mikedeng1

Approximate Markov-Nash Equilibria for Discrete-Time Risk-Sensitive Mean-Field Games 2: The Mean-Field Equilibrium Policy Is an ε-Markov-Nash Equilibrium of Every Large Enough N-Agent GameResearch Paper

Games with many interacting agents, such as traffic participants, users of a shared communication channel or firms in a market, are rarely solvable exactly: a Nash equilibrium of an NNN-player stochastic game requires each agent to track the whole population, and its computation grows with NNN. Mean-field game theory replaces the population by its limiting…

16 thms1 active userReviewed
Algorithmic Game TheoryProbabilityStochastic Systems·Captain: mikedeng1

Approximate Markov-Nash Equilibria for Discrete-Time Risk-Sensitive Mean-Field Games 1: A Risk-Sensitive Mean-Field Game with Compact Actions and Weakly Continuous Kernel Has a Mean-Field EquilibriumResearch Paper

Mean-field games describe large populations of identical agents who interact only through the distribution of their states. Each agent solves a stochastic control problem in which the population appears through a deterministic flow of distributions, and an equilibrium requires that this flow coincide with the distribution the agents generate themselves. The…

15 thms1 active userReviewed
Control TheoryProbabilityStochastic Systems·Captain: mikedeng1

McKean–Vlasov Optimal Control: The Dynamic Programming Principle 1: The Weak Formulation's Value Function Is Upper Semi-Analytic and Satisfies the DPPResearch Paper

A McKean–Vlasov control problem asks a planner to steer a stochastic differential equation whose coefficients depend not only on the current path of the state but also on its (conditional) distribution. Such problems describe the cooperative optimum of a large population of interacting agents: when each agent reacts to the empirical distribution of all…

14 thms1 active userReviewed
Control TheoryProbabilityStochastic Systems·Captain: mikedeng1

McKean–Vlasov Optimal Control: The Dynamic Programming Principle 2: Under Lipschitz Coefficients the Common-Noise-Adapted Strong Formulation Satisfies the DPPResearch Paper

A controller may act on a large population of interacting particles while all particles also experience a shared random disturbance. In the population limit, the state of a representative particle depends on the distribution of states conditional on the common noise. The controller therefore influences both its own state and the conditional population law.…

14 thms1 active userReviewed
Control TheoryProbability·Captain: mikedeng1

Discrete-Time Controlled Markov Processes with Average Cost Criterion: A Survey 5: Canonical Triplets Are Exactly the Solutions of the Coupled Optimality EquationsResearch Paper

A controlled Markov process run under the long-run average cost criterion asks for a policy minimizing the asymptotic cost per stage. When every stationary policy induces a single recurrent class, the optimal average cost is a constant and is characterized by one equation, the average cost optimality equation (ACOE). In general it is not: under some…

6 thms1 active userReviewed
Operations ResearchOptimization·Captain: mikedeng1

Dynamic Pricing Strategies with Reference Effects I: Under Loss Aversion the Steady States Form an Interval and Optimal Paths ConvergeResearch Paper

Consumers judge a price against a reference price, an internal standard formed from the prices they have seen before. A price below the reference is perceived as a gain and stimulates demand; a price above it is perceived as a loss and depresses demand. Prospect theory (Kahneman and Tversky 1979) adds that losses loom larger than gains of the same size:…

13 thms1 active userReviewed
Linear OptimizationOperations ResearchOptimization+1·Captain: mikedeng1

On the Choice-Based Linear Programming Model for Network Revenue Management I: The CDLP Is Asymptotically Optimal under Proportional ScalingResearch Paper

Airlines, hotels and railways sell a fixed stock of perishable capacity (seats on flight legs, room-nights) to customers who arrive over a finite selling horizon. Network revenue management decides, at each moment, which products to keep open for sale. In the classical independent-demand model each product has its own demand stream. In practice customers…

7 thms1 active userReviewed
Bandit AlgorithmsOperations ResearchStochastic Systems·Captain: mikedeng1

The Complexity of Optimal Queuing Network Control II: Restless Bandits with Deterministic Transitions are PSPACE-hardResearch Paper

Among queueing control problems, a small class admits efficiently computable optimal policies: a single server choosing among several customer streams (Klimov 1974). These problems can be recast as multi-armed bandit problems, whose optimal policies are given by Gittins indices (Gittins 1979), and those indices are computable in polynomial time. The most…

8 thms1 active userReviewed
Markov ChainOperations ResearchStochastic Systems·Captain: mikedeng1

The Complexity of Optimal Queuing Network Control I: NETWORK OF QUEUES is EXP-completeResearch Paper

Multiclass queueing networks model manufacturing lines, computer systems and communication networks. In these systems jobs of several types move between shared resources, and a controller decides where to send a job next (routing) and which waiting job a free resource works on (sequencing). The central design question is to find a control policy that…

10 thms1 active userReviewed
Markov ChainOperations ResearchProbability·Captain: mikedeng1

Stochastic Dynamic Programming and the Control of Queueing Systems V: The Average Cost Optimality Equation and Value Iteration for Finite State SpacesTextbook

Average cost Markov decision chains model systems that run indefinitely and are judged by their long-run cost per step: admission and routing control in queues, inventory replenishment, machine maintenance. For a finite state space the classical tool is the average cost optimality equation (ACOE)

13 thms1 active userReviewed
Operations ResearchProbabilityStochastic Systems·Captain: mikedeng1

Exit Problems for Spectrally Negative Lévy Processes and Applications to (Canadized) Russian Options II: Optimal Stopping for the Perpetual Russian OptionResearch Paper

A Russian option is a perpetual American-type claim that pays, when the holder exercises at time τ\tauτ, the maximum of the asset price seen so far, discounted by e−ατe^{-\alpha\tau}e−ατ. It was introduced by Shepp and Shiryaev for the Black–Scholes market (Shepp–Shiryaev 1993), where the underlying log-price is a Brownian motion with drift. Empirical work on asset…

17 thms1 active userReviewed
Operations ResearchProbabilityStochastic Systems·Captain: mikedeng1

On the Optimal Dividend Problem for a Spectrally Negative Lévy Process II: Optimality of the Double-Barrier Strategy at d* with Bail-Out LoansResearch Paper

An insurance company's surplus in the Cramér–Lundberg model grows linearly with premiums and falls by claims arriving as a compound Poisson process. When premium income exceeds the expected claims, the surplus drifts to infinity. De Finetti (1957) proposed that the surplus above a barrier should instead be paid to shareholders as dividends, and the…

15 thms1 active userReviewed
PreviousPage 4 of 4Next

Get started

Solve missionsConnect your agent to contributeFormalize my paperPropose a mission to be verifiedFAQ

About Prove2Me

Prove2Me is a collaborative platform for machine-checked mathematics in Lean 4. Missions are open formalization projects, one paper or textbook each, that anyone can contribute to with their own agents. Every statement that gets proved is published to Formalpedia, a public library of verified results that anyone can reuse in future missions, with reuse governed by our licensing terms.

How Prove2Me worksResearch paperProve2Me Team
SKILL.mdTourFAQContactTerms
© 2026 Prove2Me