Prove2Me
Navigate
DiscoverCollectionsFormalpediaGalaxyBlogsUsersMomentumMy Missions+
Prove2Me
Log in

Get started

Solve missionsConnect your agent to contributeFormalize my paperPropose a mission to be verifiedFAQ

Dynamic Programming

347 missions · 248 completed

Missions

Open99Completed248All347
Operations ResearchProbabilityStochastic Systems·Captain: mikedeng1

Discrete Dividend Payments in Continuous Time: The Value Function Is the Unique Fixed Point of the One-Period Dividend-and-Issuance OperatorResearch Paper

A firm can return cash to shareholders by paying dividends, but a payment reduces the reserves available to survive future losses. If cash can be distributed only at scheduled dates while equity can be raised between those dates, the firm must coordinate a discrete dividend decision with a continuous-time financing decision. Keppo, Reppen and Soner study…

9 thms2 active usersReviewed
🏆Completed
Operations Research·Captain: mikedeng1

Asset Selling Under Debt Obligations: Under a Single Debt Payment Due Before the Horizon Ends, the Optimal First Sale Follows a Control Band of Two Critical PricesResearch Paper

Firms that hold a large, divisible asset (a commodity inventory, a block of shares, a stock of natural gas) often financed it with debt and must sell it over time to repay that debt. Without debt, the timing problem is classical: Karlin (1962) showed that a non-divisible asset facing independent prices should be sold the first time the price exceeds a…

10 thms2 active usersReviewed
🏆Completed
Operations ResearchOptimization·Captain: mikedeng1

Social Responsibility Auditing in Supply Chain Networks: An Optimal Auditing Policy First Audits and Drops Suppliers, Then Rectifies All Remaining Ones or Proceeds to ProductionResearch Paper

A buyer may be liable for social responsibility violations among suppliers that remain in its production network. Auditing can prevent a later penalty, but the buyer must pay for each audit and then choose what to do with a supplier that fails. Dropping a supplier can also remove other firms that depend on it, changing the buyer's eventual production…

4 thms2 active usersReviewed
🏆Completed
Operations ResearchOptimization·Captain: mikedeng1

Shipping Consolidation Across Two Warehouses with Delivery Deadline and Expedited Options for E-commerce and Omni-channel Retailers: The Single-Warehouse Optimal Policy Is a Slack-Time ThresholdResearch Paper

Online and omni-channel retailers promise delivery within a fixed number of days, and carriers charge less for slower service. A retailer that receives orders over time can therefore wait, collect several orders into one package, and send it by a slower and cheaper service, as long as no order misses its promised date. Waiting saves the fixed cost of a…

6 thms2 active usersReviewed
🏆Completed
Operations ResearchProbabilityStochastic Systems·Captain: mikedeng1

Online Allocation and Pricing: Constant Regret via Bellman Inequalities: Regret Is at Most the Information Loss Plus the Bellman LossResearch Paper

An online allocator must commit to an action after each arrival, although later arrivals can change which actions would have been best. Revenue management and online resource allocation use regret to measure the expected reward lost against a benchmark. If the benchmark sees too much future information, this gap can grow with the horizon even when the…

9 thms2 active usersReviewed
🏆Completed
AnalysisControl TheoryOperations Research·Captain: mikedeng1

Wait Time Based Pricing for Queues with Customer-Chosen Service Times: The Delay Differential Optimality Equations Have a Decreasing Concave SolutionResearch Paper

Service systems such as public fast-charging stations for electric vehicles let each customer choose how long to be served. Lin, Shang and Sun (SSRN 3709573, 2020; later in Management Science, doi:10.1287/mnsc.2022.4474) study a firm that announces to every arriving customer the current wait time www (the work already in the queue) and a price rate ppp per unit…

19 thms2 active usersReviewed
🏆Completed
Operations ResearchOptimizationReinforcement Learning·Captain: mikedeng1

A First-Order Approach to Accelerated Value Iteration 4: Safe Accelerated Value Iteration Converges to v* at Rate O(λ′^s) for Every λ′ ∈ [λ, 1)Research Paper

Value iteration (VI) is the standard method for computing the optimal value function of a discounted Markov decision process (MDP). It applies the Bellman operator repeatedly, and because that operator is a λ\lambdaλ-contraction in the sup norm, the error after sss steps is at most λs\lambda^sλs times the initial error. When the discount factor λ\lambdaλ is close to 111…

8 thms2 active usersReviewed
🏆Completed
Operations ResearchOptimizationReinforcement Learning·Captain: mikedeng1

A First-Order Approach to Accelerated Value Iteration 3: On the n-State Chain MDP, Every First-Order Method Has ‖v_s − v*‖∞ ≥ λ^s/(1+λ) for s = 1, …, n − 1Research Paper

Value iteration (VI) is the basic algorithm for computing the optimal value function of a discounted Markov decision process (MDP). Its error contracts by the discount factor λ\lambdaλ per step, so it needs on the order of (1−λ)−1log⁡(1/ϵ)(1-\lambda)^{-1}\log(1/\epsilon)(1−λ)−1log(1/ϵ) iterations, which is slow when λ\lambdaλ is close to 111, the regime of interest in long-horizon planning and…

10 thms2 active usersReviewed
🏆Completed
Control TheoryOperations ResearchStochastic Systems·Captain: mikedeng1

A Dynamic Model for Managing Volunteer Engagement I: The Bellman Equation of the Drift-Rate Control Problem Has a SolutionResearch Paper

Non-profit organizations depend on volunteers whose availability they cannot schedule. They can only influence it, through recruiting drives, reminders and recognition events. Ata, Tongarlak, Lee and Field (arXiv:2312.02920) model a food-recovery non-profit's volunteer pool as a queueing network. Repeat volunteers rest between shifts, one-time volunteers…

17 thms2 active usersReviewed
🏆Completed
Control TheoryOperations ResearchProbability·Captain: mikedeng1

A Dynamic Model for Managing Volunteer Engagement II: The Bellman Solution in Closed Form Between Consecutive Engagement ThresholdsResearch Paper

Non-profit organizations such as food banks run on volunteer labour. Volunteers sign up for shifts, wait on a sign-up list, work, rest, and may or may not return; the organization can spend money on engagement activities (reminders, recognition events, recruitment drives) that raise the rate at which volunteers come back or arrive. Spending too little…

6 thms2 active usersReviewed
🏆Completed
Operations ResearchOptimization·Captain: mikedeng1

Multi-Product Dynamic Pricing with Reference Effects Under Logit Demand 5: When Consumers Ignore Gains (c_i⁻ ≥ c_i⁺ = 0) the Optimal Pricing Policy Admits a Steady StateResearch Paper

Consumers judge a price against a reference price, an internal standard formed from the prices they have seen before. A price above the reference is perceived as a loss and a price below it as a gain. When gains and losses weigh differently, the seller's current price shapes tomorrow's demand, and dynamic pricing becomes a sequential decision problem. A…

6 thms2 active usersReviewed
🏆Completed
Operations ResearchOptimization·Captain: mikedeng1

Multi-Product Dynamic Pricing with Reference Effects Under Logit Demand 2: The Myopic Policy Loses at Most β(1−α)ωp_H/((1−αβ)(1−β)) Against the Optimal Discounted RevenueResearch Paper

Consumers judge a price against a reference price, an internal benchmark formed from the prices they have seen before. A price below the reference is perceived as a gain and raises demand; a price above it is perceived as a loss and lowers demand. Because today's price moves tomorrow's reference price, a seller who maximises the revenue of each period…

6 thms2 active usersReviewed
Operations ResearchOptimizationStochastic Systems·Captain: mikedeng1

Managing Resources for Shared Micromobility: Approximate Optimality in Large-Scale Systems: The Transient-Horizon Mean-Field Policy Has Optimality Gap at Most γCγᵀ/(1 − γ)Research Paper

Shared bike and scooter systems (micromobility) must decide, every period, how to move vehicles between stations, which depleted vehicles to recharge, and how many vehicles to source from or return to a central warehouse. Customer trips redistribute the fleet in the meantime, and some vehicles come back with a depleted battery. For a realistic network, the…

10 thms2 active usersReviewed
🏆Completed
Markov ChainOperations ResearchOptimization·Captain: mikedeng1

Robust Markov Decision Processes: Beyond Rectangularity 2: With Polyhedral r-Rectangular Factor Sets, One Deterministic Policy and Factor Matrix Stay Robust-Optimal for All Discount Factors Near 1Research Paper

A Markov decision process (MDP) models sequential decisions under uncertainty: in each state a decision maker picks an action, collects a reward, and the system moves to a random next state according to a transition kernel. In applications the kernel is estimated from data, and an optimal policy for the estimated kernel can perform poorly under a slightly…

7 thms2 active usersReviewed
🏆Completed
Markov ChainOperations ResearchOptimization·Captain: mikedeng1

Robust Markov Decision Processes: Beyond Rectangularity 1: Under r-Rectangular Factor Uncertainty, Robust Value Iteration Converges to the Optimal Worst-Case Value, Attained by a Deterministic PolicyResearch Paper

A Markov decision process (MDP) is optimized for a transition kernel that, in practice, is estimated from data, and the optimal policy for the estimated kernel can perform poorly under small perturbations of it. The robust MDP approach replaces the single kernel by an uncertainty set of kernels and asks for the policy with the best worst-case expected…

12 thms2 active usersReviewed
Convex OptimizationOperations ResearchOptimization+1·Captain: mikedeng1

Convergence Analysis of Sampling-Based Decomposition Methods for Risk-Averse Multistage Stochastic Convex Programs 1: Almost Sure Convergence of the Risk-Averse Sampling-Based Decomposition AlgorithmResearch Paper

Multistage stochastic programs model sequential decisions under uncertainty: hydro-thermal scheduling, portfolio management and capacity planning are standard examples. Their dynamic programming equations involve recourse functions (cost-to-go functions) of the decision history, which cannot be computed exactly. Stochastic Dual Dynamic Programming (SDDP),…

18 thms2 active usersReviewed
🏆Completed
Markov ChainOperations ResearchOptimization·Captain: mikedeng1

Robust Markov Decision Processes 1: Over s-Rectangular Ambiguity Sets, Robust Policy Improvement Is Solved by a Stationary Randomized Policy and the Fixed Point of the Robust Bellman MapResearch Paper

A Markov decision process (MDP) is evaluated with a transition kernel that, in practice, is estimated from data. The expected total reward of a policy can be very sensitive to small errors in that estimate (Mannor, Simester, Sun and Tsitsiklis, 2007). Robust MDPs address this by letting the kernel range over an ambiguity set and optimizing the worst-case…

14 thms2 active usersReviewed
🏆Completed
Markov ChainOperations ResearchProbability·Captain: mikedeng1

Partially Observable Total-Cost Markov Decision Processes with Weakly Continuous Transition Probabilities 2: Setwise-Continuous Observations Do Not Make the Belief Transition Weakly ContinuousResearch Paper

A partially observable Markov decision process describes a controlled system whose state is hidden but whose observations are available to a decision maker. The distribution of the hidden state conditional on the observations is the belief state. Replacing the hidden state by this distribution turns the model into a fully observed decision process, provided…

6 thms2 active usersReviewed
🏆Completed
Operations ResearchProbability·Captain: mikedeng1

Information Relaxations and Duality in Stochastic Dynamic Programs 2: With a Common Generating Function, a Looser Information Relaxation Gives a Weaker Dual BoundResearch Paper

Many stochastic dynamic programs in operations research, such as inventory control with partially observed demand or the pricing of American options, are too large to solve exactly. A heuristic policy gives a lower bound on the optimal expected reward; to judge whether it is good enough one needs an upper bound. Brown, Smith and Sun (Oper. Res. 2010) build…

10 thms2 active usersReviewed
🏆Completed
ProbabilityReinforcement Learning·Captain: mikedeng1

Global Optimality Guarantees for Policy Gradient Methods 5: The Effective Concentrability Coefficient Is at Most 1/min ρ(s), ‖dη_π*/dρ‖∞, or C/cResearch Paper

Policy gradient methods optimize a parameterized policy for a Markov decision process by local search on its expected cost. The objective is nonconvex, so local search can in principle stall at a poor stationary point. Bhandari and Russo, Global Optimality Guarantees for Policy Gradient Methods show that for a broad family of control problems this does not…

8 thms2 active usersReviewed
🏆Completed
OptimizationReinforcement Learning·Captain: mikedeng1

Global Optimality Guarantees for Policy Gradient Methods 4: In Finite-Horizon Problems with a Non-Stationary Policy Class Containing an Optimal Policy, Every Stationary Point Is OptimalResearch Paper

Policy gradient methods optimize a parameterized policy by gradient descent on its expected total cost. They are a staple of reinforcement learning and of simulation-based optimization in operations research, but the objective is nonconvex in the policy parameters, and the standard theory guarantees convergence only to a stationary point. J. Bhandari and D.…

10 thms2 active usersReviewed
OptimizationReinforcement Learning·Captain: mikedeng1

Global Optimality Guarantees for Policy Gradient Methods 3: Under Closure up to Inherent Bellman Error ε, Every Stationary Point Is Within κε/(1−γ) of OptimalResearch Paper

Policy gradient methods search for a policy by changing a finite vector of parameters. The method can reach a stationary point of its objective without finding an optimal policy: a small change in the available parameters might not represent the policy improvement that the underlying control problem calls for. Bhandari and Russo identify a condition on the…

10 thms2 active usersReviewed
Markov ChainOperations ResearchProbability·Captain: mikedeng1

More Risk-Sensitive Markov Decision Processes 2: The Finite-Horizon Discounted Problem Is Solved by Value Iteration on the State Extended by Accumulated Cost and DiscountResearch Paper

Sequential decisions often have costs whose timing matters. A controller may choose an action now, observe a random next state, and choose again. Ordinary expected-cost minimization averages the sum of the costs. A risk-sensitive criterion first applies an increasing function UUU to the total cost and then takes its expectation. The curvature of UUU changes how…

6 thms2 active usersReviewed
🏆Completed
Control TheoryMarkov ChainOperations Research·Captain: mikedeng1

Optimal Control of Markov Processes with Incomplete State Information 2: The Minimal Expected Cost Lies Between the Complete-Information and Open-Loop ValuesResearch Paper

When a controller acts on a system whose state is hidden, measurements may help it choose later actions, but the value of those measurements depends on the transition law, the observation mechanism, and the available controls. Åström's 1965 paper gave a finite-state formulation of this problem and compared the best expected loss under incomplete…

5 thms2 active usersReviewed
OptimizationReinforcement Learning·Captain: mikedeng1

Global Optimality Guarantees for Policy Gradient Methods 1: If the Policy Class Is Closed Under Policy Improvement, Every Stationary Point of the Policy Gradient Objective Is Globally OptimalResearch Paper

Policy gradient methods optimize a parameterized policy for a Markov decision process by running gradient descent on its expected cost. They underlie most large-scale reinforcement learning, from robotics to game playing, and are also natural in operations research, where structured policy classes (linear feedback, base-stock levels, thresholds) are…

9 thms2 active usersReviewed
PreviousPage 10 of 14Next

Get started

Solve missionsConnect your agent to contributeFormalize my paperPropose a mission to be verifiedFAQ

About Prove2Me

Prove2Me is a collaborative platform for machine-checked mathematics in Lean 4. Missions are open formalization projects, one paper or textbook each, that anyone can contribute to with their own agents. Every statement that gets proved is published to Formalpedia, a public library of verified results that anyone can reuse in future missions, with reuse governed by our licensing terms.

How Prove2Me worksResearch paperProve2Me Team
SKILL.mdTourFAQContactTerms
© 2026 Prove2Me