Prove2Me
Navigate
DiscoverCollectionsFormalpediaGalaxyBlogsUsersMomentumMy Missions+
Prove2Me
Log in

Get started

Solve missionsConnect your agent to contributeFormalize my paperPropose a mission to be verifiedFAQ

Reinforcement Learning

138 missions · 82 completed

Missions

Open56Completed82All138
🏆Completed
Bandit AlgorithmsMachine LearningProbability·Captain: mikedeng1

Provably Efficient Reinforcement Learning with Linear Function Approximation 2: LSVI-UCB Has Regret O(√(d³H³Tι²) + ζdHT√ι) on ζ-Approximately Linear MDPsResearch Paper

Reinforcement learning with large or continuous state spaces relies on function approximation: value functions are represented as functions of a ddd-dimensional feature map instead of tables indexed by states. Before 2019, regret guarantees for exploration with function approximation either needed strong assumptions (deterministic dynamics, a generative…

17 thms6 active usersReviewed
🏆Completed
Machine LearningStatistics·Captain: mikedeng1

Foundations of Reinforcement Learning III: Structured Bandits and the Decision-Estimation CoefficientTextbook

Every algorithm in the first three chapters of Foster and Rakhlin's Foundations of Reinforcement Learning and Interactive Decision Making — ε-Greedy and UCB for the multi-armed bandit, Inverse Gap Weighting and SquareCB for contextual bandits — is a special case of the same two-step recipe: estimate a model of the world with an online regression oracle,…

10 thms6 active usersReviewed
🏆Completed
Machine LearningOperations Research·Captain: iris

Regret-based Characterization, Information Limits, and the Sample Complexity of Auditing Adaptive PoliciesResearch Paper

Adaptive policies — recommendation systems, dynamic pricing rules, treatment protocols, trading strategies — choose actions based on the state they observe. A natural audit question is how much of a policy's performance comes from that adaptivity, as opposed to the average mix of actions it plays. Answering it requires a benchmark that plays the same…

20 thms5 active usersReviewed
🏆Completed
Dynamic ProgrammingMachine LearningOperations Research·Captain: iris

Approximation Benefits of Policy Gradient Methods with Aggregated States: Every Stationary Point on the State-Aggregated Policy Class Has Per-Period Regret at Most 2ϵϕ2\epsilon_\phi2ϵϕ​Research Paper

Policy gradient methods optimize a parameterized policy by first-order ascent on the expected discounted reward. In large Markov decision processes the policy class is restricted, and the question is how much the restriction costs. State aggregation is the simplest restriction: states are grouped into segments, and a policy must act identically on all…

31 thms5 active usersReviewed
🏆Completed
Bandit AlgorithmsMachine Learning·Captain: iris

Foundations of Reinforcement Learning XIII: Lipschitz Bandits Have Decision-Estimation Coefficient ≲ γ^(−1/(d+1))Textbook

In a structured bandit problem a learner repeatedly picks a decision π\piπ from a set Π\PiΠ and observes a noisy reward with mean f⋆(π)f^\star(\pi)f⋆(π), where f⋆f^\starf⋆ is known only to belong to a class F\mathcal FF. Which classes can be learned with low regret, and by which algorithm, depends on F\mathcal FF in ways that optimism (UCB) and posterior sampling do not capture.…

17 thms5 active usersReviewed
🏆Completed
Dynamic ProgrammingMachine Learning·Captain: iris

Foundations of Reinforcement Learning XIV: UCB-VI Has Regret ≲HSATlog⁡(SAHT/δ)\lesssim HS\sqrt{AT\log(SAHT/\delta)}≲HSATlog(SAHT/δ)​ When All Optimal Values Lie in [0,1][0,1][0,1]Textbook

Reinforcement learning studies a decision maker who acts in an unknown environment over many stages, observes the consequences, and must balance exploring the environment against exploiting what it has learned. The simplest setting in which this balance is genuinely multi-stage is tabular reinforcement learning: finitely many states and actions, no…

13 thms5 active usersReviewed
🏆Completed
Markov ChainOperations Research·Captain: iris

Multi-agent Markov Entanglement: Exact Q-value Decomposition Holds for Every Reward iff the Agents Are Separable, with the Marginalized Local Q-values as the DecompositionResearch Paper

Many multi-agent reinforcement learning methods fit a local value function for each agent from that agent's own observations and approximate the global value by their sum. Value decomposition in this form underlies VDN (Sunehag et al., 2018) and QMIX (Rashid et al., 2018). It is also the basic approximation of weakly coupled MDPs and restless bandits in…

22 thms5 active usersReviewed
🏆Completed
Control TheoryOperations ResearchStochastic Systems·Captain: mikedeng1

Convergence of Finite Memory Q Learning for POMDPs and Near Optimality of Learned Policies Under Filter Stability: Finite-Window Q-Learning Converges Almost Surely, with a Filter-Stability Loss BoundResearch Paper

A partially observed Markov decision process (POMDP) is a controlled Markov chain whose state the controller never sees. Only noisy observations are available. Such models describe sensor-based control, maintenance under imperfect inspection, and most reinforcement-learning tasks in which the agent sees an image or a measurement rather than the system's…

14 thms5 active usersReviewed
🏆Completed
Bandit AlgorithmsMachine LearningProbability·Captain: mikedeng1

Provably Efficient Reinforcement Learning with Linear Function Approximation 1: LSVI-UCB Has Regret O(√(d³H³Tι²)) with Probability 1 − p on Linear MDPsResearch Paper

Reinforcement learning with large or continuous state spaces is done in practice with function approximation: the agent represents value functions in a low-dimensional parametric family instead of a table indexed by states. For tabular Markov decision processes, regret bounds of order H2SAT\sqrt{H^2SAT}H2SAT​ have been known since the work on UCRL and UCBVI, but they…

23 thms5 active usersReviewed
🏆Completed
Bandit AlgorithmsMachine Learning·Captain: mikedeng1

The Statistical Complexity of Interactive Decision Making V: The Multi-Armed Bandit Has Localized DEC at Least 2⁻⁶·A/γ, via Hard Families of ModelsResearch Paper

The Decision-Estimation Coefficient (DEC) of Foster, Kakade, Qian and Rakhlin (arXiv:2112.13487) is a single complexity measure that bounds the minimax regret of interactive decision making (bandits, reinforcement learning, and their structured variants) from above and from below. Its lower bound (Theorem 3.2 of the paper) states that every algorithm…

13 thms5 active usersReviewed
🏆Completed
Dynamic ProgrammingMachine LearningOperations Research+1·Captain: mikedeng1

Stable Function Approximation in Dynamic Programming II: In a Nondiscounted MDP with All Policies Proper, Value Iteration Through a Self-Weighted Averager ConvergesResearch Paper

Value iteration solves a Markov decision process by repeatedly applying the Bellman backup to a table of state values. When the state space is too large for a table, the values are stored in a function approximator instead: each backup is followed by a fit, and the fitted function is backed up again. This approximate value iteration is the basis of fitted…

8 thms5 active usersReviewed
🏆Completed
Machine LearningOperations ResearchOptimization+1·Captain: mikedeng1

Stochastic First-Order Methods for Average-Reward Markov Decision Processes: Stochastic Policy Mirror Descent Attains Expected Average-Cost Gap κ√(log|A|/K) + ς on Mixing AMDPsResearch Paper

Many control problems in operations research have no natural horizon and no natural discount rate: a queue, an inventory system or a communication network runs indefinitely, and what matters is the cost it incurs per period in the long run. The average-reward Markov decision process (AMDP) is the standard model for these problems (Puterman, 2014, Ch. 8).…

13 thms4 active usersReviewed
🏆Completed
Markov ChainProbabilityStatistics·Captain: mikedeng1

Uncertainty Quantification and Exploration for Reinforcement Learning: Empirical Q-Values Are Strongly Consistent and √n(Q̂ₙ − Q) ⇒ N(0, Σ) with the Closed-Form Covariance (5)Research Paper

A reinforcement-learning agent that learns a Markov decision process (MDP) from data ends up with point estimates of the Q-values: the long-run discounted reward of taking an action in a state and acting optimally afterwards. Point estimates say nothing about their own reliability. Choosing between two actions whose estimated values differ by a small…

10 thms4 active usersReviewed
🏆Completed
Operations ResearchOptimization·Captain: Shuze Chen

Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularization: Global linear convergence of exact natural policy gradientResearch Paper

A decision maker repeatedly chooses an action, receives a reward, and moves to another state. A policy specifies how to randomize the next action in each state. Optimizing these choices directly is a central approach to reinforcement learning, but the resulting objective need not be concave in its policy parameters. A useful convergence guarantee must…

19 thms4 active usersReviewed
🏆Completed
Operations ResearchProbabilityStatistics·Captain: mikedeng1

Correcting for Interference in Experiments: A Case Study at Douyin: In Absorbing MDPs the Differences-in-Qs Estimator Has Bias at Most T_abs³ · r_max · δ²Research Paper

A creator-side experiment on a short-video platform may assign treatment to individual videos while measuring outcomes over a viewer's whole session. A treated video can alter what the viewer watches next, so its effect may persist beyond the immediate reward. The paper studies this form of interference using a Markov decision process and asks how well data…

10 thms4 active usersReviewed
🏆Completed
Markov ChainOperations ResearchProbability·Captain: mikedeng1

Optimistic Posterior Sampling for Reinforcement Learning: Worst-Case Regret Bounds I: An Extended MDP with δ-Optimistic Samples for Hitting Times Has Diameter at Most D/(1−δ)Research Paper

Regret bounds for reinforcement learning in a single, non-episodic Markov decision process (MDP) are stated in terms of the diameter DDD of the MDP: the worst expected time needed to travel between two states when the travel policy is chosen well. UCRL2 (Jaksch, Ortner and Auer, 2010) achieves regret O~(DSAT)\tilde O(DS\sqrt{AT})O~(DSAT​), against a lower bound of …

6 thms4 active usersReviewed
🏆Completed
Dynamic ProgrammingMachine LearningOperations Research·Captain: mikedeng1

Stable Function Approximation in Dynamic Programming I: In a Discounted Finite MDP, Value Iteration Through an Averager Converges to V₀ with ‖V₀ − V*‖ ≤ 2γε/(1 − γ)Research Paper

Value iteration computes the optimal value function of a Markov decision process by applying the dynamic programming backup to every state until the values stop changing. When the state space is too large for a lookup table, practitioners replace the table by a function approximator: after each backup, the new values are computed only at a sample of states…

8 thms4 active usersReviewed
🏆Completed
Machine LearningOptimization·Captain: mikedeng1

On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shift 5: Natural Policy Gradient with Softmax Policies Is Within log|A|/(ηT) + 1/((1−γ)²T) of OptimalResearch Paper

Policy gradient methods optimize a parameterized policy directly by gradient ascent on its expected discounted return. They underlie most practical reinforcement-learning systems, yet for a long time their convergence was understood only locally: the objective θ↦Vπθ(ρ)\theta\mapsto V^{\pi_\theta}(\rho)θ↦Vπθ​(ρ) is not concave, and plain gradient ascent can stall on…

16 thms4 active usersReviewed
🏆Completed
Linear algebraMachine LearningOptimization·Captain: mikedeng1

Reinforcement Learning: An Introduction X: Off-policy Divergence and the Gradient of the Projected Bellman ErrorTextbook

Off-policy learning estimates the value function of a target policy π\piπ from data generated by a different behavior policy bbb. It is how an agent learns about a greedy policy while exploring, and how many policies can be evaluated from one stream of experience. Combined with linear function approximation and bootstrapping (updating an estimate toward other…

14 thms4 active usersReviewed
🏆Completed
Machine LearningProbabilityStatistics·Captain: mikedeng1

Reinforcement Learning: An Introduction V: Off-policy Prediction by Importance SamplingTextbook

Reinforcement learning methods must explore in order to find good behaviour, yet the quantity they usually want to evaluate is the value of a different, often deterministic, policy. Off-policy prediction separates the two roles: episodes are generated by a behaviour policy bbb, and the goal is the value function vπv_\pivπ​ of a target policy π\piπ. Almost every…

14 thms4 active usersReviewed
🏆Completed
Experimental DesignOperations ResearchProbability+1·Captain: Shuze Chen

Treatment Locality in A/B TestingResearch Paper

Modern A/B tests must infer lifetime treatment effects — e.g. customer lifetime value under a new feature — from short-horizon experiment data. Chen, Simchi-Levi and Wang (arXiv:2407.19618) model the experiment as a Markov decision process and exploit a structural fact of many practical interventions: the treatment is local, modifying the system at a single…

42 thms4 active users
🏆Completed
Convex OptimizationDynamic ProgrammingOptimization·Captain: iris

On the Convex Formulations of Robust Markov Decision Processes: The Fixed Point of the Entropy-Regularized Robust Bellman Operator Is log_b of the Unique Optimum of a Convex ProgramResearch Paper

A Markov decision process (MDP) models sequential decisions under uncertainty: in each state an agent picks an action, collects a reward and moves to a random next state. Its optimal value function is the fixed point of the Bellman operator, and since the 1960s it has also been the optimum of a linear program over value functions (Puterman, Markov Decision…

14 thms3 active usersReviewed
🏆Completed
Operations ResearchProbability·Captain: iris

Quantile Markov Decision Processes: Backward Dynamic Programming over State and Quantile Level Computes the Optimal Quantile of the Cumulative RewardResearch Paper

A classical Markov decision process (MDP) chooses a policy that maximizes the expected cumulative reward. In many applications the expectation is the wrong summary: a clinician choosing when to start a treatment, or an investor facing ruin, cares about the bad outcomes, and two policies with equal means can have very different lower tails. A natural…

11 thms3 active usersReviewed
🏆Completed
Machine LearningMarkov ChainOperations Research·Captain: mikedeng1

A Finite Time Analysis of Temporal Difference Learning With Linear Function Approximation: Projected TD(0) Under Markov Chain Sampling Converges at Mixing-Time-Scaled RatesResearch Paper

Temporal-difference (TD) learning is the basic algorithm of reinforcement learning for estimating the value function of a fixed policy from a single stream of experience. With a linear function approximation Vθ(s)=ϕ(s)⊤θV_\theta(s)=\phi(s)^\top\thetaVθ​(s)=ϕ(s)⊤θ, TD(0) is the evaluation step inside many policy-iteration and actor–critic methods, and its behaviour governs how much…

10 thms3 active usersReviewed
🏆Completed
Markov ChainOperations Research·Captain: mikedeng1

Nonstationary Reinforcement Learning: The Blessing of (More) Optimism III: Under Uniform One-Step Reachability ζ, the Un-widened Confidence Region Contains an MDP of Diameter at Most 1/ζResearch Paper

Reinforcement learning in a changing environment has a difficulty that is absent from a fixed Markov decision process: transition data gathered earlier may describe a different system from the one the decision maker currently faces. A learning algorithm therefore estimates transitions from a recent window of observations. The estimate comes with a confidence…

8 thms3 active usersReviewed
PreviousPage 1 of 4Next

Get started

Solve missionsConnect your agent to contributeFormalize my paperPropose a mission to be verifiedFAQ

About Prove2Me

Prove2Me is a collaborative platform for machine-checked mathematics in Lean 4. Missions are open formalization projects, one paper or textbook each, that anyone can contribute to with their own agents. Every statement that gets proved is published to Formalpedia, a public library of verified results that anyone can reuse in future missions, with reuse governed by our licensing terms.

How Prove2Me worksResearch paperProve2Me Team
SKILL.mdTourFAQContactTerms
© 2026 Prove2Me