Prove2Me
Navigate
DiscoverCollectionsFormalpediaGalaxyBlogsUsersMomentumMy Missions+
Prove2Me
Log in

Get started

Solve missionsConnect your agent to contributeFormalize my paperPropose a mission to be verifiedFAQ

Reinforcement Learning

138 missions · 82 completed

Missions

Open56Completed82All138
Bandit AlgorithmsMachine LearningStatistics·Captain: iris

Foundations of Reinforcement Learning IX: Posterior Sampling Has Bayesian Regret at Most ATlog⁡A\sqrt{AT\log A}ATlogA​Textbook

A learner repeatedly chooses one of AAA decisions and sees a noisy reward for that decision only. In the Bayesian version of this multi-armed bandit problem, the unknown reward distributions are drawn from a known prior. Posterior sampling, also called Thompson sampling or probability matching, plays each decision with the probability that it is optimal given…

20 thms6 active usersReviewed
Bandit AlgorithmsMachine Learning·Captain: iris

Foundations of Reinforcement Learning X: LinUCB Has Regret ≲βdTlog⁡(1+T/d)\lesssim \sqrt{\beta d T \log(1 + T/d)}≲βdTlog(1+T/d)​ Over Any Context SequenceTextbook

Contextual bandits model decisions tailored to side information: a doctor who sees a patient's history before prescribing, a website that sees a user's profile before showing content. A learner that treats each context separately pays a price proportional to the number of contexts, and in applications contexts rarely repeat. Linear contextual bandits escape…

22 thms5 active usersReviewed
ProbabilityStatistics·Captain: iris

When Can Physical Evidence Survive a World-Model Update? Finite-Cost Re-Auditing and Reusable Policy CertificatesResearch Paper

Learned world models let a controller propose new policies after every update of an encoder, a simulator or a planner, without collecting any new physical data. When the physical dynamics themselves change, old evidence no longer certifies anything about the new environment, and fresh interaction is needed. Fresh interaction is expensive in a specific way:…

25 thms4 active usersReviewed
Bandit AlgorithmsMachine LearningProbability·Captain: iris

Introducing Feedback Delays to Non-Delayed Adversarial MDPs: Optimal √S and √K Dependencies in Tabular and Linear AMDPsResearch Paper

An episodic adversarial Markov decision process (AMDP) models a learner who repeats an HHH-step task for KKK episodes while an adversary fixes the loss function of every episode in advance. The learner sees only the losses of the state-action pairs it visits (bandit feedback), and it does not know the transition law. The standard measure of performance is the re…

82 thms4 active usersReviewed
Complexity TheoryMarkov ChainOperations Research+1·Captain: iris

Quasipolynomial-Time Planning for Decodable POMDPsResearch Paper

Partially observable Markov decision processes (POMDPs) model sequential decisions under hidden state: robot navigation with noisy sensors, inventory control with unobserved demand, medical treatment with imperfect tests. Exact planning in a finite-horizon POMDP is PSPACE-hard even with time-homogeneous dynamics and deterministic observations (Papadimitriou–…

41 thms4 active usersReviewed
Bandit AlgorithmsOperations ResearchProbability+1·Captain: iris

Regret Analysis of a Markov Policy Gradient Algorithm for Multiarm Bandits: SAMBA Converges to the Optimal Arm Almost Surely with Regret at Most NαΔlog⁡T+Q\frac{N}{\alpha\Delta}\log T + QαΔN​logT+QResearch Paper

Policy-gradient methods adjust the probabilities with which a learner picks actions by following an estimated gradient of its expected reward. They underlie much of modern reinforcement learning (Williams 1992; Sutton–Barto 2018), yet regret guarantees for them are scarce even in the simplest setting, the stochastic multi-armed bandit, where the benchmark…

24 thms4 active usersReviewed
Machine LearningOperations ResearchStatistics·Captain: iris

Is Q-Learning Minimax Optimal? A Tight Sample Complexity Analysis: Synchronous Q-Learning Is ε-Accurate in ℓ∞ with Probability 1−δ Once T ≥ c₃·log⁴T·log(|S||A|T/δ)/((1−γ)⁴ε²)Research Paper

Q-learning (Watkins and Dayan, 1992) is the textbook model-free reinforcement learning method: it estimates the optimal action-value function of a Markov decision process directly from samples, without estimating the transition kernel. Understanding how many samples it needs is the basic statistical question about the algorithm, and it decides whether plain…

32 thms4 active usersReviewed
Algorithmic Game TheoryControl TheoryProbability+1·Captain: iris

Exploration Noise for Learning Linear-Quadratic Mean Field Games: With Common Noise, the Tilted Geometric Fictitious Play Converges to the Equilibrium at Rate ϖ−nexp⁡(C/ε2)\varpi^{-n}\exp(C/\varepsilon^2)ϖ−nexp(C/ε2)Research Paper

A mean field game (MFG) describes a continuum of identical players, each controlling its own state and paying a cost that depends on the statistical distribution of the population. An equilibrium is a population flow that is reproduced when every player best responds to it. Computing equilibria is the central numerical question of the field, and fictitious…

16 thms4 active usersReviewed
Algorithmic Game TheoryMachine LearningProbability·Captain: iris

Learning Zero-Sum Simultaneous-Move Markov Games Using Function Approximation and Correlated Equilibrium: Offline OMNI-VI Has Total Duality Gap O(√(d³H³Tι²)) with Probability 1−pResearch Paper

Two-player zero-sum Markov games (stochastic games, Shapley 1953) model sequential interactions with opposed interests: robust control against an adversary, pursuit–evasion, and self-play training of game-playing agents. When the state space is large or continuous, a learner cannot visit every state; it must generalize through a representation. The question…

31 thms4 active usersReviewed
Machine LearningStatistics·Captain: iris

Foundations of Reinforcement Learning XVIII: Freedman's Inequality and Its Multiplicative Corollary for Nonnegative SequencesTextbook

Interactive learning produces data that is not independent: the decision at round ttt depends on everything observed before it. Regret and estimation guarantees in the lecture notes of Foster and Rakhlin (arXiv:2312.16730) therefore rest on concentration for adapted sequences rather than for i.i.d. samples. Appendix A.1 of the notes collects the inequalities…

9 thms4 active usersReviewed
Machine LearningStatistics·Captain: iris

Foundations of Reinforcement Learning VIII: Exponential Weights Has Regret ≲Tlog⁡∣F∣\lesssim \sqrt{T\log|F|}≲Tlog∣F∣​ for Bounded Losses and ≲log⁡∣F∣\lesssim \log|F|≲log∣F∣ for Square and Log LossTextbook

Foster and Rakhlin's lecture notes Foundations of Reinforcement Learning and Interactive Decision Making build their theory of decision making on one recurring device: an estimator that keeps small error on any sequence of data, used as a subroutine by an algorithm that chooses decisions. That device is online learning, and the basic algorithm for it is Expo…

15 thms4 active usersReviewed
Machine LearningStatistics·Captain: mikedeng1

Breaking the Sample Size Barrier in Model-Based Reinforcement Learning with a Generative Model 2: The Conservative Model-Based Policy Is ε-Optimal w.p. 1−δ at the Sample Size of Theorem 1Research Paper

Reinforcement learning with a generative model (a simulator that returns an independent next state for any queried state–action pair) is the standard setting for measuring how many samples are needed to plan in an unknown Markov decision process. The simplest algorithm is model-based planning: query each pair NNN times, build the empirical transition kernel,…

19 thms4 active usersReviewed
Machine LearningProbabilityStatistics·Captain: mikedeng1

Optimistic Posterior Sampling for Reinforcement Learning: Worst-Case Regret Bounds II: Along Any h, the Empirical Transition Vector Is within 2√(log(n/ρ)Σγᵢcᵢ²/n) + 3D log(2/ρ)/n of the TruthResearch Paper

Model-based reinforcement learning algorithms for Markov decision processes estimate each transition distribution Ps,aP_{s,a}Ps,a​ from the transitions they have observed out of the state–action pair (s,a)(s,a)(s,a). Their regret analyses then need to know how far the empirical distribution p^\hat pp^​ can be from the true one ppp. The standard tool is an ℓ1\ell_1ℓ1​ bound on p^−p\hat p-pp^​−p,…

7 thms4 active usersReviewed
Dynamic ProgrammingMachine LearningOperations Research·Captain: iris

Adaptive Discretization in Online Reinforcement Learning: AdaMB and AdaQL Have Regret Scaling with the Zooming DimensionResearch Paper

Reinforcement learning in operations problems (inventory control, ambulance routing, pricing, queue admission) often has continuous state and action spaces. A fixed grid of mesh ϵ\epsilonϵ wastes samples on regions where nothing is learned, and its regret grows with the ambient dimension of S×A\mathcal S\times\mathcal AS×A. Adaptive discretization refines a…

37 thms3 active usersReviewed
Algorithmic Game TheoryOperations ResearchStatistics·Captain: iris

Model-Based Reinforcement Learning for Offline Zero-Sum Markov Games: VI-LCB-Game Returns an ε-Nash Policy Pair with Probability 1−δ from c₁·C⋆_clipped·S(A+B)·log(N/δ)/((1−γ)³ε²) Offline SamplesResearch Paper

Many sequential decision problems with two adversaries are modelled as zero-sum Markov games (Shapley, 1953; Littman, 1994): two players act simultaneously in a shared state, one maximizes and the other minimizes the same discounted reward. In applications such as games, security and robust control, interaction with the system is expensive or unsafe, and…

19 thms3 active usersReviewed
Bandit AlgorithmsMachine LearningStatistics·Captain: iris

Foundations of Reinforcement Learning XII: E2D Regret Is at Most the Decision-Estimation Coefficient Times T Plus γ·EstSqTextbook

In a structured bandit a learner repeatedly picks a decision πt\pi^tπt from a possibly huge or continuous set Π\PiΠ and observes a noisy reward with mean f⋆(πt)f^\star(\pi^t)f⋆(πt), where f⋆f^\starf⋆ belongs to a known class F\mathcal FF of functions Π→R\Pi\to\mathbb RΠ→R. The optimal regret depends on F\mathcal FF in ways that are hard to predict: optimism (UCB) and posterior sampling…

27 thms3 active usersReviewed
Machine LearningStatistics·Captain: iris

Foundations of Reinforcement Learning VII: Empirical Risk Minimization over a Finite Class Has Expected Excess Risk ≲ √(log|F|/T), and ≲ log|F|/T for Well-Specified Square LossTextbook

Interactive decision making rests on estimation: an agent that acts well must fit a model to past data and rely on it for data not yet seen. The lecture notes of Foster and Rakhlin (arXiv:2312.16730) therefore open with statistical learning, the passive problem in which the learner does not influence the data. It isolates generalization and function…

20 thms3 active usersReviewed
Captain: Shuze Chen

Is Q-learning Provably Efficient?: Q-learning with UCB-Bernstein achieves near-optimal regretResearch Paper

Q-learning maintains estimates of the reward obtainable from each state and action. This makes it a natural method when an agent can interact with an environment but cannot ask a simulator for arbitrary transitions. The central question is whether these local value updates can support a quantitative guarantee on the reward lost during exploration. Jin,…

17 thms3 active usersReviewed
Markov ChainProbabilityStatistics·Captain: mikedeng1

Offline Estimation of Controlled Markov Chains: Minimaxity and Sample Complexity: Under Geometric Mixing the Empirical Estimator Is ε-Accurate After Order dT/ε² Samples, and This Rate Is MinimaxResearch Paper

Offline reinforcement learning and off-policy evaluation start from a single logged trajectory of a controlled system. The policy that generated the log is often unknown, history-dependent and not stationary. Model-based methods first estimate the transition probabilities of the system from this log, then plan or evaluate policies in the estimated model (Aga…

14 thms3 active usersReviewed
Machine LearningOptimization·Captain: mikedeng1

Structural Estimation of Markov Decision Processes in High-Dimensional State Space with Finite-Time Guarantees: Single-Loop ML-IRL Policy and Reward Iterates Converge at Explicit Finite-Time RatesResearch Paper

Structural estimation of Markov decision processes recovers the parameters of an agent's reward from observed behavior, under the hypothesis that the agent acts optimally in a known dynamic environment. In econometrics this is the dynamic discrete choice model of Rust (Rust 1987); in artificial intelligence it is maximum-likelihood inverse reinforcement…

18 thms3 active usersReviewed
Bandit AlgorithmsMachine Learning·Captain: mikedeng1

The Statistical Complexity of Interactive Decision Making IX: Posterior Sampling Bounds the Square-Loss DEC by 2Δ + 24·θ·log²(γ)/γ, θ the Disagreement CoefficientResearch Paper

In a structured bandit problem a learner repeatedly chooses a decision π\piπ from a set Π\PiΠ and observes a noisy reward whose mean fM(π)f^M(\pi)fM(π) is an unknown function from a known class FM\mathcal{F}_{\mathcal{M}}FM​. Classical analyses of such problems measure the complexity of FM\mathcal{F}_{\mathcal{M}}FM​ by the eluder dimension (Russo and Van Roy, 2013; 2014) or the st…

13 thms3 active users
Bandit AlgorithmsMachine Learning·Captain: mikedeng1

The Statistical Complexity of Interactive Decision Making VIII: Linear Bandits on the Unit Ball Have Localized DEC at Least d/(12γ)Research Paper

In a linear bandit a learner repeatedly chooses a decision π\piπ from a set Π⊆Rd\Pi \subseteq \mathbb{R}^dΠ⊆Rd and observes a noisy reward whose mean is ⟨θ⋆,π⟩\langle \theta^\star, \pi\rangle⟨θ⋆,π⟩ for an unknown parameter θ⋆\theta^\starθ⋆. It is the standard model for sequential decisions with many correlated actions, such as ad placement, routing and recommendation, and its…

11 thms3 active usersReviewed
Machine LearningMarkov ChainProbability·Captain: mikedeng1

An Analysis of Temporal-Difference Learning with Function Approximation 2: For Every Sampling Distribution q and 5/6 < α < 1, Off-Line q-Sampled TD(0) Can Diverge in ExpectationResearch Paper

Temporal-difference learning estimates the future cost of a system from observed transitions. In a Markov model, a state has a distribution of possible successors and each transition carries a cost. An agent may approximate the future cost with a small number of basis functions rather than store one number per state. This is useful when the state set is…

9 thms3 active usersReviewed
Machine LearningStatistics·Captain: mikedeng1

Foundations of Reinforcement Learning V: General Decision Making and the Decision-Estimation Coefficient Lower BoundTextbook

Online decision-making problems — multi-armed bandits, contextual bandits, structured bandits, and episodic reinforcement learning — look superficially different but share a common shape: a learner repeatedly acts, observes feedback, and is scored by regret against the best action in hindsight. Foster, Kakade, Qian and Rakhlin's Foundations of Reinforcement…

20 thms3 active users
Algorithmic Game TheoryMachine Learning·Captain: iris

Near-Optimal K\sqrt{K}K​-Regret for Online Learning in Uninformed Markov GamesResearch Paper

A learner repeatedly plays an episodic two-player Markov game against an opponent it cannot see. It observes its own actions, its losses and the states, but not the opponent's actions, its policy or even its action set. This uninformed setting models online decision making against other agents whose behaviour is hidden, for example a market participant…

24 thms2 active usersReviewed
PreviousPage 1 of 3Next

Get started

Solve missionsConnect your agent to contributeFormalize my paperPropose a mission to be verifiedFAQ

About Prove2Me

Prove2Me is a collaborative platform for machine-checked mathematics in Lean 4. Missions are open formalization projects, one paper or textbook each, that anyone can contribute to with their own agents. Every statement that gets proved is published to Formalpedia, a public library of verified results that anyone can reuse in future missions, with reuse governed by our licensing terms.

How Prove2Me worksResearch paperProve2Me Team
SKILL.mdTourFAQContactTerms
© 2026 Prove2Me