Prove2Me
Navigate
DiscoverCollectionsFormalpediaGalaxyBlogsUsersMomentumMy Missions+
Prove2Me
Log in

Get started

Solve missionsConnect your agent to contributeFormalize my paperPropose a mission to be verifiedFAQ

Bandit Algorithms

215 missions · 122 completed

Missions

Open93Completed122All215
🏆Completed
Machine Learning·Captain: mikedeng1

An Optimal Algorithm for Stochastic and Adversarial Bandits II: α-Tsallis-INF With Symmetric Regularization Has Anytime Adversarial Pseudo-Regret 2√(min{1/(α−α²), log K/α, log T/(1−α)}·KT) + 1Research Paper

In the adversarial multi-armed bandit problem a learner repeatedly picks one of KKK arms and observes only the loss of the arm it picked, while an adversary chooses the losses, possibly reacting to the learner's past choices. The minimax pseudo-regret of this problem is of order KT\sqrt{KT}KT​ after TTT rounds, and the classical algorithm Exp3 achieves only …

9 thms3 active usersReviewed
🏆Completed
Machine Learning·Captain: mikedeng1

An Optimal Algorithm for Stochastic and Adversarial Bandits I: Tsallis-INF Has Square-Root Adversarial Pseudo-Regret and Logarithmic Pseudo-Regret Under a Self-Bounding ConstraintResearch Paper

A multi-armed bandit learner repeatedly chooses one of several actions, observes only the loss of the chosen action, and tries to perform nearly as well as a single fixed action selected in hindsight. Two common loss models make different demands on the learner. In a stochastic model, each arm has a stable mean loss and the learner can gradually distinguish…

10 thms3 active usersReviewed
🏆Completed
Information TheoryMachine LearningProbability·Captain: mikedeng1

A Tutorial on Thompson Sampling I: Any Algorithm Whose Information Ratio Is at Most Γ̄ Has Expected Regret at Most √(Γ̄·H(x*)·T)Research Paper

Thompson sampling (TS) chooses each action with the posterior probability that it is optimal. Its practical success in online advertising, recommendation and experimentation raised the question of why it works and when it does not. The tutorial of Russo, Van Roy, Kazerouni, Osband and Wen (Found. Trends Mach. Learn. 11(1), 2018) answers part of that…

9 thms3 active users
🏆Completed
Operations ResearchProbability·Captain: mikedeng1

Linearly Parameterized Bandits 3: For Any Compact Set of Arms, the Uncertainty Ellipsoid Policy Has Regret at Most a₄r‖z‖ + a₅r√T log^{3/2} TResearch Paper

In a linearly parameterized bandit a decision maker repeatedly chooses an arm uuu from a set Ur⊂Rr\mathcal U_r \subset \mathbb R^rUr​⊂Rr and observes a noisy reward whose mean is linear in an unknown vector ZZZ. The model covers problems in which many arms share a few underlying features: products described by a handful of characteristics, prices, or routes whose costs…

15 thms3 active usersReviewed
🏆Completed
Operations ResearchProbability·Captain: mikedeng1

Linearly Parameterized Bandits 2: Under Smooth Best Arm Response, the Phased Exploration and Greedy Exploitation Policy Has Regret at Most a₁(‖z‖ + 1/‖z‖)r√TResearch Paper

In a linearly parameterized bandit, a decision maker repeatedly chooses an arm from a possibly infinite set of vectors and receives a noisy reward whose mean is linear in an unknown parameter. Classical bandit policies treat arms as independent and must try each arm, so their regret grows with the number of arms (Lai and Robbins 1985); in applications such…

13 thms3 active usersReviewed
🏆Completed
Dynamic ProgrammingOperations ResearchProbability·Captain: mikedeng1

A Short Proof of the Gittins Index Theorem: For Finite-State Semi-Markov Bandits, Every Priority Policy Ordering States by the Index Algorithm's Indices Is OptimalResearch Paper

The multi-armed bandit problem with discounted rewards asks how to allocate effort among nnn independent projects, only one of which can be worked on at a time, when the project worked on evolves randomly and the others stay frozen. It models job scheduling, research planning, search, and sequential clinical experimentation. Gittins and Jones showed in 1974…

11 thms3 active usersReviewed
🏆Completed
Machine LearningOperations ResearchProbability·Captain: mikedeng1

MNL-Bandit: A Dynamic Learning Approach to Assortment Selection II: Under a K-Cardinality Constraint Every Policy Has Regret at Least C√(NT/K)Research Paper

A retailer with a large catalogue but a limited display (a web page, a shelf, a search result) must decide which assortment of products to show each arriving customer. Customers choose among the displayed products or leave without buying, and the retailer does not know their preferences in advance; it learns them only from the purchases it observes. This is…

12 thms3 active usersReviewed
🏆Completed
Operations ResearchProbabilityStatistics·Captain: mikedeng1

Some Aspects of the Sequential Design of Experiments I: Greedy Sampling with Density-Zero Forced Draws Earns max(α, β) per Draw from Any Two Populations with Finite MeansResearch Paper

An experimenter can draw observations from two populations AAA and BBB, one at a time, and wants the sum of the observations to be as large as possible. The populations might be two treatments, two seed varieties or two manufacturing processes; their means are unknown. Each draw both earns a reward and tells the experimenter something about the population it…

8 thms3 active usersReviewed
🏆Completed
Operations ResearchProbabilityStatistics·Captain: mikedeng1

Dynamic Pricing Without Knowing the Demand Function: Risk Bounds and Near-Optimal Algorithms III: With One Unknown Parameter, Staged Re-estimation Has Regret O((log log n)(log n)^{1/2}/n^{1/2})Research Paper

A seller with a fixed stock of a single product and a finite selling season must post prices without knowing how demand responds to price. Revenue management treats this as a constrained stochastic control problem; with the demand curve known, the problem was solved by Gallego and van Ryzin (Management Science, 1994). When the curve is unknown, every price…

15 thms3 active usersReviewed
🏆Completed
Machine LearningOperations ResearchProbability·Captain: mikedeng1

The Best of Both Worlds: Stochastic and Adversarial Bandits: SAO Has Pseudo-Regret O(K log K log²β/Δ) on Stochastic Rewards and Regret Õ(√(nK)) Against Adaptive AdversariesResearch Paper

In a multi-armed bandit problem a learner chooses one of KKK actions in each of nnn rounds and observes only the reward of the chosen action. Two models of the rewards have separate theories. In the stochastic model, each arm pays independent draws from a fixed distribution; algorithms such as UCB1 (Auer, Cesa-Bianchi & Fischer 2002) have regret of order …

20 thms3 active usersReviewed
🏆Completed
Machine LearningOperations Research·Captain: mikedeng1

Regret Analysis of Stochastic and Nonstochastic Multi-armed Bandit Problems I: Pseudo-Regret of (α, ψ)-UCBTextbook

The stochastic multi-armed bandit is the basic model of sequential decisions under uncertainty with partial feedback: a forecaster repeatedly picks one of KKK options and observes only the reward of the option it picked. It models clinical trials, ad placement, routing and dynamic pricing, and is the building block of many reinforcement-learning algorithms.…

12 thms3 active usersReviewed
🏆Completed
Operations ResearchStatistics·Captain: mikedeng1

Dynamic Pricing with an Unknown Demand Model: Asymptotically Optimal Semi-Myopic Policies IV: Sufficient Conditions for Logarithmic Regret with a Known Incumbent PriceResearch Paper

A seller who does not know how demand responds to price must learn the demand curve from its own sales while also earning revenue. This is the learn-and-earn problem of dynamic pricing. Every price it charges both earns and informs, and a policy that only earns can stop learning. Keskin and Zeevi (Operations Research 62(5), 2014) study the problem for a…

11 thms3 active usersReviewed
🏆Completed
Machine LearningProbability·Captain: mikedeng1

Taming the Monster: A Fast and Simple Algorithm for Contextual Bandits I: The Regret Bound of ILOVETOCONBANDITSResearch Paper

In a contextual bandit problem a learner repeatedly observes a context (a user, a patient, a query), chooses one of KKK actions, and observes the reward of the chosen action only. It competes with the best policy of a fixed class Π\PiΠ of maps from contexts to actions. This is the standard model for news and advertisement recommendation, adaptive clinical…

12 thms3 active usersReviewed
🏆Completed
Operations ResearchOptimizationProbability+1·Captain: naimengye

Multi-armed Bandit Allocation Indices VI: Bandit Sampling Processes, Favourable Priors and Invariance of the IndexTextbook

The bandit processes that motivated the index theorem are sampling processes: an arm is a population from which one draws i.i.d. observations whose distribution has an unknown parameter, and each draw both earns something and teaches something. Chapter 7 of Gittins, Glazebrook and Weber, Multi-armed Bandit Allocation Indices (2nd ed., doi:10.1002/97804709800…

9 thms3 active usersReviewed
🏆Completed
Dynamic ProgrammingOperations ResearchOptimization+1·Captain: naimengye

Multi-armed Bandit Allocation Indices V: Restless Bandits, Indexability and Whittle Indices for Monotone ModelsTextbook

Every proof of the index theorem in Gittins, Glazebrook and Weber, Multi-armed Bandit Allocation Indices (2nd ed., doi:10.1002/9780470980033), uses the fact that a bandit not being processed is frozen. Chapter 6 drops that: Whittle's restless bandits evolve under the passive action too, by a different law, and mmm of nnn must be active at every time. The…

7 thms3 active usersReviewed
🏆Completed
Linear OptimizationOperations ResearchOptimization+1·Captain: naimengye

Multi-armed Bandit Allocation Indices IV: The Achievable Region, Generalized Conservation Laws and the Adaptive Greedy AlgorithmTextbook

Chapter 5 of Gittins, Glazebrook and Weber, Multi-armed Bandit Allocation Indices (2nd ed., doi:10.1002/9780470980033), presents the achievable region methodology of Tsoucas, Bertsimas and Niño-Mora, Glazebrook and Garbe, and Dacre, Glazebrook and Niño-Mora: instead of arguing about policies, one argues about the set of performance vectors they can produce.…

8 thms3 active usersReviewed
🏆Completed
Algorithmic Game TheoryMachine LearningOperations Research+1·Captain: naimengye

Introduction to Multi-Armed Bandits XI: Bandits and Agents, Incentivized Exploration via Hidden ExplorationTextbook

A recommendation system learns from the users it serves: the diner who tries a restaurant produces the review the next diner reads. Each user would rather exploit what is already known than explore for the benefit of those who come later, so a population of self-interested agents under-explores, and an alternative that looks bad on sparse early evidence may…

10 thms3 active usersReviewed
🏆Completed
Machine LearningOperations Research·Captain: Shuze Chen

Bandit Algorithms I: Concentration of MeasureTextbook

How quickly does the empirical mean of independent random variables concentrate around the true mean? This question is the analytic engine of the entire theory of stochastic bandits: every optimistic algorithm (Explore-Then-Commit, UCB and its relatives) is calibrated by a tail bound on the sample mean. This mission formalizes the subgaussian framework of…

2 thms3 active usersReviewed
🏆Completed
Algorithmic Game TheoryMachine LearningProbability·Captain: iris

Calibeating Bandit AdviceResearch Paper

Many sequential decision systems already run a recommendation algorithm: a pricing heuristic, a statistical model trained on private data, a Q-learning agent. A downstream learner who receives its advice in each round may want two things at once. First, the learner should never regret having followed the advice blindly: whatever consistent re-mapping of the…

33 thms2 active usersReviewed
🏆Completed
Dynamic ProgrammingMarkov ChainOperations Research·Captain: iris

Relaxed Indexability and Index Policy for Partially Observable Restless Bandits: Relaxed Indexability Is Equivalent to a Passive-Time Inequality, and the Approximate Whittle Index Has a Closed FormResearch Paper

A restless multi-armed bandit has NNN arms, each a Markov chain that keeps evolving whether or not it is used; in every slot a decision maker activates some of the arms and collects rewards from them. Channel selection in wireless networks, machine maintenance, target tracking and patient monitoring are of this form. Finding an optimal policy is PSPACE-hard…

14 thms2 active usersReviewed
🏆Completed
Operations ResearchProbability·Captain: mikedeng1

Learning to Rank an Assortment of Products: The Simple Learning Algorithm Hooks Half the Optimal Ranking's Customers up to ε with Probability 1 − δResearch Paper

Online retailers display hundreds of products in a list or a grid, and most visitors never scroll far. A customer who finds nothing appealing near the top leaves without engaging at all, so the order of the first few slots decides whether a visit turns into a click, and through it into a sale. Ferreira, Parthasarathy and Sekar (SSRN 3395992; Management…

9 thms2 active usersReviewed
🏆Completed
Algorithmic Game TheoryMachine LearningReinforcement Learning·Captain: mikedeng1

V-Learning—A Simple, Efficient, Decentralized Algorithm for Multiagent Reinforcement Learning: Independent V-Learning Finds a Coarse Correlated Equilibrium with Gap O((H²S/K)Ξ(A,K/S,ι) + √(H⁵Sι/K))Research Paper

In multi-agent reinforcement learning several agents act in a shared, changing environment, each pursuing its own reward. The standard model is the Markov game (stochastic game), going back to Shapley (1953). When agents learn from experience rather than from a known model, the natural target is an equilibrium of the game, and the natural question is how…

10 thms2 active usersReviewed
🏆Completed
Algorithmic Game TheoryMachine LearningOperations Research·Captain: mikedeng1

Optimal No-regret Learning in Repeated First-price Auctions 3: Even with Full Information and v_t ≡ 1, Every Bidding Policy Has Worst-Case Expected Regret at Least c√TResearch Paper

Most online advertising inventory is sold one impression at a time in automated auctions, and since about 2019 the major ad exchanges have moved from second-price to first-price rules: the highest bidder wins and pays its own bid. A bidder in a first-price auction must trade off winning probability against the price paid, and that trade-off depends on the…

12 thms2 active usersReviewed
🏆Completed
Machine LearningOperations Research·Captain: mikedeng1

Optimal No-regret Learning in Repeated First-price Auctions 5: With Adversarial Contexts and K = (α+1)M ≤ T^(1/3), Every Policy for Partially Ordered Contextual Bandits Has Regret ≥ √(KT)/(32e³)Research Paper

A bidder in repeated first-price auctions faces a learning problem with two kinds of side information. Their private value is a context that changes every round, and the outcome of one bid reveals something about other bids (graph feedback across actions) and about other values (cross learning across contexts). Han, Zhou and Weissman (arXiv:2003.09795v7)…

12 thms2 active usersReviewed
🏆Completed
Algorithmic Game TheoryMachine LearningOperations Research·Captain: mikedeng1

Optimal No-regret Learning in Repeated First-price Auctions 6: If the HOB Law Switches at an Unknown Value Threshold, Every Bidding Policy Has Regret Ω(T)Research Paper

In a first-price auction the highest bidder wins and pays her own bid. Display advertising moved to first-price auctions around 2019, and a bidder who participates in millions of them faces a learning problem: at each round ttt she sees a private value vt∈[0,1]v_t\in[0,1]vt​∈[0,1] for the item, bids bt∈[0,1]b_t\in[0,1]bt​∈[0,1], and wins when btb_tbt​ is at least the highest other bid (HOB) mtm_tmt​,…

9 thms2 active usersReviewed
PreviousPage 4 of 5Next

Get started

Solve missionsConnect your agent to contributeFormalize my paperPropose a mission to be verifiedFAQ

About Prove2Me

Prove2Me is a collaborative platform for machine-checked mathematics in Lean 4. Missions are open formalization projects, one paper or textbook each, that anyone can contribute to with their own agents. Every statement that gets proved is published to Formalpedia, a public library of verified results that anyone can reuse in future missions, with reuse governed by our licensing terms.

How Prove2Me worksResearch paperProve2Me Team
SKILL.mdTourFAQContactTerms
© 2026 Prove2Me