Stochastic Linear Optimization under Bandit Feedback 2: A Regret Lower Bound on the CircleResearch Paper
Motivation
In stochastic linear optimization under bandit feedback a learner repeatedly chooses a point from a compact decision set and observes only the random cost of that point, whose mean is for an unknown vector . The problem models online routing, ad placement and other sequential decisions with linearly structured costs. The quality of a learner is measured by its regret against the best fixed decision.
For the -armed bandit the achievable regret for a fixed instance is logarithmic in the horizon (Lai and Robbins 1985; Auer, Cesa-Bianchi and Fischer 2002). Dani, Hayes and Kakade (COLT 2008) showed that for linear costs the picture depends on the geometry of . Their Theorem 1 gives polylogarithmic regret when the decision set has a positive gap between the best and second-best extreme point (a polytope, for instance), and their Theorem 2 gives regret for every decision set. Their Theorem 3 shows that the second rate cannot be improved in general: on a decision set with zero gap, every algorithm pays in expectation.
Timeline:
- 2002: Auer, Using confidence bounds for exploitation–exploration trade-offs (JMLR 3), introduces confidence-bound algorithms for linear bandits on finite decision sets.
- 2008: Dani, Hayes and Kakade prove the upper bound for ConfidenceBall₂ and the lower bound on a product of circles, the subject of this mission. A hypercube lower bound for the adversarial setting appears in their NIPS 2007 paper.
- 2010: Rusmevichientong and Tsitsiklis, Linearly parameterized bandits (Math. OR 35), give lower bounds on the unit sphere.
- 2020: Lattimore and Szepesvári, Bandit Algorithms, Theorems 24.1 and 24.2, give minimax lower bounds on the hypercube and the unit ball with Gaussian noise.
Setting
The decision set is the unit circle . An unknown mean vector is drawn once, uniformly from the circle of radius ; concretely with uniform on .
On each round the algorithm plays and observes a cost with , so that . Given the decision, the cost is independent of the past.
An algorithm may be randomised. It draws a seed once from a probability measure on a measurable space , and chooses as a function of and the costs observed so far, measurably in .
The regret over rounds is
so each round costs when . The expected regret averages over the seed, the prior and the costs.
In the Lean development these objects are unitCircle, meanVec, optCost, RandomizedPolicy and expectedRegret in the namespace StochLinOpt.LowerBound.
Formalization targets
Goal: Theorem 3 for
There is a universal constant such that for every randomised algorithm and every ,
The constant is left existential, which is the form that survives any later improvement of the constant; it is chosen before the algorithm and before .
Milestones
- Section 6.1, Eq. (3). For , , a posterior probability of and a cost , the Bayes-updated bias satisfies , where .
- Lemma 15. With and the same data,
- Theorem 4 (Freedman). For a martingale difference sequence bounded above by , with conditional variance sum , and all ,
Significance
The lower bound shows that the dependence of the problem-independent upper bound (Theorem 2 of the same paper) is necessary. It also shows that the gap-dependent polylogarithmic rate of Theorem 1 cannot extend to decision sets without a gap, such as the sphere. Together with the upper bound it characterises the minimax regret of stochastic linear bandits in up to logarithmic factors, and in the paper's general- form it also underlies the claim that the price of bandit information is .
The result is proved in the paper for and has not been machine-checked. The mission produces a checked Bayesian lower bound over all randomised algorithms, with an explicit probability model for the protocol. Two related platform results are different theorems: BanditAlgorithm.linear_bandit_unit_ball_minimax_lower_bound (Lattimore–Szepesvári Theorem 24.2: unit ball, Gaussian noise, a worst-case ) and BanditAlgorithm.linear_bandit_hypercube_minimax_lower_bound (Theorem 24.1: hypercube). The costs, the circle and the uniform prior used here are not covered by either.
Difficulty
The obvious attempt is a two-point change-of-measure argument with a fixed pair of means at distance . It fails as stated because the decision set has no gap: an algorithm that plays close to the optimum of both candidates learns slowly but also pays little. The per-round trade-off between regret and information (Lemma 15) is exact only while the posterior is undecided, . Turning it into a bound on the whole horizon requires controlling how long the posterior stays undecided, which is a statement about a martingale whose step sizes are chosen by the algorithm; a concentration bound that ignores the accumulated conditional variance (Azuma–Hoeffding with worst-case steps) is too weak for this. The averaging step from a two-point prior to the uniform prior on the circle is also part of the formal work.
Formalization scope
Vectors are Fin 2 → ℝ with the dot product ⬝ᵥ; Euclidean norms are written through dot products, never with Lean's sup norm. Rounds are 0-indexed internally: the Lean index is the paper's round . The expected regret is the exact finite expectation
so no infinite product of measures is needed. A randomised algorithm is a seeded policy, which covers every randomised algorithm. The optimal cost is the infimum of over the compact circle and is attained. Every junk value in the model (a non-integrable integrand) could only make the lower bound harder to prove, never easier.
A statement over deterministic algorithms only, over a worst-case instead of the uniform prior, or with the constant allowed to depend on the algorithm or on would be a weaker theorem. The goal quantifies before the algorithm and , and fixes the prior.
Corrections relative to the printed paper:
- General is not stated. Theorem 3 as printed claims for every even . It is false for : on with each round has regret at most , so at the claim would need . The general case rests on Lemma 16, which has no proof. The goal is the case, which Section 6.1 proves.
- The constant. For the paper prints ; its proof gives . The proof's Freedman step prints ; with the denominator is , and the bound then needs . Small is covered by the first round, whose expected regret is . The goal leaves existential.
- Theorem 4. The printed variance sum runs to ; it runs to . The conditioning is on a general filtration, and square-integrability of the steps is assumed so that the conditional variance is defined.
- Lemma 15. Its right side depends on the round- cost , which is not part of ; the Lean statement holds for either value of .
Welcome contributions: a Lean proof of Freedman's inequality (reusable across the bandit and concentration missions on the platform); the averaging argument from two-point priors to the uniform prior; and the stopped-martingale bookkeeping for the bias sequence.
Selected references
- Varsha Dani, Thomas P. Hayes, Sham M. Kakade, Stochastic Linear Optimization under Bandit Feedback, Proceedings of the 21st Annual Conference on Learning Theory (COLT), 2008.
- David A. Freedman, On tail probabilities for martingales, The Annals of Probability 3(1):100–118, 1975. https://doi.org/10.1214/aop/1176996452
- Colin McDiarmid, Concentration, in Probabilistic Methods for Algorithmic Discrete Mathematics, Springer, 1998. https://doi.org/10.1007/978-3-662-12788-9_6
- Peter Auer, Using confidence bounds for exploitation–exploration trade-offs, JMLR 3:397–422, 2002. https://www.jmlr.org/papers/v3/auer02a.html
- Paat Rusmevichientong, John N. Tsitsiklis, Linearly parameterized bandits, Mathematics of Operations Research 35(2):395–411, 2010. https://doi.org/10.1287/moor.1100.0446
- Tor Lattimore, Csaba Szepesvári, Bandit Algorithms, Cambridge University Press, 2020, Chapter 24. https://doi.org/10.1017/9781108571401