Optimal Best Arm Identification with Fixed Confidence IV: Asymptotic Optimality of the Track-and-Stop StrategyResearch Paper
Motivation
In best arm identification with fixed confidence, a learner samples unknown distributions (arms) sequentially and must, as early as possible, name the arm with the largest mean, while being wrong with probability at most a prescribed risk . The problem models adaptive A/B/n testing, clinical and simulation-based selection among alternatives, and the "ranking and selection" problem of operations research and simulation optimization. The quantity of interest is the sample complexity , the expected number of samples a strategy takes before stopping.
Timeline of the question this mission formalizes:
- Chernoff (1959) introduced sequential tests based on generalized likelihood ratios for adaptive design of experiments, with a finite set of hypotheses (doi:10.1214/aoms/1177706205).
- Kaufmann, Cappé and Garivier (2016, JMLR) proved a change-of-measure lower bound on the sample complexity of every -PAC strategy (arXiv:1407.4443).
- Garivier and Kaufmann (COLT 2016) identified the exact constant in that lower bound and gave the first strategy, Track-and-Stop, whose sample complexity matches it asymptotically as (arXiv:1602.04589). This mission covers the upper-bound half of that paper.
Setting
A canonical one-parameter exponential family is a family of laws , , on with density with respect to a reference measure ; is twice differentiable and strictly convex, and has mean . Bernoulli, Poisson and Gaussian laws with known variance are examples. The divergence is the Kullback–Leibler divergence between the members with means and .
A bandit model assigns a member of the family to each arm. The class consists of models with a unique optimal arm . At each round the learner picks an arm as a function of past observations, observes a reward drawn from that arm's law, and at a stopping time recommends an arm. is the number of draws of arm in the first rounds and its empirical mean.
With and the probability simplex, the characteristic time is
and the maximizer are the optimal proportions of arm draws.
Track-and-Stop combines two ingredients:
- a sampling rule that tracks the plug-in proportions while forcing each arm to be drawn about times: C-Tracking tracks the cumulated sum of projections of onto , ; D-Tracking draws an under-sampled arm when some , and otherwise the arm maximizing ;
- Chernoff's stopping rule, which stops at the first at which some arm beats every other arm in a generalized likelihood ratio test, , here with .
Formalization targets
Goal: Theorem 14 (p. 13)
For and , Chernoff's stopping rule with combined with C-Tracking or D-Tracking satisfies
for every .
Milestones
- Lemma 15 (p. 20): greedy tracking of cumulated proportions keeps .
- Lemma 7 (p. 7): C-Tracking ensures and .
- Lemma 8 (p. 7): D-Tracking ensures , and proportions within of after a time that does not depend on the trajectory, once the plug-in targets are within .
- Proposition 9 (p. 8): under either rule, almost surely.
- Lemma 18 (p. 27): an explicit with for .
- Proposition 13 (p. 11): with any sampling rule whose proportions converge almost surely to , almost surely and almost surely.
Significance
Theorem 1 of the same paper shows for every -PAC strategy, and . Theorem 14 with therefore shows that the lower bound is attained: is the exact asymptotic sample complexity of best arm identification in exponential family models, and Track-and-Stop is asymptotically optimal.
The result is proved in the paper. What is not available is a machine-checked proof for exponential families. The platform already holds a Lean development of the Gaussian case following Lattimore and Szepesvári, Bandit Algorithms, Ch. 33, stated for one existentially chosen policy with a different threshold. This mission asks for the universal statement: every run of either tracking rule, for every exponential family, with the paper's thresholds. The tracking lemmas (Lemmas 15, 7, 8) are deterministic combinatorics and reusable by any tracking-based algorithm.
Difficulty
The obvious argument plugs the almost-sure behaviour of Proposition 13 into an expectation. That step fails: almost-sure convergence of does not control , because on the rare events where the empirical means are far from the stopping time may be very large. Theorem 14 needs a quantitative concentration of on events whose complements have summable probability, which in turn relies on the forced exploration guaranteed by the lower bounds on (the concentration step of App. D, Lemmas 19–20).
A second obstacle is the regularity of : the tracking lemmas only transfer convergence of to convergence of through the continuity of on , proved from the characterization of in §2.2 (Proposition 6). The GLR statistic also needs its closed form (7) near , which requires the empirical means to lie in the interior of the mean space.
Formalization scope
- Model. The exponential family is a structure with a nonempty open interval, each normalized, twice continuously differentiable and on . Openness and are added to the paper's "convex, twice differentiable"; strict convexity is what makes unique. Bandit models are parameter vectors with ; arms are indexed . is the set of parameter vectors with a unique arm of largest mean .
- Protocol. Policies, the trajectory law , pull counts, empirical means and are the platform's published definitions (
BanditPolicy,BanditTrajectory,TrackAndStop). uses Kullback–Leibler divergences of the arm laws over the class and takes values in . Trajectory coordinate is round . An arm never drawn has empirical mean . - Target map. is undefined in the paper when (an unsampled arm, ties, a mean outside ). Every tracking statement quantifies over every target map with values in that returns optimal proportions on , over every choice of projections, and over every tie-breaking, including randomized ones.
- Stopping rule. The two maxima in are suprema over in the extended reals. is written without subtracting infinities. The stopping time is the first at which the test succeeds, if none. "" is for ; is added so that is defined.
- Values in . Expectations of , the ratios and live in . No statement converts them to reals, so an infinite expected stopping time is never read as .
- Corrections, disclosed. Proposition 9's printed is . Lemma 18 adds and , without which its expressions are undefined.
- Ruled out. Specializing to Gaussian arms, or asserting that some sampling policy achieves the bound, would restate existing platform results and is not this theorem: the goal is about every C-Tracking or D-Tracking run in every exponential family.
- Welcome contributions. Exponential-family facts ( is the mean, the KL formula, concentration of empirical means); the continuity of (Proposition 6, App. A.3); Lemma 17 (App. B.2), from which Lemma 8 follows; the closed form (7) of the GLR statistic.
Selected references
- A. Garivier, E. Kaufmann, Optimal Best Arm Identification with Fixed Confidence, COLT 2016, JMLR W&CP 49. arXiv:1602.04589v2
- E. Kaufmann, O. Cappé, A. Garivier, On the Complexity of Best-Arm Identification in Multi-Armed Bandit Models, JMLR 17, 2016. arXiv:1407.4443
- H. Chernoff, Sequential Design of Experiments, Ann. Math. Statist. 30(3), 1959. doi:10.1214/aoms/1177706205
- T. Lattimore, C. Szepesvári, Bandit Algorithms, Cambridge University Press, 2020, Ch. 33. doi:10.1017/9781108571401