JAO Theorem 5, main regime (so )
OpenBanditAlgorithm.mdp_regret_lower_bound_jao_large_diameterFor any algorithm and any , and , there is an MDP with states, actions and diameter at most on which, from every initial state, the expected regret after steps is at least .
This is the main regime , equivalently , which is the standing assumption of JAO's construction (p. 1582). The hard instance is built from a two-state gadget: states (reward ) and (reward ) with actions, where for every action except a single good action for which it is , and throughout; the gadget has diameter . One takes copies, exactly one of which carries a good action, and joins them with additional zero-reward actions per state inducing an -ary tree on the -states, giving a composite diameter of at most .
The analysis then identifies all -states, collapsing the composite MDP to a single two-state MDP with actions; learning is easier and its optimal average reward is the same, so a lower bound for transfers. On the argument is the multi-armed bandit lower bound of Auer et al. (2002b): the optimal average reward is , and with the averaged regret over a uniformly random planted pair exceeds .
This is the original statement, with the explicit constant and the initial state universally quantified, as opposed to the Lattimore-Szepesvari restatement (Bandit Algorithms, Theorem 38.7) which asserts the bound for , with an unspecified universal constant. The hypothesis is not cosmetic: JAO's construction uses a two-state gadget with actions together with further actions per state to connect the copies, so at the construction does not exist, and the final constant is obtained from .
⚠ Not provable from the source as stated — see below. The constant on does not follow from JAO Section 6. The proof ends at with , , ; since the composite MDP must fit in states and actions, and , hence and the proven bound is at most . The exact supremum over is , attained at . The rest of Section 6 is correct: its bracket at evaluates to as claimed. The error is a single unstated substitution in the final line.
This node is kept only as a record of the statement as published. Work instead on BanditAlgorithm.mdp_regret_lower_bound_jao_universal_constant, which carries the same hypotheses and the same scaling with a universal but unspecified constant. See mission comment afe04135 for the full computation.
import Definitions.Def_FiniteMDPLearning import Mathlib.Data.Real.Sqrt import Mathlib.Analysis.SpecialFunctions.Log.Basic open MeasureTheory ProbabilityTheory
theorem BanditAlgorithm.mdp_regret_lower_bound_jao_large_diameter :
∀ S A T : ℕ, ∀ D : ℝ, 10 ≤ S → 10 ≤ A →
20 * (Real.log S / Real.log A) ≤ D → D * S * A ≤ (T : ℝ) → 12 ≤ D →
∀ π : MDPPolicy S A,
∃ M : FiniteMDP S A,
mdpDiameterENN M ≤ ENNReal.ofReal D ∧
∀ s : Fin S,
0.015 * Real.sqrt (D * S * A * T) ≤
∫ h, mdpRegret M T h ∂(mdpMeasure M (mdpStateDirac s) π T) := by
sorry