Step 2 of the MDP lower bound: some family member forces regret
OpenBanditAlgorithm.arena_family_step_twoStep 2 of the MDP lower bound: the change-of-measure core.
Fix a layered arena and a family of arenas that agree with off a single transition row, member planting an advantage at the leaf--action pair . Let be the number of members and fix an arbitrary learner . If the advantage is tuned to
then some member of the family forces into expected regret at least
the subtracted term being the cost of a single transient episode.
The argument is the classical one. No learner can distinguish alternatives that differ in one row without paying for it: the counts it allocates to the planted pairs sum to a fixed budget, so by pigeonhole some pair receives at most a share, and the divergence decomposition then bounds the KL divergence between that alternative and the reference by , which the tuning of keeps below a constant.
The hypotheses are exactly the arithmetic facts the parameter tuning must check. hlo, htail1 and htail2 deliver the lower bound of Claim 38.10 through a Chernoff certificate for the truncated leaf counter; hhi gives its upper bound; hcapD says the truncation level lies below the pathwise cap ; and hc3D calibrates against , which plays the role of the diameter in Claim 38.11.
The counting is done with truncated counts rather than a stopping time: truncCount records the visits to a pair among the first rounds spent at leaves. This yields and pathwise, which is what licenses the bounded change-of-measure inequality without any measurability side conditions.
import Definitions.Def_LayeredArena import Mathlib.Data.Real.Sqrt open MeasureTheory ProbabilityTheory open scoped NNReal open BanditAlgorithm BanditAlgorithm.LayeredArena
theorem BanditAlgorithm.arena_family_step_two
{S A : ℕ} [NeZero S] {δ Δ : ℝ≥0}
(hδ1 : δ ≤ 1) (hδ0 : (0 : ℝ) < δ) (hΔ4 : Δ ≤ 1 / 4) (hΔ2 : Δ ≤ 1 / 2)
(E₀ : LayeredArena S A)
(Efam : ↥(countedPairs E₀.leafNat) → LayeredArena S A)
(hg : ∀ p, E₀.good = (Efam p).good) (hb : ∀ p, E₀.bad = (Efam p).bad)
(hr : ∀ p, E₀.root = (Efam p).root) (hl : ∀ p, E₀.lvl = (Efam p).lvl)
(hdp : ∀ p, E₀.depth = (Efam p).depth) (hc : ∀ p, E₀.child = (Efam p).child)
(hsl : ∀ p, (Efam p).specialLeaf = p.val.1)
(hsa : ∀ p, (Efam p).specialAction = p.val.2)
(hS : 0 < S) (hA : 0 < A)
(π : MDPPolicy S A) (n N k : ℕ) (D c₁ c₂ c₃ : ℝ)
(hcard : Fintype.card ↥(countedPairs (A := A) E₀.leafNat) = k) (hk : 2 ≤ k)
(hn0 : 0 < n) (hN0 : 0 < N) (hD : 0 < D)
(hc₁ : 0 < c₁) (hc₂ : 0 < c₂) (hc₃ : 0 < c₃)
(hcapD : (N : ℝ) ≤ n / D) (hc₃D : c₃ * D ≤ 1 / (δ : ℝ))
(ε : ℝ) (hlo : c₁ * n / D ≤ (1 - ε) * N)
(hhi : ((n : ℝ) + 1 / (δ : ℝ)) / E₀.epiLen (δ : ℝ) ≤ c₂ * n / D)
(htail1 : E₀.depth + N * (E₀.depth + 2) ≤ n)
(htail2 : (2 : ℝ) ^ N ≤ ε * (1 + (δ : ℝ)) ^ N
* (mx (δ : ℝ)) ^ (n - E₀.depth - N * (E₀.depth + 2)))
(hΔtune : (Δ : ℝ) = c₁ * ((k : ℝ) - 1) / 2 * Real.sqrt (D / (2 * c₂ * n * k))) :
∃ p : ↥(countedPairs E₀.leafNat),
c₁ ^ 2 * c₃ / 16 * Real.sqrt (D * k * n / (2 * c₂)) - (1 / 2 + (Δ : ℝ)) / (δ : ℝ)
≤ ∫ h, mdpRegret ((Efam p).toMDP hδ1 hΔ2) n h
∂(mdpMeasure ((Efam p).toMDP hδ1 hΔ2)
(mdpStateDirac (Efam p).root) π n) := by
sorry