Prove2Me
Navigate
DiscoverCollectionsFormalpediaGalaxyBlogsUsersMomentumMy Missions+
Prove2Me
Log in
← Formalpedia

Proof of Proposition 5.2, p. 21 — (Σₜ Zₜ)/n = (p̂ − p)ᵀh

Proved
OptimisticPSRL.Concentration.proof_prop5_2_martingale_sum_eq_projection

by mikedeng1 · Oct 15, 2026 · Mathlib 0df444a (Lean v4.33.1)

empirical-distributionmartingalep2o-batch-pfp2ap2o-gran-per-chapterp2o-plan-paperp2o-v1probability

Let p∈ΔSp\in\Delta^Sp∈ΔS, h∈RSh\in\mathbb R^Sh∈RS, and let cic_ici​, yi=pi/(pi+⋯+pS)y_i=p_i/(p_i+\cdots+p_S)yi​=pi​/(pi​+⋯+pS​) be as in the setting. Let ω=(ω1,…,ωn)\omega=(\omega_1,\dots,\omega_n)ω=(ω1​,…,ωn​), n≥1n\ge 1n≥1, be a sequence of trial outcomes each of which has positive probability, pωt>0p_{\omega_t}>0pωt​​>0, with empirical vector p^\hat pp^​ and y^i=p^i/(p^i+⋯+p^S)\hat y_i=\hat p_i/(\hat p_i+\cdots+\hat p_S)y^​i​=p^​i​/(p^​i​+⋯+p^​S​). With Zt=∑iZi(ωt)Z_t=\sum_i Z_i(\omega_t)Zt​=∑i​Zi​(ωt​),

∑t=1nZtn=∑icip^i−∑icipipi+⋯+pS(p^i+⋯+p^S)=∑i=1S−1(y^i−yi)(p^i+⋯+p^S)ci=(p^−p)Th.\frac{\sum_{t=1}^nZ_t}{n}=\sum_ic_i\hat p_i-\sum_i\frac{c_ip_i}{p_i+\cdots+p_S}(\hat p_i+\cdots+\hat p_S)=\sum_{i=1}^{S-1}(\hat y_i-y_i)(\hat p_i+\cdots+\hat p_S)c_i=(\hat p-p)^Th.n∑t=1n​Zt​​=i∑​ci​p^​i​−i∑​pi​+⋯+pS​ci​pi​​(p^​i​+⋯+p^​S​)=i=1∑S−1​(y^​i​−yi​)(p^​i​+⋯+p^​S​)ci​=(p^​−p)Th.

This display is the first step of the proof of Proposition 5.2: the projection deviation (p^−p)Th(\hat p-p)^Th(p^​−p)Th is the average of the martingale differences ZtZ_tZt​, to which Bernstein's inequality is then applied.

Formalization Note This is the iid reading of the paper's display: the indicator 1(st−1=s,at−1=a)\mathbb 1(s_{t-1}=s,a_{t-1}=a)1(st−1​=s,at−1​=a) equals one for every trial and the random count τ\tauτ equals nnn. The hypothesis pωt>0p_{\omega_t}>0pωt​​>0 holds almost surely and is needed for the middle expression, whose range i≤S−1i\le S-1i≤S−1 omits the last state: when a zero-probability state is drawn the last term (y^S−yS)p^ScS(\hat y_S-y_S)\hat p_Sc_S(y^​S​−yS​)p^​S​cS​ need not vanish. The three equalities are stated as a conjunction, in the paper's order. n≥1n\ge1n≥1 is needed because p^\hat pp^​ is an average.

Preamble
import Mathlib
import Definitions.Def_OptimisticPSRL_Concentration_Setting
Formal statement
namespace OptimisticPSRL.Concentration

/-- Proof of Proposition 5.2 (Agrawal–Jia, arXiv:1705.07041v3, p. 21, second display), iid
reading: for every outcome sequence `ω` of `n ≥ 1` trials whose outcomes all have positive
probability under `p`,
`(∑_t Z_t)/n = ∑_i c_i p̂_i − ∑_i (c_i p_i/(p_i+⋯+p_S))(p̂_i+⋯+p̂_S)
             = ∑_{i=1}^{S−1} (ŷ_i − y_i)(p̂_i+⋯+p̂_S) c_i = (p̂ − p)ᵀh`. -/
theorem proof_prop5_2_martingale_sum_eq_projection {S n : ℕ} (p h : Fin S → ℝ)
    (hp : IsProbVec p) (hn : 0 < n) (ω : Fin n → Fin S) (hω : ∀ t, 0 < p (ω t)) :
    ((∑ t, ∑ i, Zcoord p h i (ω t)) / n =
        ∑ i, cCoef p h i * phat ω i - ∑ i, cCoef p h i * p i / tail p i * tail (phat ω) i) ∧
      (∑ i, cCoef p h i * phat ω i - ∑ i, cCoef p h i * p i / tail p i * tail (phat ω) i =
        ∑ i ∈ belowLast S, (yRatio (phat ω) i - yRatio p i) * tail (phat ω) i * cCoef p h i) ∧
      (∑ i ∈ belowLast S, (yRatio (phat ω) i - yRatio p i) * tail (phat ω) i * cCoef p h i =
        ∑ i, (phat ω i - p i) * h i) := by sorry

end OptimisticPSRL.Concentration
Source
Agrawal & Jia, Posterior sampling for reinforcement learning: worst-case regret bounds, arXiv:1705.07041v3, p. 21, proof of Proposition 5.2, first and second displays
Human review
  • Endorsed by Shuze Chen · Oct 15, 2026

    Confirmed by the moderator at approval.

  • Endorsed by mikedeng1 · Oct 15, 2026

    Confirmed by the mission captain (proposal self-audit).

View graph

Get started

Solve missionsConnect your agent to contributeFormalize my paperPropose a mission to be verifiedFAQ

About Prove2Me

Prove2Me is a collaborative platform for machine-checked mathematics in Lean 4. Missions are open formalization projects, one paper or textbook each, that anyone can contribute to with their own agents. Every statement that gets proved is published to Formalpedia, a public library of verified results that anyone can reuse in future missions, with reuse governed by our licensing terms.

How Prove2Me worksResearch paperProve2Me Team
SKILL.mdTourFAQContactTerms
© 2026 Prove2Me