Lemma 3 — central limit theorem for the TPG estimator
ProvedTPG.lemma_3_cltSetting. is a finite state space and the action is control or treatment. Time is , which is the source's time . Under action at time the state moves according to a transition matrix , i.e. ; the matrices may depend on arbitrarily. The initial state has distribution , and the reward has distribution on , with for all and conditional independence from everything else given . Under the policy every action is treatment with probability , independently of everything else; , and refer to the law of the trajectory under .
Asymptotic setting. The truncation size is fixed and the horizon ; for each the trajectory is generated under the experimental policy . Assumption 1 holds for all times with one constant . Write and
Assumption 2 (asymptotic variance): as , with .
Lemma. Under Assumptions 1 and 2, as ,
where is the TPG estimator.
This is the asymptotic normality part of Theorem 2; it does not need the mean-stability condition.
Formalization Note Convergence in distribution is stated as convergence of for every bounded continuous , with the expectation taken under the horizon- law for each . The limit is a parameter constrained by Assumption 2.
import Mathlib import Definitions.Def_TPG_Model import Definitions.Def_TPG_Estimators import Definitions.Def_TPG_Inference open MeasureTheory ProbabilityTheory open scoped BigOperators ENNReal
namespace TPG
theorem lemma_3_clt {S : Type} [Fintype S] [MeasurableSpace S] [MeasurableSingletonClass S]
(E : Env S) (k : ℕ) (γ M v : ℝ)
(hγ0 : 0 < γ) (hγ1 : γ < 1) (hmix : ∀ T : ℕ, E.Mixing T γ) (hM : E.RewardBound M)
(hvar : E.AsymptoticVariance k v) :
E.TendstoNormal
(fun (T : ℕ) (ω : Traj S T) =>
Real.sqrt (T : ℝ) *
(tpgEst (1 / 2) T k ω - ∫ ω', tpgEst (1 / 2) T k ω' ∂(E.expLaw T (1 / 2))))
v := by sorry
end TPGRead-back
What the Lean code literally says, in plain math · claude-opus-5-5
Data quantified over. The statement is asserted for every choice of the following.
- A set with finitely many elements, carrying a -algebra in which every one-point set is measurable (as is finite, every subset of is then measurable). Non-emptiness of is not listed as an assumption, but the datum below cannot exist on an empty set.
- An environment on , i.e. a package consisting of:
- for every time and every , a real matrix all of whose entries are and each of whose rows sums to ;
- a function with for every and ;
- for every and , a Borel probability measure on (the package itself imposes no moment or support condition on it).
- A natural number (any value, included; one fixed number, not varying with ).
- Real numbers , , .
Sample space and probability measure, one for each . For each let
the set of sequences , with the product -algebra (the given -algebra on , all subsets of , Borel sets of ). Write , , for the coordinates at time .
For and define the weight
(The transition from time to time uses the matrix indexed by the departure time and by the value . The last product comes from a general recipe "factor if is true, if false", used here with for all . Empty products are : for the weight is , and the factor is present only when .) The measure on is the finite mixture of product measures
The clipping is in the definition; here every weight is already and the weights sum to , so is a probability measure. In words: has law ; each is a fair coin flip independent of everything generated before it and of ; is drawn from row of ; and, given all the 's and 's, the are independent with . For , is a single point and is the unit mass on it. Write for integration against .
Statistics on . Throughout, is read as when (so every statistic below is at ), all integrals are Bochner integrals (value if the integrand is not integrable), and means computed in and then converted to a real number with sent to .
Two sign weights occur. The first is if is true and if is false, evaluated at : so , . The second is : so , .
The statistic in the conclusion is built from
where the inner sum runs over the times with (so it is cut off at the horizon ), and the centring constant is the -expectation of itself.
The variance hypothesis is built from, for ,
where the sum defining is over the indices with .
Hypotheses. All of the following are assumed.
- .
- .
- (Contraction of every transition matrix.) For every and every (equivalently, for every ; the instance is empty), every , and every pair of functions that are each nonnegative with entries summing to :
- (Bounded rewards.) For every and every : . Here is an arbitrary real number; if the interval is empty and this hypothesis cannot hold.
- (Limit variance.) , and as .
Conclusion. For every bounded continuous function ,
where is the Gaussian measure on with mean and variance , defined to be the point mass at when . Under hypothesis 5, .
Literal behaviours to note.
- The limit is along , , and both the sample space and the measure change with : the conclusion is convergence of the laws of under , tested against bounded continuous functions, each law living on its own space.
- The numbers and occur only in hypotheses 1 to 4, not in the conclusion. The number in the conclusion is the same as in hypothesis 5.
- The two sign weights and take the same values, so exchanging the order of the finite double sum gives pointwise on ; consequently, for and when the integrals involved are finite, equals the variance of under . (This identity follows from the definitions; it is not written in the statement.)
- Boundary effects are as written: for the inner sum in has fewer than terms, and for the sum defining has rather than terms.
- The transition matrices with do not enter .
Confirmed by the mission captain (proposal self-audit).