Proposition 4 — TPG under general Bernoulli randomization is unbiased for the truncated policy gradient
ProvedTPG.proposition_4_general_bernoulliSetting. is a finite state space and the action is control or treatment. Time is , which is the source's time . Under action at time the state moves according to a transition matrix , i.e. ; the matrices may depend on arbitrarily. The initial state has distribution , and the reward has distribution on , with for all and conditional independence from everything else given . Under the policy every action is treatment with probability , independently of everything else; , and refer to the law of the trajectory under .
Fix an experimental allocation and a truncation size . With the score , the TPG estimator and the truncated -values under are
Write , , with , and . The hypothetical truncated policy value is, with ,
and is the value function.
Proposition. For every , and :
- ;
- is differentiable at with derivative equal to the same quantity, ;
- for , for all ;
- is differentiable at and .
This extends the policy-gradient interpretation of the main text (Propositions 1 and 3) from to every interior experimental allocation.
Formalization Note is defined by the matrix expression of the source, with a truncated subtraction; item 3 is stated on , where is a policy.
import Mathlib import Definitions.Def_TPG_Model import Definitions.Def_TPG_Estimators open MeasureTheory ProbabilityTheory open scoped BigOperators ENNReal
namespace TPG
theorem proposition_4_general_bernoulli {S : Type} [Fintype S] [DecidableEq S] [MeasurableSpace S] [MeasurableSingletonClass S]
(E : Env S) (T k : ℕ) (θ M : ℝ)
(hT : 0 < T) (hk : k ≤ T) (hθ0 : 0 < θ) (hθ1 : θ < 1) (hM : E.RewardBound M) :
(∫ ω, tpgEst θ T k ω ∂(E.expLaw T θ)
= (1 / (T : ℝ)) * ∑ u : Fin T, (E.Qk θ T k u true - E.Qk θ T k u false))
∧ HasDerivAt (fun q : ℝ => E.JkGen T k θ q)
((1 / (T : ℝ)) * ∑ u : Fin T, (E.Qk θ T k u true - E.Qk θ T k u false)) θ
∧ (∀ q : ℝ, 0 ≤ q → q ≤ 1 → E.JkGen T T θ q = E.J T q)
∧ HasDerivAt (fun q : ℝ => E.J T q) (∫ ω, tpgEst θ T T ω ∂(E.expLaw T θ)) θ := by sorry
end TPGRead-back
What the Lean code literally says, in plain math · claude-opus-5-5
What is quantified. The statement holds for every choice of the following data.
-
A finite set , carrying an arbitrary -algebra in which every singleton is measurable (as is finite, every subset of is then measurable). Equality of elements of is assumed decidable, a technical assumption with no mathematical content (it is used to form identity matrices).
-
An "environment" on , meaning the following package:
- for every natural number (all of them, not only those below the horizon) and every Boolean value , a real matrix whose entries are all and whose rows each sum to ;
- a function with for all and ;
- for every and every Boolean , a Borel probability measure on .
Because sums to , such an can only exist when is nonempty.
-
Natural numbers and , with the hypotheses and . The value is allowed.
-
A real number with the hypotheses and (both strict).
-
A real number (no sign condition), with the hypothesis that for every and every Boolean . The number occurs nowhere else in the statement. If the interval is empty and this hypothesis cannot be satisfied.
The conclusion is the conjunction of four claims, (a), (b), (c), (d) below. The objects they involve are defined first.
The sample space. Let be the set of all sequences with each , with the product -algebra (the given one on , the discrete one on the Booleans, the Borel one on ). Write , and for the three components of . Times run over .
The measures . Let be any real number (not only one in ). For and a Boolean sequence define the real number
(The middle product is empty, equal to , when . The matrix governs the passage from index to index .) Then
the sum running over all pairs , where denotes a point mass and the big tensor product is the product measure on . Under the -th summand, and for all almost surely, and are independent with distributed according to .
Negative weights are replaced by . When no weight is negative, is a probability measure, and it is the joint law of the following process: has distribution ; each is with probability , independently of ; given the past, has distribution ; and, given all the and , the are independent with distributed according to . When lies outside some weights are negative and are set to , and is then a finite measure that is in general not of total mass .
The function . For every real ,
a function defined for all real . Integrals are Lebesgue (Bochner) integrals, which by library convention are if the integrand is not integrable; under the hypothesis on each satisfies almost everywhere for each measure involved.
The random variable . For a natural number (used below with and with ) define on
The inner sum is over the times in the window from to inclusive, cut off at the last time . For the window is every with .
The numbers and . For a measure on and an event , let denote the measure (by library convention this is the zero measure if ; here it is used only with and , events of -measure and , both positive). For each time and each Boolean define
the sum over the same kind of window (with ) of the conditional expectation of under given the single event . The conditioning is on the value of only, not on or anything else. Put
The matrix expression . This is built from matrices only, with no measure on and no replacement of negative numbers by . For a Boolean and let be the mean of . For a real number put
and for natural numbers let be the ordered matrix product of the matrices with indices , understood to be the identity matrix when . For a natural number (used below with and ), the fixed , and any real , define
where is read as a row vector, as a column vector, and each summand is the resulting scalar. In words: in the -th summand the first matrices (indices ) are formed with parameter , the remaining matrices (indices , at most of them) are formed with parameter , and the final vector is formed with parameter . The subtraction is truncated at .
Conclusion. All four of the following hold.
(a) An equality of real numbers:
(b) The function from to , with the first argument held fixed at , is differentiable at the point (ordinary two-sided derivative), and its derivative there equals :
(c) For every real number with and ,
Here the truncation index is , not . With one has for every , so is the identity matrix and the left-hand side is , in which no longer appears. Nothing is asserted for outside .
(d) The function is differentiable at the point (ordinary two-sided derivative), and its derivative there is
Here again the window index is , not : uses, for each , all times with .
In (b) and (d), "differentiable at with derivative " means as in . The number occurs only in claims (a) and (b); claims (c) and (d) do not involve . The same real number serves as the parameter of the measure , as the parameter in , as the first argument of , and as the point at which the derivatives are taken.
Confirmed by the mission captain (proposal self-audit).