Proposition 3 — the truncated policy gradient is the average truncated -value difference
ProvedTPG.proposition_3_truncated_policy_gradientSetting. is a finite state space and the action is control or treatment. Time is , which is the source's time . Under action at time the state moves according to a transition matrix , i.e. ; the matrices may depend on arbitrarily. The initial state has distribution , and the reward has distribution on , with for all and conditional independence from everything else given . Under the policy every action is treatment with probability , independently of everything else; , and refer to the law of the trajectory under .
Fix a truncation size . For a reward time , the hypothetical truncated policy follows the experimental policy at the times and switches to at time ; denotes the law of under it. The truncated policy value and the truncated -value are
Proposition. For every and , the function is differentiable at and
The right-hand side is the expectation of the TPG estimator , so the estimator is unbiased for the truncated policy gradient.
Formalization Note In the formal definition the hypothetical policy keeps treatment probability after time as well, which does not affect the law of .
import Mathlib import Definitions.Def_TPG_Model import Definitions.Def_TPG_Estimators open MeasureTheory ProbabilityTheory open scoped BigOperators ENNReal
namespace TPG
theorem proposition_3_truncated_policy_gradient {S : Type} [Fintype S] [MeasurableSpace S] [MeasurableSingletonClass S]
(E : Env S) (T k : ℕ) (M : ℝ) (hT : 0 < T) (hk : k ≤ T) (hM : E.RewardBound M) :
HasDerivAt (fun θ : ℝ => E.Jk T k θ)
((1 / (T : ℝ)) * ∑ t : Fin T, (E.Qk (1 / 2) T k t true - E.Qk (1 / 2) T k t false))
(1 / 2) := by sorry
end TPGRead-back
What the Lean code literally says, in plain math · claude-opus-5-5
What is quantified. The statement holds for every choice of the following data.
-
A finite set , carrying an arbitrary -algebra in which every singleton is measurable (as is finite, every subset of is then measurable).
-
An "environment" on , meaning the following package:
- for every natural number (all of them, not only those below the horizon) and every Boolean value , a real matrix whose entries are all and whose rows each sum to ;
- a function with for all and ;
- for every and every Boolean , a Borel probability measure on .
Because sums to , such an can only exist when is nonempty.
-
Natural numbers and , with the hypotheses and . The value is allowed.
-
A real number (no sign condition), with the hypothesis that for every and every Boolean . The number occurs nowhere else in the statement. If the interval is empty and this hypothesis cannot be satisfied.
The sample space. Let be the set of all sequences with each , with the product -algebra (the given one on , the discrete one on the Booleans, the Borel one on ). Write , and for the three components of . Times run over .
The family of measures . Let be any sequence of real numbers indexed by (the entries need not lie in ). For and a Boolean sequence define the real number
(The middle product is empty, equal to , when . The matrix governs the passage from index to index ; matrices with are never used.) Then
the sum running over all pairs , where denotes a point mass and the big tensor product is the product measure on . Under the -th summand, and for all almost surely, and are independent with distributed according to .
Negative weights are replaced by . When all with lie in , no weight is negative, is a probability measure, and it is the joint law of the following process: has distribution ; each is with probability , independently of ; given the past, has distribution ; and, given all the and , the are independent with distributed according to . When some lies outside , some weights are negative and are set to , and is then a finite measure that is in general not of total mass .
The function being differentiated. For a real number (any real) and a time , let be the sequence
and let for . So under the parameter is the fixed number at the times that precede by more than steps, and is at all other times (the times with , of which there are at most , the time itself, and all times after ). Define, for every real ,
Each summand uses its own measure , which depends on . This is a function defined for all real . Integrals are Lebesgue (Bochner) integrals, which by library convention are if the integrand is not integrable; under the hypothesis on each satisfies almost everywhere for each measure involved.
The claimed value of the derivative. Let for the constant sequence (this is also for every ). For a measure on and an event , let denote the measure (by library convention this is the zero measure if ; here it is used only with and , an event of -measure ). For each time and each Boolean define the real number
the sum over the times in the window from to inclusive (cut off at the last time ) of the conditional expectation of under given the single event . The conditioning is on the value of only, not on or on anything else, so is a number and not a function on .
Conclusion. The function is differentiable at the point (in the ordinary two-sided sense), and its derivative there is
Precisely: as in , where is the right-hand side above. The number appears in three places: it is hard-wired into the measures for the early times, it is the parameter of the measure used for the , and it is the point of differentiation. No other value is addressed.
Boundary cases of that the statement includes. If , then is for and for , and each window consists of the single time , so . If or (both permitted by ), the condition never holds for , so is, for every , the measure with constant sequence , and each window consists of all times with .
Confirmed by the mission captain (proposal self-audit).