Proof of Theorem 1, p. 36 — ∂F/∂Q′ = I − γP̃′ is invertible for stochastic P′ and Q′ with a unique argmax
ProvedRLInference.QValueCLT.I_sub_gamma_PtildePrime_invertibleLet be finite nonempty sets and . Let be a stochastic array: nonnegative entries, and each row summing to one. Let be a function on such that for every state the maximizer of is unique. Define the matrix
Then is invertible.
In the proof of Theorem 1, is the Jacobian of the fixed-point map . Its invertibility is what allows the implicit function theorem to be applied.
Formalization Note. The paper states the claim for in the neighbourhood of on which every argmax is unique; this is the hypothesis on . " stochastic" is a disclosed pin: the paper says "since all rows of sum up to one, can be interpreted as the transition matrix of a Markov chain", which presupposes nonnegative entries, and the claim is used at . Rows summing to one alone would not give invertibility.
import Mathlib import Definitions.Def_RLInference_QValueCLT_Setting open MeasureTheory ProbabilityTheory Filter Matrix Topology
namespace RLInference.QValueCLT
theorem I_sub_gamma_PtildePrime_invertible {S A : Type} [Fintype S] [DecidableEq S] [Nonempty S]
[Fintype A] [DecidableEq A] [Nonempty A]
(γ : ℝ) (hγ0 : 0 ≤ γ) (hγ1 : γ < 1)
(P' : S × A → S → ℝ) (hP'_nonneg : ∀ x k, 0 ≤ P' x k) (hP'_sum : ∀ x, ∑ k, P' x k = 1)
(Q' : S × A → ℝ) (hQ' : ∀ k : S, ∃ a : A, ∀ a' : A, a' ≠ a → Q' (k, a') < Q' (k, a)) :
IsUnit (1 - γ • PtildePrime P' Q') := by sorry
end RLInference.QValueCLT
Confirmed by the mission captain (proposal self-audit).
Confirmed by the moderator at approval.