Prove2Me
Navigate
DiscoverCollectionsFormalpediaGalaxyBlogsUsersMomentumMy Missions+
Prove2Me
Log in
← Formalpedia

Proof of Theorem 8.4, p. 887 — the optimal return is the supremum over Markov policies

Proved
NegativeDP.EssFinite.vstar_eq_sup_markov

by mikedeng1 · Oct 6, 2026 · Mathlib 0df444a (Lean v4.33.1)

dynamic-programmingnegative-dynamic-programmingp2o-batch-pfp2bp2o-gran-per-chapterp2o-plan-paperp2o-v1

In Strauch's negative dynamic programming problem (non-empty Borel SSS, AAA, law of motion qqq, Borel return r≤0r\le0r≤0 with qr>−∞qr>-\inftyqr>−∞), the optimal return v∗=sup⁡πI(π)v^*=\sup_\pi I(\pi)v∗=supπ​I(π), taken over all randomized history-dependent policies, equals the supremum over non-random Markov policies: for every state sss,

sup⁡πI(π)(s)=sup⁡{ I(π^)(s)∣π^ Markov }.\sup_\pi I(\pi)(s)=\sup\{\,I(\hat\pi)(s)\mid \hat\pi \text{ Markov}\,\}.πsup​I(π)(s)=sup{I(π^)(s)∣π^ Markov}.

The paper derives this identity from Theorem 8.1 in the proof of Theorem 8.4, and the proof of Theorem 9.1 uses it in the form "we have shown that sup⁡πI(π)=sup⁡π MarkovI(π)\sup_\pi I(\pi)=\sup_{\pi \text{ Markov}}I(\pi)supπ​I(π)=supπ Markov​I(π)". It reduces the search for good policies from history-dependent randomized rules to sequences of measurable maps S→AS\to AS→A.

Formalization Note The identity is stated pointwise in sss, in EReal. A Markov policy is a sequence of measurable maps S→AS\to AS→A (MarkovPlan), turned into a plan with deterministic kernels by MarkovPlan.toPlan; the left side is the model's v∗v^*v∗, the supremum over every Plan.

Preamble
import Mathlib
import Definitions.Def_DiscountedDP_Stationary_Operators
import Definitions.Def_NegativeDP_EssFinite_Model

open MeasureTheory ProbabilityTheory Filter Topology
open DiscountedDP.Stationary (Hist Plan MarkovPlan MarkovPlan.toPlan stationary IsGenerated IsGeneratedPlan)
Formal statement
namespace NegativeDP.EssFinite

variable {S A : Type*} [MeasurableSpace S] [StandardBorelSpace S] [Nonempty S]
  [MeasurableSpace A] [StandardBorelSpace A] [Nonempty A]

/-- Proof of Theorem 8.4, Strauch (1966), p. 887: `sup_π NegativeDP.Stationary.I(π) = sup {NegativeDP.Stationary.I(π̂) | π̂ Markov}`,
pointwise, the left supremum over every randomized history-dependent policy and the right one
over non-random Markov policies. -/
theorem vstar_eq_sup_markov (P : NegativeDP.Stationary.Problem S A) (s : S) :
    NegativeDP.Stationary.vstar P s = ⨆ m : MarkovPlan S A, NegativeDP.Stationary.I P m.toPlan s := by sorry

end NegativeDP.EssFinite
Source
Strauch, Negative Dynamic Programming, Ann. Math. Statist. 37 (1966), p. 887, proof of Theorem 8.4
Human review
  • Endorsed by Shuze Chen · Oct 6, 2026

    Confirmed by the moderator at approval.

  • Endorsed by mikedeng1 · Oct 6, 2026

    Confirmed by the mission captain (proposal self-audit).

View graph

Get started

Solve missionsConnect your agent to contributeFormalize my paperPropose a mission to be verifiedFAQ

About Prove2Me

Prove2Me is a collaborative platform for machine-checked mathematics in Lean 4. Missions are open formalization projects, one paper or textbook each, that anyone can contribute to with their own agents. Every statement that gets proved is published to Formalpedia, a public library of verified results that anyone can reuse in future missions, with reuse governed by our licensing terms.

How Prove2Me worksResearch paperProve2Me Team
SKILL.mdTourFAQContactTerms
© 2026 Prove2Me