Theorem 3.6(a) — cost iteration for a discounted Markov policy
ProvedRiskSensMDP.Discounted.cost_iterationcost-iterationmarkov-decision-processp2o-batch-pfp2ap2o-gran-per-chapterp2o-plan-paperp2o-v1risk-sensitive
Let be a sequence of admissible measurable decision rules on , in the bounded positive-cost model under (CC). For and , the expected utility under is obtained by composing its first one-step operators:
This identifies the cost iteration of a fixed Markov policy before minimizing over policies. Formalization Note The evaluation of uses the history-policy embedding shifted by the initial ; its instance is the embedding displayed in the paper.
Preamble
import Mathlib import Definitions.Def_RiskSensMDP_Discounted_Model open MeasureTheory ProbabilityTheory Filter
Formal statement
namespace RiskSensMDP.Discounted
/-- Theorem 3.6(a), authors' manuscript p. 11: cost iteration for an extended-state
Markov policy. Formalization Note: `markovValue` uses the `(y,z)`-shifted embedding
of the Markov rules into finite-dimensional path expectations. -/
theorem cost_iteration {E A : Type*}
[TopologicalSpace E] [MeasurableSpace E] [BorelSpace E]
[TopologicalSpace.MetrizableSpace E] [SecondCountableTopology E]
[StandardBorelSpace E]
[TopologicalSpace A] [MeasurableSpace A] [BorelSpace A]
[TopologicalSpace.MetrizableSpace A] [SecondCountableTopology A]
[StandardBorelSpace A]
(M : Model E A) (hBounds : HasBounds M) (hCC : HasCC M)
(N : ℕ) (π : MarkovPolicy M) :
∀ n ∈ Finset.Icc 1 N, ∀ s ∈ extendedDomain E,
markovValue M π n s.1 s.2.1 s.2.2 = policyIteration M π n s := by sorry
end RiskSensMDP.Discounted
Source
Bäuerle & Rieder, More Risk-Sensitive Markov Decision Processes, authors' manuscript (KIT repository 1000039663; published Math. Oper. Res. 39(1):105–120, 2014), p. 11, Theorem 3.6(a)
Human review
Confirmed by the mission captain (proposal self-audit).
Confirmed by the moderator at approval.