Theorem 2.4.22 — in convex models the convex functions in satisfy the Structure Assumption
ProvedBauerleRiederMDP.ConvexModels.theorem_2_4_22Throughout, and are Borel sets carrying the componentwise order, and is a Markov Decision Model with planning horizon : are the admissible state–action pairs, the transition kernels, the one-stage rewards and the terminal reward. The model has an upper bounding function (Definition 2.4.1): , and on . Write
A maximizer of at time is a decision rule (a measurable with ) with for all .
Suppose that for all :
- is convex and ;
- for all convex , is convex for all ;
- is convex for all ;
- is convex;
- for all convex there exists a maximizer of , where .
Then the sets
and satisfy the Structure Assumption . (SAN) for sets , means: consists of measurable -valued functions, , , and for every and the operator is well defined, lies in , and has a maximizer in .
With Theorem 2.3.8, the value functions of such a model are convex and the maximizers in form an optimal policy.
Formalization Note is neither convex nor compact. Convex value functions may take the value .
Differences from the source:
- encoding: , Borel spaces with the componentwise order (pp.29, 35) → subtypes of measurable sets
SE ⊆ Fin d → ℝ,SA ⊆ Fin m → ℝwith the induced σ-algebra and the componentwise order; reason: a subset of that is a Borel space is a Borel subset of ; same instances - encoding: a stochastic kernel from to and (Definition 2.1.1) → the platform's
MarkovDecisionModel, whose kernel and reward are given on all of ; reason: only values on enter the statement; every model of the book extends (e.g. , off ) - encoding: functions with values in and 'whenever the integral exists' →
ERealvalues,erealIntegral= integral of minus integral of ; reason: for and the positive part has finite integral, so this is the book's integral - encoding: (SAN) (ii) ' is well-defined' → every integral , , exists (finite positive or negative part); reason: Definition 2.3.1 a) defines 'whenever the integral exists'
- encoding: 'convex' / 'concave' for -valued functions (no numbered definition) →
ConvexOnVia/ConcaveOnVia: the two-point inequality inERealalong the inclusion into (or ); reason: for functions without the value this is convexity of the epigraph (hypograph); allowed as in - reading: (unqualified in the theorem) → hypothesis (decision rules); reason: (SAN), p.23, takes ; without it the conclusion fails for a containing a non-measurable map
import Mathlib import Definitions.Def_MDPFinance_StructuredModels_Model import Definitions.Def_MDPFinance_StructuredModels_Policy import Definitions.Def_MDPFinance_StructuredModels_Operators import Definitions.Def_MDPFinance_StructuredModels_BoundingFunction import Definitions.Def_BauerleRiederMDP_ConvexModels_StructureAssumption import Definitions.Def_BauerleRiederMDP_ConvexModels_ConvexOnVia open MeasureTheory ProbabilityTheory
namespace BauerleRiederMDP.ConvexModels
open MDPFinance.StructuredModels
/-- **Theorem 2.4.22** (Bäuerle & Rieder, *Markov Decision Processes with Applications to
Finance* (2011), Theorem 2.4.22, p.38, PDF p.53). Let `E ⊂ ℝ^d`, `A ⊂ ℝ^m` be Borel sets and
suppose a Markov Decision Model with upper bounding function `b` is given such that for all
`n = 0, 1, …, N − 1`:
(i) `E` is convex and `D_n = E × A`;
(ii) for all convex `v ∈ IB_b^+`, `x ↦ ∫ v(x′) Q_n(dx′|x,a)` is convex for all `a ∈ A`;
(iii) `x ↦ r_n(x,a)` is convex for all `a ∈ A`;
(iv) `g_N` is convex;
(v) for all convex `v ∈ IB_b^+` there exists a maximizer `f_n ∈ Δ_n` of `v`.
Then `IM_n := {v ∈ IB_b^+ | v convex}` and `Δ_n` satisfy the Structure Assumption (SAN).
**Formalization Note.** `E`, `A` are subtypes of the Borel sets `SE ⊂ ℝ^d`, `SA ⊂ ℝ^m`
(`Fin d → ℝ` with its Borel σ-algebra); `A` is neither convex nor compact. The model is the
platform's `MarkovDecisionModel`; convexity of `[−∞, ∞)`-valued functions on `E` is
`ConvexOnVia` along the inclusion `E ⊂ ℝ^d`, and the value `−∞` is allowed in `IM_n`.
(SAN) includes measurability (`IM_n ⊂ IM(E)`) and the existence of the integrals in `T_n v`.
`Δ_n ⊂ F_n` (decision rules) is the book's standing meaning of `Δ_n` in (SAN), p.23. -/
theorem theorem_2_4_22 {d m N : ℕ} {SE : Set (Fin d → ℝ)} {SA : Set (Fin m → ℝ)}
(hSE : MeasurableSet SE) (hSA : MeasurableSet SA)
(M : MarkovDecisionModel SE SA N) (b : SE → ℝ) (cr cg αb : ℝ)
(hb : IsUpperBoundingFunction M b cr cg αb)
(Δ : ℕ → Set (SE → SA)) (hΔ : ∀ n < N, Δ n ⊆ {f | IsDecisionRule M n f})
(hE : Convex ℝ SE) (hD : ∀ n < N, M.D n = Set.univ)
(hQ : ∀ n < N, ∀ v ∈ IBbPlus b, ConvexOnVia Subtype.val Set.univ v →
∀ a : SA, ConvexOnVia Subtype.val Set.univ (fun x => erealIntegral (M.Q n (x, a)) v))
(hr : ∀ n < N, ∀ a : SA, ConvexOnVia Subtype.val Set.univ (fun x => (M.r n (x, a) : EReal)))
(hg : ConvexOnVia Subtype.val Set.univ (fun x => (M.g x : EReal)))
(hmax : ∀ n < N, ∀ v ∈ IBbPlus b, ConvexOnVia Subtype.val Set.univ v →
∃ f ∈ Δ n, IsMaximizer M n v f) :
StructureAssumption M (fun _ => {v ∈ IBbPlus b | ConvexOnVia Subtype.val Set.univ v}) Δ := by sorry
end BauerleRiederMDP.ConvexModels
Confirmed by the mission captain (proposal self-audit).
Confirmed by the moderator at approval.