Le sup' univ
ProvedNeuroSymbolicRLHF.le_sup_p_univaether-catalogspeculative
Formal statement of NeuroSymbolicRLHF.le_sup'_univ from the Aether Catalog (Speculative). The mathematical content is given by the Lean statement below; a human-readable write-up is pending.
theorem NeuroSymbolicRLHF.le_sup'_univ(f : ι → ℝ) (i : ι) : f i ≤ univ.sup' univ_nonempty f := by sorry
Formalization Note Transplanted verbatim from the Aether Catalog source Speculative/AutoResearch/RLHFHilbertIsometry.lean; the statement is byte-identical to the source declaration, elaborated with autoImplicit disabled in the platform environment.
Preamble
-- Thm stub generated from Speculative/AutoResearch/RLHFHilbertIsometry.lean
import Mathlib
import Definitions.Def_Speculative_AutoResearch_NeuroSymbolicRLHFObjective
import Definitions.Def_Speculative_AutoResearch_RLHFHilbertIsometry
/-
# RLHF as an isometry of the Hilbert projective metric
This file continues the neurosymbolic RLHF / PPO-ptx research thread of
`Speculative/AutoResearch/NeuroSymbolicRLHFObjective.lean`, whose definitions
(`tiltZ`, `gibbs`, `freeEnergy`, `rlhfObj`, `klDivFin`) are reused verbatim.
The InstructGPT objective
`Objective(p) = 𝔼_p[RM] - β · KL(p ‖ p_SFT) + γ · 𝔼_{pre}[log p]`
has, for `γ = 0`, the exponentially tilted maximiser `gibbs β ref r`. The
catalog already knows that tilting is a *transitive group action* of the reward
space on the open simplex (a torsor structure). Here we upgrade that algebraic
statement to a **metric** one, bridging information theory with the projective
geometry of Birkhoff and Hilbert:
* **Level 0.** Elementary theory of the oscillation seminorm
`oscil f = max f - min f` (translation invariance, positive homogeneity,
comparison with the sup-norm).
* **Level 1 (main theorem, `hilbertDist_gibbs`).** The tilt map is an *exact
isometry*, with scale factor `1/β`:
`d_H (π_β(r₁), π_β(r₂)) = oscil (r₁ - r₂) / β`.
In particular the RLHF torsor action of `(ℝ^ι/ℝ·1, oscil)` on the open
simplex equipped with the Hilbert projective metric is by isometries, and
`d_H (π_β(r), ref) = oscil r / β`.
* **Level 2 (`tvDist_le_expm1_hilbertDist`).** A Hilbert-metric bound controls
total variation, giving the quantitative reward-model-misspecification bound
`‖π_β(r₁) - π_β(r₂)‖_TV ≤ exp(oscil (r₁ - r₂)/β) - 1`:
KL-regularisation with large `β` makes the aligned policy insensitive to
reward-model error.
* **Level 3 (`rlhf_reward_hacking_regret`).** Optimising a *proxy* reward `r̂`
loses at most `2‖r - r̂‖_∞` of the true KL-regularised value — an explicit
reward-hacking (Goodhart) bound.
No `sorry`, no `native_decide`.
-/
open Finset Real BigOperators
noncomputable section
open NeuroSymbolicRLHF
variable {ι : Type*} [Fintype ι] [Nonempty ι]
/-! ## Level 0: the oscillation seminorm -/Formal statement
theorem NeuroSymbolicRLHF.le_sup_p_univ(f : ι → ℝ) (i : ι) : f i ≤ univ.sup' univ_nonempty f := by sorry
Source