RLHF never hurts: the optimal value dominates the value of the SFT reference policy.
ProvedRLHF.freeEnergy_ge_referenceaether-catalognumbertheory
RLHF never hurts: the optimal value dominates the value of the SFT reference policy.
theorem RLHF.freeEnergy_ge_reference{β : ℝ} {r p : Ω → ℝ} (hβ : 0 < β) (hp : IsPosDist p) :
∑ y, p y * r y ≤ freeEnergy β r p := by sorry
Formalization Note Transplanted verbatim from the Aether Catalog source NumberTheory/RLHFTemperatureSpectrum.lean; the statement is byte-identical to the source declaration, elaborated with autoImplicit disabled in the platform environment.
Preamble
-- Thm stub generated from NumberTheory/RLHFTemperatureSpectrum.lean
import Mathlib
import Definitions.Def_NumberTheory_RLHFGibbsVariational
import Definitions.Def_NumberTheory_RLHFTemperatureSpectrum
/-!
# The RLHF free-energy spectrum and a von Mangoldt reward model
Building on `NumberTheory.RLHFGibbsVariational`, we study the *free energy*
`V(β) = β log Z(β)`, which by the Gibbs variational principle is the optimal value of
the KL-regularized RLHF objective at temperature `β`.
Main results:
* `RLHF.freeEnergy_antitone` — `V` is antitone in the KL coefficient `β`:
stronger regularization can only lower the achievable value.
* `RLHF.freeEnergy_le_of_le` — `V(β) ≤ sup r` (no reward hacking beyond the reward ceiling).
* `RLHF.freeEnergy_ge_reference` — `V(β) ≥ 𝔼_p[r]` (RLHF never hurts).
* `RLHF.gibbs_ne_reference_of_nonconstant` and `RLHF.strict_improvement` — RLHF strictly
improves on the SFT reference exactly when the reward model is non-constant.
* Number-theoretic instantiation: reward `r(n) = Λ(n)` (von Mangoldt) on the response
space `{1, …, N}` with the uniform SFT reference. Then the free energy is squeezed,
`ψ(N)/N ≤ V(β) ≤ log N` (`RLHF.vonMangoldt_freeEnergy_ge_chebyshev`,
`RLHF.vonMangoldt_freeEnergy_le_log`), and for `N ≥ 2` the lower bound is *strict*
(`RLHF.vonMangoldt_strict_improvement`): the alignment gain is powered exactly by the
irregularity of the primes.
-/
open RLHF
open Finset ArithmeticFunction
variable {Ω : Type*} [Fintype Ω] [Nonempty Ω]Formal statement
theorem RLHF.freeEnergy_ge_reference{β : ℝ} {r p : Ω → ℝ} (hβ : 0 < β) (hp : IsPosDist p) :
∑ y, p y * r y ≤ freeEnergy β r p := by sorrySource