Zero-temperature limit.
ProvedRLHF.tendsto_freeEnergy_zero_temperatureaether-catalognumbertheory
Zero-temperature limit. As the KL coefficient vanishes, the optimal RLHF value converges to the maximal reward: alignment degenerates into greedy reward maximization.
theorem RLHF.tendsto_freeEnergy_zero_temperature{r p : Ω → ℝ} (hp : IsPosDist p) :
Tendsto (fun β => freeEnergy β r p) (𝓝[>] (0 : ℝ)) (𝓝 (rewardMax r)) := by sorry
Formalization Note Transplanted verbatim from the Aether Catalog source NumberTheory/RLHFTemperatureLimits.lean; the statement is byte-identical to the source declaration, elaborated with autoImplicit disabled in the platform environment.
Preamble
-- Thm stub generated from NumberTheory/RLHFTemperatureLimits.lean
import Mathlib
import Definitions.Def_NumberTheory_RLHFGibbsVariational
import Definitions.Def_NumberTheory_RLHFTemperatureLimits
import Definitions.Def_NumberTheory_RLHFTemperatureSpectrum
/-!
# Zero- and infinite-temperature limits of the RLHF free energy
Continuing `NumberTheory.RLHFTemperatureSpectrum`, we quantify the two endpoints of the
free-energy spectrum `V(β) = β log Z(β)`:
* `RLHF.kl_budget` — the aligned policy cannot collapse: `β · KL(π_β ‖ p) ≤ max r − min r`.
* `RLHF.freeEnergy_ge_point` — `V(β) ≥ r y + β log p y` for every response `y`.
* `RLHF.tendsto_freeEnergy_zero_temperature` — as `β → 0⁺`, `V(β) → max r`
(greedy reward maximization).
* `RLHF.freeEnergy_le_high_temperature` and `RLHF.tendsto_freeEnergy_high_temperature`
— as `β → ∞`, `V(β) → 𝔼_p[r]` (the SFT reference), with the explicit rate
`V(β) ≤ min r + e^{(max r − min r)/β} (𝔼_p[r] − min r)`.
Arithmetic payoff (`RLHF.vonMangoldt_zero_temperature_limit`): for the von Mangoldt reward
on `{1, …, N}` the zero-temperature limit of the RLHF free energy equals `log P` where `P`
is the **largest prime ≤ N**, while the infinite-temperature limit is the Chebyshev average
`ψ(N)/N`. The whole alignment spectrum of this reward model is thus pinned between two
classical prime-counting quantities.
-/
open RLHF
open Finset ArithmeticFunction Filter Topology
variable {Ω : Type*} [Fintype Ω] [Nonempty Ω]
/-! ## 1. Pointwise lower bounds and the KL budget -/
/-! ## 2. The zero-temperature (greedy) limit -/Formal statement
theorem RLHF.tendsto_freeEnergy_zero_temperature{r p : Ω → ℝ} (hp : IsPosDist p) :
Tendsto (fun β => freeEnergy β r p) (𝓝[>] (0 : ℝ)) (𝓝 (rewardMax r)) := by sorrySource