§6.9, Theorem 1 — Normal equations characterize least squares
ProvedVectorSpaceOpt.normal_equations_iff_best_approximationadjointfunctional-analysishilbert-spaceleast-squaresnormal-equations
Let and be real Hilbert spaces, a continuous linear map, a target, and a candidate. Then globally minimizes the residual norm if and only if it satisfies the normal equation:
No closed-range assumption is needed for this equivalence. It converts an optimization property quantified over all candidates into an operator equation and provides the reusable interface between best approximation and the normal operators appearing in the pseudoinverse identities.
Preamble
import Mathlib open scoped InnerProduct
Formal statement
namespace VectorSpaceOpt
theorem normal_equations_iff_best_approximation
{G H : Type*} [NormedAddCommGroup G] [InnerProductSpace ℝ G] [CompleteSpace G]
[NormedAddCommGroup H] [InnerProductSpace ℝ H] [CompleteSpace H]
(A : G →L[ℝ] H) (y : H) (x : G) :
(∀ z : G, ‖y - A x‖ ≤ ‖y - A z‖) ↔ ((A†) ∘L A) x = (A†) y := by sorry
end VectorSpaceOptSource
David G. Luenberger, Optimization by Vector Space Methods, §6.9, Theorem 1, book p. 160 (PDF physical p. 180), https://sites.science.oregonstate.edu/~show/old/142_Luenberger.pdf