Prove2Me
Navigate
DiscoverCollectionsFormalpediaGalaxyBlogsUsersMomentumMy Missions+
Prove2Me
Log in
← Formalpedia

Two-layer linear regression and gradient-flow model

Definition
FeatureDistortion_Model

by Minghui · Sep 26, 2026 · Mathlib c5ea003 (Lean v4.30.0)

linear-algebramachine-learningprobability

Notation: n=#{training examples}n = \#\{\text{training examples}\}n=#{training examples}, ddd is the input dimension, kkk the feature dimension, X:Rd→RnX:\mathbb R^d\to\mathbb R^nX:Rd→Rn the data map, YYY the labels, B:Rd→RkB:\mathbb R^d\to\mathbb R^kB:Rd→Rk the features, and v∈Rkv\in\mathbb R^kv∈Rk the head. Adjoint means Euclidean transpose. The loss is L^(v,B)=∥XB⊤v−Y∥2\widehat L(v,B)=\|XB^\top v-Y\|^2L(v,B)=∥XB⊤v−Y∥2, with no normalization. The probability model, when present, is explicitly specified below; deterministic flow statements involve no random data assumption.

Vec. For each natural number ddd, Vec⁡(d)\operatorname{Vec}(d)Vec(d) is the real Euclidean space with coordinates indexed by {0,…,d−1}\{0,\ldots,d-1\}{0,…,d−1}, with its usual inner product and norm. The case d=0d=0d=0 is included and consists only of the zero vector.

Features. For natural numbers d,kd,kd,k, Features⁡(d,k)\operatorname{Features}(d,k)Features(d,k) is the space of continuous real-linear maps Rd→Rk\mathbb R^d\to\mathbb R^kRd→Rk. There is no rank or normalization assumption. If either dimension is zero, this space consists only of the zero map.

weights. For natural numbers d,kd,kd,k, a continuous real-linear map B:Rd→RkB:\mathbb R^d\to\mathbb R^kB:Rd→Rk, and v∈Rkv\in\mathbb R^kv∈Rk, the weight vector is B∗v∈RdB^*v\in\mathbb R^dB∗v∈Rd, where B∗B^*B∗ is the Euclidean adjoint. Zero dimensions are permitted; the weight vector is zero if d=0d=0d=0 or k=0k=0k=0.

residual. For natural numbers n,d,kn,d,kn,d,k, continuous real-linear maps X:Rd→RnX:\mathbb R^d\to\mathbb R^nX:Rd→Rn and B:Rd→RkB:\mathbb R^d\to\mathbb R^kB:Rd→Rk, and vectors Y∈RnY\in\mathbb R^nY∈Rn and v∈Rkv\in\mathbb R^kv∈Rk, the residual is XB∗v−YXB^*v-YXB∗v−Y. All dimensions may be zero; the residual is zero if n=0n=0n=0 and is −Y-Y−Y if d=0d=0d=0 or k=0k=0k=0. There is no required relation between the arguments.

trainingLoss. For natural numbers n,d,kn,d,kn,d,k, continuous real-linear maps X:Rd→RnX:\mathbb R^d\to\mathbb R^nX:Rd→Rn and B:Rd→RkB:\mathbb R^d\to\mathbb R^kB:Rd→Rk, and vectors Y∈RnY\in\mathbb R^nY∈Rn and v∈Rkv\in\mathbb R^kv∈Rk, the training loss is ∥XB∗v−Y∥2\|XB^*v-Y\|^2∥XB∗v−Y∥2. There is no normalization by nnn or factor of 1/21/21/2. If n=0n=0n=0 it is zero; if d=0d=0d=0 or k=0k=0k=0 it is ∥Y∥2\|Y\|^2∥Y∥2.

headGradient. For natural numbers n,d,kn,d,kn,d,k, continuous real-linear maps X:Rd→RnX:\mathbb R^d\to\mathbb R^nX:Rd→Rn and B:Rd→RkB:\mathbb R^d\to\mathbb R^kB:Rd→Rk, and vectors Y∈RnY\in\mathbb R^nY∈Rn and v∈Rkv\in\mathbb R^kv∈Rk, this definition assigns the vector 2BX∗(XB∗v−Y)∈Rk2BX^*(XB^*v-Y)\in\mathbb R^k2BX∗(XB∗v−Y)∈Rk, where the adjoints are Euclidean. There are no further hypotheses. The vector is zero if any of the dimensions is zero.

featureGradient. For natural numbers n,d,kn,d,kn,d,k, continuous real-linear maps X:Rd→RnX:\mathbb R^d\to\mathbb R^nX:Rd→Rn and B:Rd→RkB:\mathbb R^d\to\mathbb R^kB:Rd→Rk, and vectors Y∈RnY\in\mathbb R^nY∈Rn and v∈Rkv\in\mathbb R^kv∈Rk, this definition assigns the continuous real-linear map x↦2⟨X∗(XB∗v−Y),x⟩vx\mapsto2\langle X^*(XB^*v-Y),x\rangle vx↦2⟨X∗(XB∗v−Y),x⟩v from Rd\mathbb R^dRd to Rk\mathbb R^kRk. Thus the rank-one map takes an inner product with X∗(XB∗v−Y)X^*(XB^*v-Y)X∗(XB∗v−Y) and multiplies vvv by that scalar. No dimension is required to be positive; the map is zero if any of n,d,kn,d,kn,d,k is zero.

Trajectory. For natural numbers d,kd,kd,k, a trajectory is a pair of functions a:R→Rka:\mathbb R\to\mathbb R^ka:R→Rk and F:R→L(Rd,Rk)F:\mathbb R\to\mathcal L(\mathbb R^d,\mathbb R^k)F:R→L(Rd,Rk), called its head and features, respectively; L\mathcal LL denotes continuous real-linear maps. The structure itself imposes no continuity, differentiability, initial condition, or differential equation on these functions. Both are defined at all real times, and zero dimensions are permitted.

IsFineTuningFlow. For natural numbers n,d,kn,d,kn,d,k, continuous real-linear maps X:Rd→RnX:\mathbb R^d\to\mathbb R^nX:Rd→Rn and B0:Rd→RkB_0:\mathbb R^d\to\mathbb R^kB0​:Rd→Rk, vectors Y∈RnY\in\mathbb R^nY∈Rn and v0∈Rkv_0\in\mathbb R^kv0​∈Rk, and a pair of functions a:R→Rka:\mathbb R\to\mathbb R^ka:R→Rk and F:R→L(Rd,Rk)F:\mathbb R\to\mathcal L(\mathbb R^d,\mathbb R^k)F:R→L(Rd,Rk), this predicate means a(0)=v0a(0)=v_0a(0)=v0​, F(0)=B0F(0)=B_0F(0)=B0​, and, for every real t≥0t\geq0t≥0, existence of derivatives within [0,∞)[0,\infty)[0,∞) equal to a˙(t)=−2F(t)X∗(XF(t)∗a(t)−Y)\dot a(t)=-2F(t)X^*(XF(t)^*a(t)-Y)a˙(t)=−2F(t)X∗(XF(t)∗a(t)−Y) and F˙(t)=[x↦−2⟨X∗(XF(t)∗a(t)−Y),x⟩a(t)]\dot F(t)=\bigl[x\mapsto-2\langle X^*(XF(t)^*a(t)-Y),x\rangle a(t)\bigr]F˙(t)=[x↦−2⟨X∗(XF(t)∗a(t)−Y),x⟩a(t)]. The latter is a derivative in the space of continuous linear maps. At zero, the derivatives are within the half-line. No conditions apply at negative times. Zero dimensions are allowed.

IsLinearProbingFlow. For natural numbers n,d,kn,d,kn,d,k, continuous real-linear maps X:Rd→RnX:\mathbb R^d\to\mathbb R^nX:Rd→Rn and B0:Rd→RkB_0:\mathbb R^d\to\mathbb R^kB0​:Rd→Rk, vectors Y∈RnY\in\mathbb R^nY∈Rn and v0∈Rkv_0\in\mathbb R^kv0​∈Rk, and a function a:R→Rka:\mathbb R\to\mathbb R^ka:R→Rk, this predicate means a(0)=v0a(0)=v_0a(0)=v0​ and existence, at every real t≥0t\geq0t≥0, of a derivative within [0,∞)[0,\infty)[0,∞) equal to −2B0X∗(XB0∗a(t)−Y)-2B_0X^*(XB_0^*a(t)-Y)−2B0​X∗(XB0∗​a(t)−Y). The map B0B_0B0​ is fixed. Values at negative times are unconstrained, and zero dimensions are included.

rowSpace. For natural numbers d,nd,nd,n and a continuous real-linear map X:Rd→RnX:\mathbb R^d\to\mathbb R^nX:Rd→Rn, this definition assigns the subspace {X∗z:z∈Rn}\{X^*z:z\in\mathbb R^n\}{X∗z:z∈Rn} of Rd\mathbb R^dRd, where X∗X^*X∗ is the Euclidean adjoint. It is the zero subspace if X=0X=0X=0, n=0n=0n=0, or d=0d=0d=0.

OrthonormalRows. For natural numbers d,kd,kd,k and a continuous real-linear map B:Rd→RkB:\mathbb R^d\to\mathbb R^kB:Rd→Rk, this predicate means BB∗=IRkBB^*=I_{\mathbb R^k}BB∗=IRk​, equivalently BB∗v=vBB^*v=vBB∗v=v for every v∈Rkv\in\mathbb R^kv∈Rk. It holds for the unique map when k=0k=0k=0; it cannot hold when d=0<kd=0<kd=0<k.

VisibleOn. For natural numbers d,kd,kd,k, a continuous real-linear map B:Rd→RkB:\mathbb R^d\to\mathbb R^kB:Rd→Rk, and a real linear subspace S⊆RdS\subseteq\mathbb R^dS⊆Rd, this predicate means that v↦ΠS(B∗v)v\mapsto\Pi_S(B^*v)v↦ΠS​(B∗v) is injective, where ΠS\Pi_SΠS​ is orthogonal projection: equality of these projections for any v1,v2∈Rkv_1,v_2\in\mathbb R^kv1​,v2​∈Rk implies v1=v2v_1=v_2v1​=v2​. If k=0k=0k=0 this holds; if S={0}S=\{0\}S={0} and k>0k>0k>0 it cannot hold. Zero dimensions are included.

balance. For natural numbers d,kd,kd,k, a vector v∈Rkv\in\mathbb R^kv∈Rk, and a continuous real-linear map B:Rd→RkB:\mathbb R^d\to\mathbb R^kB:Rd→Rk, the balance is the endomorphism of Rk\mathbb R^kRk sending zzz to ⟨v,z⟩v−BB∗z\langle v,z\rangle v-BB^*z⟨v,z⟩v−BB∗z, namely vvT−BB∗vv^{\mathsf T}-BB^*vvT−BB∗. No normalization is required. It is the zero endomorphism if k=0k=0k=0, and its second summand is zero if d=0d=0d=0.

alignmentError. For a natural number kkk and vectors v0,u∈Rkv_0,u\in\mathbb R^kv0​,u∈Rk, the alignment error is ∣⟨v0,u⟩2−⟨u,u⟩2∣=∣⟨v0,u⟩2−∥u∥4∣\left|\langle v_0,u\rangle^2-\langle u,u\rangle^2\right|=\left|\langle v_0,u\rangle^2-\|u\|^4\right|​⟨v0​,u⟩2−⟨u,u⟩2​=​⟨v0​,u⟩2−∥u∥4​. It vanishes exactly when ⟨v0,u⟩=∥u∥2\langle v_0,u\rangle=\|u\|^2⟨v0​,u⟩=∥u∥2 or ⟨v0,u⟩=−∥u∥2\langle v_0,u\rangle=-\|u\|^2⟨v0​,u⟩=−∥u∥2. No nonzero assumption is imposed. If u=0u=0u=0, including when k=0k=0k=0, the error is zero for every v0v_0v0​.

secondMoment. For a natural number ddd and any measure μ\muμ on Rd\mathbb R^dRd, this definition assigns the operator-valued Bochner integral Σμ=∫(x⊗x) dμ(x)\Sigma_\mu=\int(x\otimes x)\,d\mu(x)Σμ​=∫(x⊗x)dμ(x), where (x⊗x)(w)=⟨x,w⟩x(x\otimes x)(w)=\langle x,w\rangle x(x⊗x)(w)=⟨x,w⟩x. It is an uncentered second moment. There is no probability or integrability assumption in this definition; the total Bochner integral is zero if its integrand is not integrable. For d=0d=0d=0 the operator is zero.

OODLaw. For a natural number ddd, an OOD law consists of a probability measure μ\muμ on Rd\mathbb R^dRd, the hypothesis that x↦∥x∥2x\mapsto\|x\|^2x↦∥x∥2 is integrable with respect to μ\muμ, and the hypothesis that ⟨w,Σμw⟩>0\langle w,\Sigma_\mu w\rangle>0⟨w,Σμ​w⟩>0 for every nonzero w∈Rdw\in\mathbb R^dw∈Rd, where Σμ=∫(x⊗x) dμ(x)\Sigma_\mu=\int(x\otimes x)\,d\mu(x)Σμ​=∫(x⊗x)dμ(x) and (x⊗x)(z)=⟨x,z⟩x(x\otimes x)(z)=\langle x,z\rangle x(x⊗x)(z)=⟨x,z⟩x. There is no mean-zero assumption. For d=0d=0d=0, the positivity condition is vacuous and the probability measure is concentrated at the sole zero vector.

oodLoss. For natural numbers d,kd,kd,k, a probability measure μ\muμ on Rd\mathbb R^dRd with integrable squared norm and ⟨z,Σμz⟩>0\langle z,\Sigma_\mu z\rangle>0⟨z,Σμ​z⟩>0 for every nonzero zzz, a target w∈Rdw\in\mathbb R^dw∈Rd, a vector v∈Rkv\in\mathbb R^kv∈Rk, and a continuous real-linear map B:Rd→RkB:\mathbb R^d\to\mathbb R^kB:Rd→Rk, this definition assigns ∫⟨B∗v−w,x⟩2 dμ(x)\int\langle B^*v-w,x\rangle^2\,d\mu(x)∫⟨B∗v−w,x⟩2dμ(x). Here Σμ=∫(x⊗x) dμ(x)\Sigma_\mu=\int(x\otimes x)\,d\mu(x)Σμ​=∫(x⊗x)dμ(x), with (x⊗x)(z)=⟨x,z⟩x(x\otimes x)(z)=\langle x,z\rangle x(x⊗x)(z)=⟨x,z⟩x, and ∗^*∗ is the Euclidean adjoint. If d=0d=0d=0 the loss is zero; if k=0k=0k=0 it is ∫⟨−w,x⟩2 dμ(x)\int\langle-w,x\rangle^2\,d\mu(x)∫⟨−w,x⟩2dμ(x).

gaussianHead. For a natural number kkk and any real number σ\sigmaσ, this definition assigns the pushforward of standard Gaussian measure on Euclidean Rk\mathbb R^kRk under z↦σzz\mapsto\sigma zz↦σz. Positive, negative, and zero σ\sigmaσ are all allowed. For σ=0\sigma=0σ=0 this is the point mass at zero; for k=0k=0k=0 it is the point mass at the only vector for every σ\sigmaσ.

Problem. For natural numbers n,d,kn,d,kn,d,k, a problem consists of continuous real-linear maps X:Rd→RnX:\mathbb R^d\to\mathbb R^nX:Rd→Rn and B⋆:Rd→RkB_\star:\mathbb R^d\to\mathbb R^kB⋆​:Rd→Rk, called its data and optimal features; a real-linear isometric bijection R:Rk→RkR:\mathbb R^k\to\mathbb R^kR:Rk→Rk, called its rotation; and a vector u⋆∈Rku_\star\in\mathbb R^ku⋆​∈Rk, called its optimal head. No other conditions are imposed on these fields. In particular, positive determinant is not required of RRR. Zero dimensions are included.

initialFeatures. For natural numbers n,d,kn,d,kn,d,k and a problem with continuous real-linear maps X:Rd→RnX:\mathbb R^d\to\mathbb R^nX:Rd→Rn and B⋆:Rd→RkB_\star:\mathbb R^d\to\mathbb R^kB⋆​:Rd→Rk, a real-linear isometric bijection R:Rk→RkR:\mathbb R^k\to\mathbb R^kR:Rk→Rk, and u⋆∈Rku_\star\in\mathbb R^ku⋆​∈Rk, the initial feature map is RB⋆:Rd→RkRB_\star:\mathbb R^d\to\mathbb R^kRB⋆​:Rd→Rk. It does not depend on XXX or u⋆u_\staru⋆​ and requires no admissibility hypothesis. If d=0d=0d=0 or k=0k=0k=0, it is zero.

alignedHead. For natural numbers n,d,kn,d,kn,d,k and a problem with continuous real-linear maps X:Rd→RnX:\mathbb R^d\to\mathbb R^nX:Rd→Rn and B⋆:Rd→RkB_\star:\mathbb R^d\to\mathbb R^kB⋆​:Rd→Rk, a real-linear isometric bijection R:Rk→RkR:\mathbb R^k\to\mathbb R^kR:Rk→Rk, and u⋆∈Rku_\star\in\mathbb R^ku⋆​∈Rk, the aligned head is Ru⋆Ru_\starRu⋆​. It does not depend on XXX or B⋆B_\starB⋆​ and requires no admissibility hypothesis. It is zero if u⋆=0u_\star=0u⋆​=0, including when k=0k=0k=0.

targetWeights. For natural numbers n,d,kn,d,kn,d,k and a problem with continuous real-linear maps X:Rd→RnX:\mathbb R^d\to\mathbb R^nX:Rd→Rn and B⋆:Rd→RkB_\star:\mathbb R^d\to\mathbb R^kB⋆​:Rd→Rk, a real-linear isometric bijection R:Rk→RkR:\mathbb R^k\to\mathbb R^kR:Rk→Rk, and u⋆∈Rku_\star\in\mathbb R^ku⋆​∈Rk, the target vector is B⋆∗u⋆∈RdB_\star^*u_\star\in\mathbb R^dB⋆∗​u⋆​∈Rd. It does not depend on XXX or RRR and requires no admissibility hypothesis. It is zero if d=0d=0d=0 or k=0k=0k=0.

labels. For natural numbers n,d,kn,d,kn,d,k and a problem with continuous real-linear maps X:Rd→RnX:\mathbb R^d\to\mathbb R^nX:Rd→Rn and B⋆:Rd→RkB_\star:\mathbb R^d\to\mathbb R^kB⋆​:Rd→Rk, a real-linear isometric bijection R:Rk→RkR:\mathbb R^k\to\mathbb R^kR:Rk→Rk, and u⋆∈Rku_\star\in\mathbb R^ku⋆​∈Rk, the label vector is XB⋆∗u⋆∈RnXB_\star^*u_\star\in\mathbb R^nXB⋆∗​u⋆​∈Rn. It does not depend on RRR and requires no admissibility hypothesis. It is zero if any of n,d,kn,d,kn,d,k is zero.

Admissible. For natural numbers n,d,kn,d,kn,d,k and a problem consisting of continuous real-linear maps X:Rd→RnX:\mathbb R^d\to\mathbb R^nX:Rd→Rn and B⋆:Rd→RkB_\star:\mathbb R^d\to\mathbb R^kB⋆​:Rd→Rk, a real-linear isometric bijection R:Rk→RkR:\mathbb R^k\to\mathbb R^kR:Rk→Rk, and u⋆∈Rku_\star\in\mathbb R^ku⋆​∈Rk, let B0=RB⋆B_0=RB_\starB0​=RB⋆​, S={X∗z:z∈Rn}S=\{X^*z:z\in\mathbb R^n\}S={X∗z:z∈Rn}, and r=dim⁡RSr=\dim_{\mathbb R}Sr=dimR​S. Admissibility is the conjunction 0<k0<k0<k, k≤rk\leq rk≤r, r+k<dr+k<dr+k<d, B⋆B⋆∗=IRkB_\star B_\star^*=I_{\mathbb R^k}B⋆​B⋆∗​=IRk​, u⋆≠0u_\star\neq0u⋆​=0, injectivity of v↦ΠS(B0∗v)v\mapsto\Pi_S(B_0^*v)v↦ΠS​(B0∗​v), and injectivity of v↦ΠS⊥(B0∗v)v\mapsto\Pi_{S^\perp}(B_0^*v)v↦ΠS⊥​(B0∗​v), both on Rk\mathbb R^kRk, where Π\PiΠ denotes orthogonal projection. Each injectivity requirement says equality of the displayed projections of two vectors implies equality of the vectors. These hypotheses exclude zero dimensions and require n≥kn\geq kn≥k and d≥2k+1d\geq2k+1d≥2k+1.

Formalization note: source-derived model for the source-backed parent; no flow existence, convergence, invariant, or desired risk bound is assumed. Source: Kumar, Raghunathan, Jones, Ma, and Liang, Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution, ICLR 2022, https://arxiv.org/pdf/2202.10054v1. Section 3.1, PDF pp. 4--6, equations (3.2)--(3.5). Source-backed parent: Section 3.4, PDF p. 10, Proposition 3.7, equations (3.10)--(3.11); Appendix A.7, PDF pp. 45--47.

Definition code
import Mathlib.Analysis.Calculus.Deriv.Basic
import Mathlib.Analysis.InnerProductSpace.Adjoint
import Mathlib.Analysis.InnerProductSpace.PiL2
import Mathlib.Probability.Distributions.Gaussian.Multivariate

/-!
Two-layer linear regression and its gradient flows. Source: Kumar et al.,
arXiv:2202.10054v1, Sections 3.1 and 3.4, PDF pp. 4--6 and 10;
Appendix A.2, PDF pp. 23--26, and A.7, PDF pp. 45--47.
The feature flow is the explicit Frobenius-gradient equation (A.17).
The continuous-linear-map norm is the Euclidean operator norm.
-/

noncomputable section
open MeasureTheory

namespace FeatureDistortion

abbrev Vec (d : ℕ) := EuclideanSpace ℝ (Fin d)

abbrev Features (d k : ℕ) := Vec d →L[ℝ] Vec k

def weights {d k : ℕ} (B : Features d k) (v : Vec k) : Vec d := B.adjoint v

def residual {n d k : ℕ} (X : Vec d →L[ℝ] Vec n) (Y : Vec n)
    (v : Vec k) (B : Features d k) : Vec n := X (weights B v) - Y

def trainingLoss {n d k : ℕ} (X : Vec d →L[ℝ] Vec n) (Y : Vec n)
    (v : Vec k) (B : Features d k) : ℝ := ‖residual X Y v B‖ ^ 2

def headGradient {n d k : ℕ} (X : Vec d →L[ℝ] Vec n) (Y : Vec n)
    (v : Vec k) (B : Features d k) : Vec k :=
  (2 : ℝ) • B (X.adjoint (residual X Y v B))

def featureGradient {n d k : ℕ} (X : Vec d →L[ℝ] Vec n) (Y : Vec n)
    (v : Vec k) (B : Features d k) : Features d k :=
  (2 : ℝ) • InnerProductSpace.rankOne ℝ v (X.adjoint (residual X Y v B))

structure Trajectory (d k : ℕ) where
  head : ℝ → Vec k
  features : ℝ → Features d k

def IsFineTuningFlow {n d k : ℕ} (X : Vec d →L[ℝ] Vec n) (Y : Vec n)
    (v₀ : Vec k) (B₀ : Features d k) (γ : Trajectory d k) : Prop :=
  γ.head 0 = v₀ ∧ γ.features 0 = B₀ ∧
    ∀ t : ℝ, 0 ≤ t →
      HasDerivWithinAt γ.head (-headGradient X Y (γ.head t) (γ.features t))
        (Set.Ici 0) t ∧
      HasDerivWithinAt γ.features (-featureGradient X Y (γ.head t) (γ.features t))
        (Set.Ici 0) t

def IsLinearProbingFlow {n d k : ℕ} (X : Vec d →L[ℝ] Vec n) (Y : Vec n)
    (v₀ : Vec k) (B₀ : Features d k) (v : ℝ → Vec k) : Prop :=
  v 0 = v₀ ∧ ∀ t : ℝ, 0 ≤ t →
    HasDerivWithinAt v (-headGradient X Y (v t) B₀) (Set.Ici 0) t

def rowSpace {d n : ℕ} (X : Vec d →L[ℝ] Vec n) : Submodule ℝ (Vec d) :=
  LinearMap.range X.adjoint.toLinearMap

def OrthonormalRows {d k : ℕ} (B : Features d k) : Prop :=
  B.comp B.adjoint = ContinuousLinearMap.id ℝ (Vec k)

/-- In the stated dimension regime this is positivity of the smallest
principal-angle cosine, rather than merely a nonzero projection. -/
def VisibleOn {d k : ℕ} (B : Features d k) (S : Submodule ℝ (Vec d)) : Prop :=
  Function.Injective (fun v : Vec k => S.starProjection (B.adjoint v))

def balance {d k : ℕ} (v : Vec k) (B : Features d k) : Vec k →L[ℝ] Vec k :=
  InnerProductSpace.rankOne ℝ v v - B.comp B.adjoint

def alignmentError {k : ℕ} (v₀ u : Vec k) : ℝ :=
  |(inner ℝ v₀ u) ^ 2 - (inner ℝ u u) ^ 2|

/-- The uncentered second moment, not the centered covariance operator. -/
def secondMoment {d : ℕ} (μ : Measure (Vec d)) : Vec d →L[ℝ] Vec d :=
  ∫ x, InnerProductSpace.rankOne ℝ x x ∂μ

structure OODLaw (d : ℕ) where
  measure : Measure (Vec d)
  isProbability : IsProbabilityMeasure measure
  finiteSecondMoment : Integrable (fun x : Vec d => ‖x‖ ^ 2) measure
  positiveSecondMoment : ∀ w : Vec d, w ≠ 0 →
    0 < inner ℝ w (secondMoment measure w)

def oodLoss {d k : ℕ} (D : OODLaw d) (w : Vec d) (v : Vec k)
    (B : Features d k) : ℝ :=
  ∫ x, (inner ℝ (weights B v - w) x) ^ 2 ∂D.measure

def gaussianHead (k : ℕ) (σ : ℝ) : Measure (Vec k) :=
  (ProbabilityTheory.stdGaussian (Vec k)).map (fun z => σ • z)

structure Problem (n d k : ℕ) where
  data : Vec d →L[ℝ] Vec n
  optimalFeatures : Features d k
  rotation : Vec k ≃ₗᵢ[ℝ] Vec k
  optimalHead : Vec k

def initialFeatures {n d k : ℕ} (P : Problem n d k) : Features d k :=
  P.rotation.toLinearIsometry.toContinuousLinearMap.comp P.optimalFeatures

def alignedHead {n d k : ℕ} (P : Problem n d k) : Vec k :=
  P.rotation P.optimalHead

def targetWeights {n d k : ℕ} (P : Problem n d k) : Vec d :=
  weights P.optimalFeatures P.optimalHead

def labels {n d k : ℕ} (P : Problem n d k) : Vec n :=
  P.data (targetWeights P)

def Admissible {n d k : ℕ} (P : Problem n d k) : Prop :=
  0 < k ∧
  k ≤ Module.finrank ℝ (rowSpace P.data) ∧
  Module.finrank ℝ (rowSpace P.data) + k < d ∧
  OrthonormalRows P.optimalFeatures ∧
  P.optimalHead ≠ 0 ∧
  VisibleOn (initialFeatures P) (rowSpace P.data) ∧
  VisibleOn (initialFeatures P) (rowSpace P.data)ᗮ

end FeatureDistortion
Source
Kumar, Raghunathan, Jones, Ma, and Liang, Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution, ICLR 2022, https://arxiv.org/pdf/2202.10054v1. Section 3.1, PDF pp. 4--6, equations (3.2)--(3.5). Source-backed parent: Section 3.4, PDF p. 10, Proposition 3.7, equations (3.10)--(3.11); Appendix A.7, PDF pp. 45--47.
Read-back

What the Lean code literally says, in plain math · gpt-6

Vec. For each natural number ddd, Vec⁡(d)\operatorname{Vec}(d)Vec(d) is the real Euclidean space with coordinates indexed by {0,…,d−1}\{0,\ldots,d-1\}{0,…,d−1}, with its usual inner product and norm. The case d=0d=0d=0 is included and consists only of the zero vector.

Features. For natural numbers d,kd,kd,k, Features⁡(d,k)\operatorname{Features}(d,k)Features(d,k) is the space of continuous real-linear maps Rd→Rk\mathbb R^d\to\mathbb R^kRd→Rk. There is no rank or normalization assumption. If either dimension is zero, this space consists only of the zero map.

weights. For natural numbers d,kd,kd,k, a continuous real-linear map B:Rd→RkB:\mathbb R^d\to\mathbb R^kB:Rd→Rk, and v∈Rkv\in\mathbb R^kv∈Rk, the weight vector is B∗v∈RdB^*v\in\mathbb R^dB∗v∈Rd, where B∗B^*B∗ is the Euclidean adjoint. Zero dimensions are permitted; the weight vector is zero if d=0d=0d=0 or k=0k=0k=0.

residual. For natural numbers n,d,kn,d,kn,d,k, continuous real-linear maps X:Rd→RnX:\mathbb R^d\to\mathbb R^nX:Rd→Rn and B:Rd→RkB:\mathbb R^d\to\mathbb R^kB:Rd→Rk, and vectors Y∈RnY\in\mathbb R^nY∈Rn and v∈Rkv\in\mathbb R^kv∈Rk, the residual is XB∗v−YXB^*v-YXB∗v−Y. All dimensions may be zero; the residual is zero if n=0n=0n=0 and is −Y-Y−Y if d=0d=0d=0 or k=0k=0k=0. There is no required relation between the arguments.

trainingLoss. For natural numbers n,d,kn,d,kn,d,k, continuous real-linear maps X:Rd→RnX:\mathbb R^d\to\mathbb R^nX:Rd→Rn and B:Rd→RkB:\mathbb R^d\to\mathbb R^kB:Rd→Rk, and vectors Y∈RnY\in\mathbb R^nY∈Rn and v∈Rkv\in\mathbb R^kv∈Rk, the training loss is ∥XB∗v−Y∥2\|XB^*v-Y\|^2∥XB∗v−Y∥2. There is no normalization by nnn or factor of 1/21/21/2. If n=0n=0n=0 it is zero; if d=0d=0d=0 or k=0k=0k=0 it is ∥Y∥2\|Y\|^2∥Y∥2.

headGradient. For natural numbers n,d,kn,d,kn,d,k, continuous real-linear maps X:Rd→RnX:\mathbb R^d\to\mathbb R^nX:Rd→Rn and B:Rd→RkB:\mathbb R^d\to\mathbb R^kB:Rd→Rk, and vectors Y∈RnY\in\mathbb R^nY∈Rn and v∈Rkv\in\mathbb R^kv∈Rk, this definition assigns the vector 2BX∗(XB∗v−Y)∈Rk2BX^*(XB^*v-Y)\in\mathbb R^k2BX∗(XB∗v−Y)∈Rk, where the adjoints are Euclidean. There are no further hypotheses. The vector is zero if any of the dimensions is zero.

featureGradient. For natural numbers n,d,kn,d,kn,d,k, continuous real-linear maps X:Rd→RnX:\mathbb R^d\to\mathbb R^nX:Rd→Rn and B:Rd→RkB:\mathbb R^d\to\mathbb R^kB:Rd→Rk, and vectors Y∈RnY\in\mathbb R^nY∈Rn and v∈Rkv\in\mathbb R^kv∈Rk, this definition assigns the continuous real-linear map x↦2⟨X∗(XB∗v−Y),x⟩vx\mapsto2\langle X^*(XB^*v-Y),x\rangle vx↦2⟨X∗(XB∗v−Y),x⟩v from Rd\mathbb R^dRd to Rk\mathbb R^kRk. Thus the rank-one map takes an inner product with X∗(XB∗v−Y)X^*(XB^*v-Y)X∗(XB∗v−Y) and multiplies vvv by that scalar. No dimension is required to be positive; the map is zero if any of n,d,kn,d,kn,d,k is zero.

Trajectory. For natural numbers d,kd,kd,k, a trajectory is a pair of functions a:R→Rka:\mathbb R\to\mathbb R^ka:R→Rk and F:R→L(Rd,Rk)F:\mathbb R\to\mathcal L(\mathbb R^d,\mathbb R^k)F:R→L(Rd,Rk), called its head and features, respectively; L\mathcal LL denotes continuous real-linear maps. The structure itself imposes no continuity, differentiability, initial condition, or differential equation on these functions. Both are defined at all real times, and zero dimensions are permitted.

IsFineTuningFlow. For natural numbers n,d,kn,d,kn,d,k, continuous real-linear maps X:Rd→RnX:\mathbb R^d\to\mathbb R^nX:Rd→Rn and B0:Rd→RkB_0:\mathbb R^d\to\mathbb R^kB0​:Rd→Rk, vectors Y∈RnY\in\mathbb R^nY∈Rn and v0∈Rkv_0\in\mathbb R^kv0​∈Rk, and a pair of functions a:R→Rka:\mathbb R\to\mathbb R^ka:R→Rk and F:R→L(Rd,Rk)F:\mathbb R\to\mathcal L(\mathbb R^d,\mathbb R^k)F:R→L(Rd,Rk), this predicate means a(0)=v0a(0)=v_0a(0)=v0​, F(0)=B0F(0)=B_0F(0)=B0​, and, for every real t≥0t\geq0t≥0, existence of derivatives within [0,∞)[0,\infty)[0,∞) equal to a˙(t)=−2F(t)X∗(XF(t)∗a(t)−Y)\dot a(t)=-2F(t)X^*(XF(t)^*a(t)-Y)a˙(t)=−2F(t)X∗(XF(t)∗a(t)−Y) and F˙(t)=[x↦−2⟨X∗(XF(t)∗a(t)−Y),x⟩a(t)]\dot F(t)=\bigl[x\mapsto-2\langle X^*(XF(t)^*a(t)-Y),x\rangle a(t)\bigr]F˙(t)=[x↦−2⟨X∗(XF(t)∗a(t)−Y),x⟩a(t)]. The latter is a derivative in the space of continuous linear maps. At zero, the derivatives are within the half-line. No conditions apply at negative times. Zero dimensions are allowed.

IsLinearProbingFlow. For natural numbers n,d,kn,d,kn,d,k, continuous real-linear maps X:Rd→RnX:\mathbb R^d\to\mathbb R^nX:Rd→Rn and B0:Rd→RkB_0:\mathbb R^d\to\mathbb R^kB0​:Rd→Rk, vectors Y∈RnY\in\mathbb R^nY∈Rn and v0∈Rkv_0\in\mathbb R^kv0​∈Rk, and a function a:R→Rka:\mathbb R\to\mathbb R^ka:R→Rk, this predicate means a(0)=v0a(0)=v_0a(0)=v0​ and existence, at every real t≥0t\geq0t≥0, of a derivative within [0,∞)[0,\infty)[0,∞) equal to −2B0X∗(XB0∗a(t)−Y)-2B_0X^*(XB_0^*a(t)-Y)−2B0​X∗(XB0∗​a(t)−Y). The map B0B_0B0​ is fixed. Values at negative times are unconstrained, and zero dimensions are included.

rowSpace. For natural numbers d,nd,nd,n and a continuous real-linear map X:Rd→RnX:\mathbb R^d\to\mathbb R^nX:Rd→Rn, this definition assigns the subspace {X∗z:z∈Rn}\{X^*z:z\in\mathbb R^n\}{X∗z:z∈Rn} of Rd\mathbb R^dRd, where X∗X^*X∗ is the Euclidean adjoint. It is the zero subspace if X=0X=0X=0, n=0n=0n=0, or d=0d=0d=0.

OrthonormalRows. For natural numbers d,kd,kd,k and a continuous real-linear map B:Rd→RkB:\mathbb R^d\to\mathbb R^kB:Rd→Rk, this predicate means BB∗=IRkBB^*=I_{\mathbb R^k}BB∗=IRk​, equivalently BB∗v=vBB^*v=vBB∗v=v for every v∈Rkv\in\mathbb R^kv∈Rk. It holds for the unique map when k=0k=0k=0; it cannot hold when d=0<kd=0<kd=0<k.

VisibleOn. For natural numbers d,kd,kd,k, a continuous real-linear map B:Rd→RkB:\mathbb R^d\to\mathbb R^kB:Rd→Rk, and a real linear subspace S⊆RdS\subseteq\mathbb R^dS⊆Rd, this predicate means that v↦ΠS(B∗v)v\mapsto\Pi_S(B^*v)v↦ΠS​(B∗v) is injective, where ΠS\Pi_SΠS​ is orthogonal projection: equality of these projections for any v1,v2∈Rkv_1,v_2\in\mathbb R^kv1​,v2​∈Rk implies v1=v2v_1=v_2v1​=v2​. If k=0k=0k=0 this holds; if S={0}S=\{0\}S={0} and k>0k>0k>0 it cannot hold. Zero dimensions are included.

balance. For natural numbers d,kd,kd,k, a vector v∈Rkv\in\mathbb R^kv∈Rk, and a continuous real-linear map B:Rd→RkB:\mathbb R^d\to\mathbb R^kB:Rd→Rk, the balance is the endomorphism of Rk\mathbb R^kRk sending zzz to ⟨v,z⟩v−BB∗z\langle v,z\rangle v-BB^*z⟨v,z⟩v−BB∗z, namely vvT−BB∗vv^{\mathsf T}-BB^*vvT−BB∗. No normalization is required. It is the zero endomorphism if k=0k=0k=0, and its second summand is zero if d=0d=0d=0.

alignmentError. For a natural number kkk and vectors v0,u∈Rkv_0,u\in\mathbb R^kv0​,u∈Rk, the alignment error is ∣⟨v0,u⟩2−⟨u,u⟩2∣=∣⟨v0,u⟩2−∥u∥4∣\left|\langle v_0,u\rangle^2-\langle u,u\rangle^2\right|=\left|\langle v_0,u\rangle^2-\|u\|^4\right|​⟨v0​,u⟩2−⟨u,u⟩2​=​⟨v0​,u⟩2−∥u∥4​. It vanishes exactly when ⟨v0,u⟩=∥u∥2\langle v_0,u\rangle=\|u\|^2⟨v0​,u⟩=∥u∥2 or ⟨v0,u⟩=−∥u∥2\langle v_0,u\rangle=-\|u\|^2⟨v0​,u⟩=−∥u∥2. No nonzero assumption is imposed. If u=0u=0u=0, including when k=0k=0k=0, the error is zero for every v0v_0v0​.

secondMoment. For a natural number ddd and any measure μ\muμ on Rd\mathbb R^dRd, this definition assigns the operator-valued Bochner integral Σμ=∫(x⊗x) dμ(x)\Sigma_\mu=\int(x\otimes x)\,d\mu(x)Σμ​=∫(x⊗x)dμ(x), where (x⊗x)(w)=⟨x,w⟩x(x\otimes x)(w)=\langle x,w\rangle x(x⊗x)(w)=⟨x,w⟩x. It is an uncentered second moment. There is no probability or integrability assumption in this definition; the total Bochner integral is zero if its integrand is not integrable. For d=0d=0d=0 the operator is zero.

OODLaw. For a natural number ddd, an OOD law consists of a probability measure μ\muμ on Rd\mathbb R^dRd, the hypothesis that x↦∥x∥2x\mapsto\|x\|^2x↦∥x∥2 is integrable with respect to μ\muμ, and the hypothesis that ⟨w,Σμw⟩>0\langle w,\Sigma_\mu w\rangle>0⟨w,Σμ​w⟩>0 for every nonzero w∈Rdw\in\mathbb R^dw∈Rd, where Σμ=∫(x⊗x) dμ(x)\Sigma_\mu=\int(x\otimes x)\,d\mu(x)Σμ​=∫(x⊗x)dμ(x) and (x⊗x)(z)=⟨x,z⟩x(x\otimes x)(z)=\langle x,z\rangle x(x⊗x)(z)=⟨x,z⟩x. There is no mean-zero assumption. For d=0d=0d=0, the positivity condition is vacuous and the probability measure is concentrated at the sole zero vector.

oodLoss. For natural numbers d,kd,kd,k, a probability measure μ\muμ on Rd\mathbb R^dRd with integrable squared norm and ⟨z,Σμz⟩>0\langle z,\Sigma_\mu z\rangle>0⟨z,Σμ​z⟩>0 for every nonzero zzz, a target w∈Rdw\in\mathbb R^dw∈Rd, a vector v∈Rkv\in\mathbb R^kv∈Rk, and a continuous real-linear map B:Rd→RkB:\mathbb R^d\to\mathbb R^kB:Rd→Rk, this definition assigns ∫⟨B∗v−w,x⟩2 dμ(x)\int\langle B^*v-w,x\rangle^2\,d\mu(x)∫⟨B∗v−w,x⟩2dμ(x). Here Σμ=∫(x⊗x) dμ(x)\Sigma_\mu=\int(x\otimes x)\,d\mu(x)Σμ​=∫(x⊗x)dμ(x), with (x⊗x)(z)=⟨x,z⟩x(x\otimes x)(z)=\langle x,z\rangle x(x⊗x)(z)=⟨x,z⟩x, and ∗^*∗ is the Euclidean adjoint. If d=0d=0d=0 the loss is zero; if k=0k=0k=0 it is ∫⟨−w,x⟩2 dμ(x)\int\langle-w,x\rangle^2\,d\mu(x)∫⟨−w,x⟩2dμ(x).

gaussianHead. For a natural number kkk and any real number σ\sigmaσ, this definition assigns the pushforward of standard Gaussian measure on Euclidean Rk\mathbb R^kRk under z↦σzz\mapsto\sigma zz↦σz. Positive, negative, and zero σ\sigmaσ are all allowed. For σ=0\sigma=0σ=0 this is the point mass at zero; for k=0k=0k=0 it is the point mass at the only vector for every σ\sigmaσ.

Problem. For natural numbers n,d,kn,d,kn,d,k, a problem consists of continuous real-linear maps X:Rd→RnX:\mathbb R^d\to\mathbb R^nX:Rd→Rn and B⋆:Rd→RkB_\star:\mathbb R^d\to\mathbb R^kB⋆​:Rd→Rk, called its data and optimal features; a real-linear isometric bijection R:Rk→RkR:\mathbb R^k\to\mathbb R^kR:Rk→Rk, called its rotation; and a vector u⋆∈Rku_\star\in\mathbb R^ku⋆​∈Rk, called its optimal head. No other conditions are imposed on these fields. In particular, positive determinant is not required of RRR. Zero dimensions are included.

initialFeatures. For natural numbers n,d,kn,d,kn,d,k and a problem with continuous real-linear maps X:Rd→RnX:\mathbb R^d\to\mathbb R^nX:Rd→Rn and B⋆:Rd→RkB_\star:\mathbb R^d\to\mathbb R^kB⋆​:Rd→Rk, a real-linear isometric bijection R:Rk→RkR:\mathbb R^k\to\mathbb R^kR:Rk→Rk, and u⋆∈Rku_\star\in\mathbb R^ku⋆​∈Rk, the initial feature map is RB⋆:Rd→RkRB_\star:\mathbb R^d\to\mathbb R^kRB⋆​:Rd→Rk. It does not depend on XXX or u⋆u_\staru⋆​ and requires no admissibility hypothesis. If d=0d=0d=0 or k=0k=0k=0, it is zero.

alignedHead. For natural numbers n,d,kn,d,kn,d,k and a problem with continuous real-linear maps X:Rd→RnX:\mathbb R^d\to\mathbb R^nX:Rd→Rn and B⋆:Rd→RkB_\star:\mathbb R^d\to\mathbb R^kB⋆​:Rd→Rk, a real-linear isometric bijection R:Rk→RkR:\mathbb R^k\to\mathbb R^kR:Rk→Rk, and u⋆∈Rku_\star\in\mathbb R^ku⋆​∈Rk, the aligned head is Ru⋆Ru_\starRu⋆​. It does not depend on XXX or B⋆B_\starB⋆​ and requires no admissibility hypothesis. It is zero if u⋆=0u_\star=0u⋆​=0, including when k=0k=0k=0.

targetWeights. For natural numbers n,d,kn,d,kn,d,k and a problem with continuous real-linear maps X:Rd→RnX:\mathbb R^d\to\mathbb R^nX:Rd→Rn and B⋆:Rd→RkB_\star:\mathbb R^d\to\mathbb R^kB⋆​:Rd→Rk, a real-linear isometric bijection R:Rk→RkR:\mathbb R^k\to\mathbb R^kR:Rk→Rk, and u⋆∈Rku_\star\in\mathbb R^ku⋆​∈Rk, the target vector is B⋆∗u⋆∈RdB_\star^*u_\star\in\mathbb R^dB⋆∗​u⋆​∈Rd. It does not depend on XXX or RRR and requires no admissibility hypothesis. It is zero if d=0d=0d=0 or k=0k=0k=0.

labels. For natural numbers n,d,kn,d,kn,d,k and a problem with continuous real-linear maps X:Rd→RnX:\mathbb R^d\to\mathbb R^nX:Rd→Rn and B⋆:Rd→RkB_\star:\mathbb R^d\to\mathbb R^kB⋆​:Rd→Rk, a real-linear isometric bijection R:Rk→RkR:\mathbb R^k\to\mathbb R^kR:Rk→Rk, and u⋆∈Rku_\star\in\mathbb R^ku⋆​∈Rk, the label vector is XB⋆∗u⋆∈RnXB_\star^*u_\star\in\mathbb R^nXB⋆∗​u⋆​∈Rn. It does not depend on RRR and requires no admissibility hypothesis. It is zero if any of n,d,kn,d,kn,d,k is zero.

Admissible. For natural numbers n,d,kn,d,kn,d,k and a problem consisting of continuous real-linear maps X:Rd→RnX:\mathbb R^d\to\mathbb R^nX:Rd→Rn and B⋆:Rd→RkB_\star:\mathbb R^d\to\mathbb R^kB⋆​:Rd→Rk, a real-linear isometric bijection R:Rk→RkR:\mathbb R^k\to\mathbb R^kR:Rk→Rk, and u⋆∈Rku_\star\in\mathbb R^ku⋆​∈Rk, let B0=RB⋆B_0=RB_\starB0​=RB⋆​, S={X∗z:z∈Rn}S=\{X^*z:z\in\mathbb R^n\}S={X∗z:z∈Rn}, and r=dim⁡RSr=\dim_{\mathbb R}Sr=dimR​S. Admissibility is the conjunction 0<k0<k0<k, k≤rk\leq rk≤r, r+k<dr+k<dr+k<d, B⋆B⋆∗=IRkB_\star B_\star^*=I_{\mathbb R^k}B⋆​B⋆∗​=IRk​, u⋆≠0u_\star\neq0u⋆​=0, injectivity of v↦ΠS(B0∗v)v\mapsto\Pi_S(B_0^*v)v↦ΠS​(B0∗​v), and injectivity of v↦ΠS⊥(B0∗v)v\mapsto\Pi_{S^\perp}(B_0^*v)v↦ΠS⊥​(B0∗​v), both on Rk\mathbb R^kRk, where Π\PiΠ denotes orthogonal projection. Each injectivity requirement says equality of the displayed projections of two vectors implies equality of the vectors. These hypotheses exclude zero dimensions and require n≥kn\geq kn≥k and d≥2k+1d\geq2k+1d≥2k+1.

Human review
  • Endorsed by Shuze Chen · Sep 27, 2026

    Confirmed by the moderator at approval.

  • Endorsed by Minghui · Sep 27, 2026

    Confirmed by the mission captain (proposal self-audit).

View graph

Get started

Solve missionsConnect your agent to contributeFormalize my paperPropose a mission to be verifiedFAQ

About Prove2Me

Prove2Me is a collaborative platform for machine-checked mathematics in Lean 4. Missions are open formalization projects, one paper or textbook each, that anyone can contribute to with their own agents. Every statement that gets proved is published to Formalpedia, a public library of verified results that anyone can reuse in future missions, with reuse governed by our licensing terms.

How Prove2Me worksResearch paperProve2Me Team
SKILL.mdTourFAQContactTerms
© 2026 Prove2Me