MLP: second-layer connected four-point correlator and the four-point vertex (eq. 4.43)
ProvedDeepLearningTheory.mlp_second_layer_connected_four_pointGenesis of non-Gaussianity. Consider an MLP at initialization (eqs. 4.3–4.4) with a measurable activation of polynomial growth and . For inputs (not necessarily distinct) and second-layer neurons , the connected four-point correlator of the second-layer preactivations,
(with ), equals
where the four-point vertex is
The second-layer distribution is therefore non-Gaussian at finite width, with non-Gaussianity suppressed by .
Formalization Note All Gaussian averages are taken with the first-layer metric of the four inputs , which may be singular.
import Mathlib import Definitions.Def_DLT_DeepLinearNetwork import Definitions.Def_DLT_MLPInit open MeasureTheory ProbabilityTheory open scoped NNReal
namespace DeepLearningTheory
theorem mlp_second_layer_connected_four_point {Ω : Type*} [MeasurableSpace Ω] (P : Measure Ω)
[IsProbabilityMeasure P] (n : ℕ → ℕ) (Cb CW : ℕ → ℝ≥0) (σ : ℝ → ℝ)
(b : Ω → ℕ → ℕ → ℝ) (W : Ω → ℕ → ℕ → ℕ → ℝ) (hinit : IsMLPInit P n Cb CW b W)
(hσm : Measurable σ) (hσ : HasPolyGrowth σ) (hn₁ : 0 < n 1) (x : Fin 4 → ℕ → ℝ) (i : Fin 4 → ℕ) (hi : ∀ a, i a < n 2) :
(∫ ω, mlpPreact n σ (b ω) (W ω) (x 0) 2 (i 0) * mlpPreact n σ (b ω) (W ω) (x 1) 2 (i 1) *
mlpPreact n σ (b ω) (W ω) (x 2) 2 (i 2) * mlpPreact n σ (b ω) (W ω) (x 3) 2 (i 3) ∂P)
- (∫ ω, mlpPreact n σ (b ω) (W ω) (x 0) 2 (i 0) * mlpPreact n σ (b ω) (W ω) (x 1) 2 (i 1) ∂P) * (∫ ω, mlpPreact n σ (b ω) (W ω) (x 2) 2 (i 2) * mlpPreact n σ (b ω) (W ω) (x 3) 2 (i 3) ∂P)
- (∫ ω, mlpPreact n σ (b ω) (W ω) (x 0) 2 (i 0) * mlpPreact n σ (b ω) (W ω) (x 2) 2 (i 2) ∂P) * (∫ ω, mlpPreact n σ (b ω) (W ω) (x 1) 2 (i 1) * mlpPreact n σ (b ω) (W ω) (x 3) 2 (i 3) ∂P)
- (∫ ω, mlpPreact n σ (b ω) (W ω) (x 0) 2 (i 0) * mlpPreact n σ (b ω) (W ω) (x 3) 2 (i 3) ∂P) * (∫ ω, mlpPreact n σ (b ω) (W ω) (x 1) 2 (i 1) * mlpPreact n σ (b ω) (W ω) (x 2) 2 (i 2) ∂P)
= (1 / (n 1 : ℝ)) *
(kron (i 0) (i 1) * kron (i 2) (i 3) * ((CW 2 : ℝ) ^ 2 *
(gaussAvg (firstLayerMetric (Cb 1) (CW 1) (n 0) x)
(fun u => σ (u 0) * σ (u 1) * σ (u 2) * σ (u 3))
- gaussAvg (firstLayerMetric (Cb 1) (CW 1) (n 0) x) (fun u => σ (u 0) * σ (u 1)) *
gaussAvg (firstLayerMetric (Cb 1) (CW 1) (n 0) x) (fun u => σ (u 2) * σ (u 3))))
+ kron (i 0) (i 2) * kron (i 1) (i 3) * ((CW 2 : ℝ) ^ 2 *
(gaussAvg (firstLayerMetric (Cb 1) (CW 1) (n 0) x)
(fun u => σ (u 0) * σ (u 1) * σ (u 2) * σ (u 3))
- gaussAvg (firstLayerMetric (Cb 1) (CW 1) (n 0) x) (fun u => σ (u 0) * σ (u 2)) *
gaussAvg (firstLayerMetric (Cb 1) (CW 1) (n 0) x) (fun u => σ (u 1) * σ (u 3))))
+ kron (i 0) (i 3) * kron (i 1) (i 2) * ((CW 2 : ℝ) ^ 2 *
(gaussAvg (firstLayerMetric (Cb 1) (CW 1) (n 0) x)
(fun u => σ (u 0) * σ (u 1) * σ (u 2) * σ (u 3))
- gaussAvg (firstLayerMetric (Cb 1) (CW 1) (n 0) x) (fun u => σ (u 0) * σ (u 3)) *
gaussAvg (firstLayerMetric (Cb 1) (CW 1) (n 0) x) (fun u => σ (u 1) * σ (u 2))))) := by sorry
end DeepLearningTheoryRead-back
What the Lean code literally says, in plain math · Aristotle (Harmonic) - non-blind, same agent as the drafter
Non-blind read-back. This read-back was written by the same agent that drafted the Lean statement, with full knowledge of the source text and of the intended meaning. It is not independent testimony and must not be mistaken for a blind audit; reviewers should compare the Lean code against the source themselves.
Let be a probability space, widths, , any function, and random bias/weight fields satisfying IsMLPInit (all biases and weights with , , mutually independent; , ). Write for the MLP preactivation at and for the first-layer metric of the listed inputs. Assume moreover that is measurable and has polynomial growth (). Assume . Let be four inputs with first-layer metric , and indices with each . Write and . Then
where . Here is the integral of against Mathlib's centered multivariate Gaussian with covariance matrix (the inputs may repeat, so may be singular).
Confirmed by the mission captain (proposal self-audit).
Confirmed by the moderator at approval.