Theorem 1 — Deterministic Infinite-Width Neural Tangent Kernel

For every positive pair of integers $d,q$, every function $\sigma:\mathbb R\to\mathbb R$, every nonnegative real $K$ satisfying $|\sigma(s)-\sigma(t)|\le K|s-t|$ for all real $s,t$, every real $\beta>0$, every $h,N\in\mathbb N$, every family $X_0,\ldots,X_{N-1}\in\mathbb R^d$, and every real $\varepsilon>0$, the following bad-set measure tends to zero. For each $w=(w_0,\ldots,w_{h-1})\in\mathbb N^h$, let $n_0=d$, $n_\ell=w_{\ell-1}+1$ for $1\le\ell\le h$, and $n_{h+1}=q$. The parameter vector $\theta$ consists of weights $W^\ell_{ja}$ and biases $b^\ell_j$ for $0\le\ell\le h$, $0\le j<n_{\ell+1}$, and $0\le a<n_\ell$, with the product probability law $\mu_w$ under which all these real coordinates are independent $\mathcal N(0,1)$ variables. Define $z^0(x)=x$, $z^1_j(x)=d^{-1/2}\sum_{a=0}^{d-1}W^0_{ja}x_a+\beta b^0_j$, $z^{\ell+1}_j(x)=n_\ell^{-1/2}\sum_{a=0}^{n_\ell-1}W^\ell_{ja}\sigma(z^\ell_a(x))+\beta b^\ell_j$ for $1\le\ell\le h$, and $f_k(\theta,x)=z^{h+1}_k(x)$. For each parameter coordinate $p$, let $D_p f_k(\theta,x)$ be the full Fréchet derivative at $\theta$ of the scalar function $\eta\mapsto f_k(\eta,x)$, applied to the parameter-coordinate unit vector $e_p$; if that scalar function is not Fréchet differentiable at $\theta$, the whole derivative is defined to be zero, so all these $D_p$ values are zero there. Set $T_w(\theta;x,y;k,k')=\sum_p D_p f_k(\theta,x)D_p f_{k'}(\theta,y)$, where the sum includes every weight and every bias in every layer. Define $C_0(x,y)=d^{-1}\sum_{a=0}^{d-1}x_a y_a+\beta^2$ and $C_{r+1}(x,y)=\int_{\mathbb R^2}\sigma(u)\sigma(v)\,dG(S_r(x,y))(u,v)+\beta^2$, where $S_r(x,y)=\begin{pmatrix}C_r(x,x)&C_r(x,y)\\C_r(y,x)&C_r(y,y)\end{pmatrix}$. For a finite real square matrix $S$, $G(S)$ is the distribution of $S^{1/2}Z$ for a standard Gaussian vector $Z$ when $S$ is symmetric positive semidefinite and is the point mass at zero otherwise. Let $\sigma'(t)$ mean the ordinary real derivative where it exists and zero where it does not, and define $\Theta_0(x,y)=C_0(x,y)$ and $\Theta_{r+1}(x,y)=\Theta_r(x,y)\int_{\mathbb R^2}\sigma'(u)\sigma'(v)\,dG(S_r(x,y))(u,v)+C_{r+1}(x,y)$. Every integral here uses the total-integral convention, returning zero for a nonintegrable integrand. The quantity asserted to converge to zero is $\mu_w\{\theta:\exists\,0\le i,j<N,\ \exists\,0\le k,k'<q,\ \varepsilon<|T_w(\theta;X_i,X_j;k,k')-\mathbf1_{\{k=k'\}}\Theta_h(X_i,X_j)|\}$, considered in $[0,\infty]$ with its usual topology. The measure is evaluated on this set as defined; the proposition contains no separate assertion that the bad set is measurable. The convergence uses the nested-tail filter: for $h\ge1$, a property $P(w)$ holds eventually precisely when $\exists M_{h-1}\ \forall w_{h-1}\ge M_{h-1}\ \exists M_{h-2}\ \forall w_{h-2}\ge M_{h-2}\ \cdots\ \exists M_0\ \forall w_0\ge M_0,\ P(w)$, with thresholds permitted to depend on later width coordinates already fixed. In particular, for every real $\delta>0$, the displayed bad-set measure is less than $\delta$ eventually in this sense; the first hidden width has the innermost tail and the last has the outermost tail. This is simultaneous control of every input pair in each chosen finite family and every pair of output coordinates, including distinct outputs; all entries use the same parameter sample at a fixed width tuple, with no coupling between different tuples specified. The assumptions allow $K=0$, nonsmooth Lipschitz activations with the derivative conventions just stated, repeated or zero inputs, $N=0$, and $h=0$. When $N=0$ the bad set is empty. When $h=0$ the width filter is concentrated on its single empty tuple, so the convergence assertion requires the bad-set measure of the single affine network to be exactly zero. No positive hidden-layer count, input normalization, or differentiability hypothesis is imposed; $d,q,\beta$ are strictly positive and each hidden width is at least one.