Proposition 1 — Gaussian Process Limit at Initialization

For every positive pair of integers $d,q$, every function $\sigma:\mathbb R\to\mathbb R$, every nonnegative real $K$ satisfying $|\sigma(s)-\sigma(t)|\le K|s-t|$ for all real $s,t$, every real $\beta>0$, every $h,N\in\mathbb N$, and every family $X_0,\ldots,X_{N-1}\in\mathbb R^d$, the joint initialized output vector defined below is almost everywhere measurable at every width tuple and converges in distribution to the stated Gaussian-constructor law. A width tuple is $w=(w_0,\ldots,w_{h-1})\in\mathbb N^h$; put $n_0=d$, $n_\ell=w_{\ell-1}+1$ for $1\le\ell\le h$, and $n_{h+1}=q$. For each tuple independently specify its probability space as all real parameters $W^\ell_{ja}$ and $b^\ell_j$, with $0\le\ell\le h$, $0\le j<n_{\ell+1}$, and $0\le a<n_\ell$, endowed with the finite product law making every weight and every bias independent with distribution $\mathcal N(0,1)$. For an input $x$, define $z^0(x)=x$, $z^1_j(x)=d^{-1/2}\sum_{a=0}^{d-1}W^0_{ja}x_a+\beta b^0_j$, and $z^{\ell+1}_j(x)=n_\ell^{-1/2}\sum_{a=0}^{n_\ell-1}W^\ell_{ja}\sigma(z^\ell_a(x))+\beta b^\ell_j$ for $1\le\ell\le h$. Write $f_k(\theta,x)=z^{h+1}_k(x)$, so the output is the final preactivation; the random vector is $(f_k(\theta,X_i))_{0\le i<N,\ 0\le k<q}\in\mathbb R^{Nq}$, using the same parameter sample for all inputs and all outputs at a fixed width tuple. Define $C_0(x,y)=d^{-1}\sum_{a=0}^{d-1}x_a y_a+\beta^2$ and $C_{r+1}(x,y)=\int_{\mathbb R^2}\sigma(u)\sigma(v)\,dG(S_r(x,y))(u,v)+\beta^2$, with $S_r(x,y)=\begin{pmatrix}C_r(x,x)&C_r(x,y)\\C_r(y,x)&C_r(y,y)\end{pmatrix}$. For a finite real square matrix $S$, $G(S)$ is the distribution of $S^{1/2}Z$ for a standard Gaussian vector $Z$ when $S$ is symmetric positive semidefinite, and is the point mass at zero otherwise; the integral is zero under the total-integral convention if its integrand is nonintegrable. The target law on $\mathbb R^{Nq}$ is $G(A)$, where $A_{(i,k),(j,k')}=\mathbf1_{\{k=k'\}}C_h(X_i,X_j)$; the limit random variable is the identity map on this probability space and is almost everywhere measurable. Convergence means weak convergence of the output laws along the following nested-tail filter: when $h\ge1$, a property $P(w)$ holds eventually precisely when $\exists M_{h-1}\ \forall w_{h-1}\ge M_{h-1}\ \exists M_{h-2}\ \forall w_{h-2}\ge M_{h-2}\ \cdots\ \exists M_0\ \forall w_0\ge M_0,\ P(w)$, with the earlier-coordinate thresholds allowed to depend on the already fixed later coordinates; equivalently, the required approximation holds in that nested-tail sense for every bounded continuous test function and every positive accuracy. Thus the first hidden width occurs in the innermost tail and the last hidden width in the outermost tail; no common growth rate or coupling between different width probability spaces is specified. The claim holds jointly for every fixed finite input family and all $q$ output coordinates. It includes $K=0$, repeated or zero inputs, and $N=0$, in which case the observed vector has no coordinates. For $h=0$ there is one empty width tuple and the network is the single affine layer above; the filter is concentrated at that tuple, so the convergence assertion requires its law to equal the target law exactly. No differentiability assumption on $\sigma$ is imposed; input and output dimensions and $\beta$ are strictly positive, and every hidden width is at least one.