Motivation
A feedforward neural network is a directed acyclic graph of neurons, each computing a fixed scalar activation of a weighted sum of its inputs; fixing the graph and the activation and letting the weights vary gives a hypothesis class. Chapter 20 of Shalev-Shwartz and Ben-David, Understanding Machine Learning: From Theory to Algorithms (doi:10.1017/CBO9781107298019), studies these classes through the book's three lenses. Approximation: every Boolean function is implemented by a network of depth 2 (Claim 20.1), but only at exponential size (Theorem 20.2), while a sign neuron implements conjunctions and disjunctions (Lemma 20.4), the bridge to Boolean circuits and hence to everything computable in bounded time. Estimation: the VC dimension of the class of sign networks over a graph with ∣E∣ edges is O(∣E∣log∣E∣) (Theorem 20.6), so the sample complexity is governed by the number of weights. Optimization: training is NP-hard even for tiny networks, and the practical answer is SGD with the gradient computed by backpropagation, whose correctness the chapter derives from the chain rule.
Setting
A layered graph has layers V0,…,VT, every edge joining Vt−1 to Vt; V0 holds the n inputs and a constant neuron outputting 1. With weights w:E→R and an activation σ, the outputs are computed layer by layer, at+1,i=∑j:(vt,j,vt+1,i)∈Ewt,i,jot,j and ot+1,i=σ(at+1,i). The class HV,E,σ={hV,E,σ,w:w:E→R} (20.1); for binary classification the output layer is a single neuron and σ is the sign function, so HV,E,sign is a set of {±1}-valued predictors on Rn. The size of the network is ∣V∣, its depth T. The growth function τH(m)=max∣C∣≤m∣HC∣ extends to classes with any finite codomain (p. 275), and the proof of Theorem 20.6 uses two of its properties, stated as Exercises 3 and 4: the growth function of a product class is at most the product of the growth functions, and likewise for a composition class. For backpropagation the activation is any differentiable σ and the loss is 21∥oT−y∥2; the backward pass sets δT=oT−y and δt=δt+1diag(σ′(at+1))Wt.
Formalization targets
Goal: Theorem 20.6
The VC dimension of HV,E,sign is O(∣E∣log∣E∣). Explicitly, for a layered graph of depth at least 1 with a single output neuron,
VCdim(HV,E,sign)≤2∣E∣log2(16∣E∣),
stated as: every m≤VCdim satisfies this bound (so the VC dimension is finite).
Milestones
Claim 20.1 (the depth-2 graph with ∣V1∣=2n+1 whose sign class contains every function {±1}n→{±1}); Theorem 20.2 (every sign network implementing all functions {0,1}n→{0,1} has 2n/3≤2∣V∣); Lemma 20.4 (conjunction and disjunction as sign neurons); Exercise 4 (growth function of a composition); the correctness of backpropagation (§20.6: the partial derivative for the edge (vt,j,vt+1,i) is δt+1,iσ′(at+1,i)ot,j). Further items: Exercise 3 (growth function of a product) and the intermediate bound τH(m)≤(em)∣E∣ of the proof of Theorem 20.6.
Significance
Theorem 20.6 is the reason networks are learnable at all in the book's sense: by the fundamental theorem, a class with finite VC dimension is agnostic PAC learnable with sample complexity linear in that dimension, and here the dimension is essentially the number of tunable parameters. The proof technique, due to Kakade and Tewari's lecture notes, is a composition-and-product argument on growth functions that applies to any layered class of threshold units and is reusable well beyond this chapter. Theorem 20.2 is the matching negative fact on expressive power, and it is a corollary of the same bound: a class that shatters 2n points needs Ω(2n) edges. Backpropagation's correctness is the one theorem about training the chapter can offer, given the hardness results, and it is the algorithm every practitioner runs.
Difficulty
Claim 20.1 and Lemma 20.4 are explicit constructions: the neuron gi(x)=sign(⟨x,ui⟩−n+1) detects x=ui because ⟨x,ui⟩≤n−2 otherwise, and the output neuron takes the disjunction; formally one must build the weight function and evaluate the forward pass on the 2n inputs. Exercises 3 and 4 are counting: a restricted product is determined by its two restricted factors, and a restricted composition f2∘f1 on C is determined by f1∣C and f2∣f1(C), with ∣f1(C)∣≤∣C∣. Theorem 20.6 then needs: the class of one neuron is the class of homogenous halfspaces on its dt,i incoming coordinates, of VC dimension at most dt,i (Mission VI), Sauer's lemma in the form τ(m)≤(em)d for every m≥1 (Mission IV; for m≤d use 2m≤(em)m), the layer class as a product and the network as a composition of layer classes, and finally the arithmetic 2m≤(em)∣E∣⇒m≤2∣E∣log2(16∣E∣), which replaces the book's appeal to Lemma A.2 (for m≥8∣E∣ one has lnm≤2∣E∣mln2). Theorem 20.2 follows from the goal with ∣E∣≤∣V∣2. Backpropagation is a chain-rule computation in a single real variable: the loss as a function of one weight is a composition of finitely many differentiable maps, and the derivative unwinds to the backward recursion; the formal effort is in the induction along layers with the natural-number indexing of the model.
Formalization scope
Layers and neurons are indexed by natural numbers: a LayeredGraph records the depth, the layer widths and, for each t<T, the finite set of edges (vt,j,vt+1,i) as pairs (i,j) within the layer widths. Weights are functions on all index triples, and only those on edges are used, so the class is the image of all weight functions, as in (20.1). The forward computation netOutput is a recursion on the layer index; netInput is at+1,i. The sign activation returns ±1 with sign(0)=−1, the book's convention elsewhere, and a neuron with no incoming edges outputs σ(0) (p. 270). The binary class signNetClass n G is Bool-valued, true iff the output neuron's input is positive, and is stated for graphs of depth at least 1 (for depth 0 the edges out of the input layer would be used but not counted in ∣E∣). Growth functions with finite codomain are growthY, an sSup over restriction sizes, well defined because the codomains are finite; the Bool case is Mission IV's growth, and VC dimension and shattering are Mission IV's. Theorem 20.6 and Theorem 20.2 are given with explicit constants derived from the proof, since O(⋅) statements have no formal content; the drafter verified max{m:2m≤(em)∣E∣}≤2∣E∣log2(16∣E∣) numerically for ∣E∣ up to 3000 and at 104,…,107, and 2n≤2∣V∣2log2(16∣V∣2)≤8∣V∣3. Backpropagation is stated for an arbitrary layered graph (phantom edges have weight 0, p. 279), any differentiable activation, and one edge at a time as a HasDerivAt of the loss in that weight; δt is defined by recursion on T−t. The book's layer indices in (20.3) are shifted by one in the statement.
Not stated: Theorem 20.3 (Turing machines), Theorem 20.5 and Exercise 1 (sigmoid approximation, which needs a convention for outputs in [−1,1] that the chapter leaves open), Theorem 20.7 and Exercise 6 (NP-hardness), Exercise 5 (the Ω(∣E∣2) sigmoid lower bound, which assumes an exact threshold), the sigmoid half of Theorem 20.2, and the SGD pseudocode of §20.6, which is a heuristic without a stated guarantee.
Selected references
- S. Shalev-Shwartz, S. Ben-David, Understanding Machine Learning: From Theory to Algorithms, Cambridge University Press, 2014, Chapter 20. doi:10.1017/CBO9781107298019
- M. Anthony, P. L. Bartlett, Neural Network Learning: Theoretical Foundations, Cambridge University Press, 1999. doi:10.1017/CBO9780511624216
- D. E. Rumelhart, G. E. Hinton, R. J. Williams, Learning representations by back-propagating errors, Nature 323, 1986. doi:10.1038/323533a0
- I. Parberry, Circuit Complexity and Neural Networks, MIT Press, 1994.
- E. B. Baum, D. Haussler, What size net gives valid generalization?, Neural Computation 1(1), 1989. doi:10.1162/neco.1989.1.1.151