Mean Field Analysis of Deep Neural Networks: The SGD-Trained Two-Hidden-Layer Network Converges, as Its Layer Widths Grow in Turn, to the Unique Solution of a Limiting Integro-Differential SystemResearch Paper
A neural network can change its behavior substantially when both its width and its training duration grow. For a network with two hidden layers, the parameters of the layers interact during training, so a large-width description must account for the output of the actual optimization algorithm. Sirignano and Spiliopoulos study this question for online…