Distributed Stochastic Optimization with Large Delays: Under Polynomially Growing Delays, Asynchronous SGD with Tuned Step-Sizes Drives the Mean-Square Gradient Norm to ZeroResearch Paper
Large-scale learning problems, such as empirical risk minimization over millions of samples, are solved in practice by stochastic gradient descent (SGD) run on many processors at once. In a master–worker system, or in a multi-processor system with shared memory, a worker reads the current parameter, computes a stochastic gradient and writes its update back.…