Suboptimal Local Minima Exist for Wide Neural Networks with Smooth Activations: For Generic Inputs and Any Depth and Width, Some Output Data Gives the Squared Loss a Sub-Optimal Local MinimumResearch Paper
Training a neural network means minimizing a non-convex loss over its weights. Local search methods can only certify local optimality, so a recurring question in the theory of neural network optimization is whether every local minimum of the training loss is a global minimum. A local minimum that is not global is called a sub-optimal (or bad) local minimum.