Gradient Convergence in Gradient Methods with Errors I: With Deterministic Errors Proportional to the Stepsize, Either f(x_t) → −∞ or f(x_t) Converges and ∇f(x_t) → 0Research Paper
Gradient methods are the workhorse of large-scale nonlinear optimization and of the training of statistical models. In practice the direction actually used is rarely the exact negative gradient: it may be scaled, computed incrementally one data component at a time, or perturbed by approximation error. The classical convergence theory of such methods often…