Last-Iterate Stationarity of Muon with Finitely Many Newton-Schulz IterationsResearch Paper
Muon is an optimizer for matrix-shaped parameters in neural-network training. At each step it forms a momentum average of stochastic gradients and replaces that matrix by an approximately orthogonalized version before taking a step. In practice the orthogonalization is not the exact polar factor. It is a fixed small number (commonly ) of quintic Newton–…