A Singular Value Thresholding Algorithm for Matrix Completion 2: Convergence of the SVT Iteration under General Convex ConstraintsResearch Paper
Motivation
Singular value thresholding (SVT) is a first-order method introduced by Cai, Candès and Shen (SIAM J. Optim. 20 (2010)) for recovering a low-rank matrix from incomplete or indirect information. Its basic form, for matrix completion, alternates a soft-thresholding of singular values with a gradient step on a dual variable, and needs only one sparse singular value decomposition per iteration. That is what made nuclear-norm heuristics usable on matrices with tens of thousands of rows and columns, where interior-point methods for the equivalent semidefinite program do not fit in memory.
Matrix completion is only one constraint set. In applications the data are noisy linear measurements , and the constraint takes the form of componentwise error bounds or norm balls around the data (§3.3 of the paper). Section 3.2 of the paper extends the method to a general finite family of convex constraints, and §4.2 proves that the extended iteration converges. This mission formalizes that extension and its convergence theorem, Theorem 4.4.
Setting
Let be natural numbers and the real matrices, with the Frobenius inner product and norm . The nuclear norm is the sum of the singular values of . For a fixed the objective is
A matrix is a subgradient of a function at , written , if for all .
Let be convex and put . On , and is the Euclidean norm. The constrained problem is
with Lagrangian for . A pair with is primal-dual optimal if it is a saddle point:
The paper's standing assumption "strong duality holds" is the existence of such a pair.
The iteration (3.5) starts from and, for step sizes , sets for
where has entries . It is Uzawa's method for (3.4): an exact minimization in the primal variable followed by a projected ascent step on the dual. When is affine, the minimization is a singular value thresholding step, which gives the algorithm its name.
The analysis of §4.2 assumes is Lipschitz in the sense
for a constant .
Formalization targets
Goal: Theorem 4.4 (p. 1969)
If and strong duality holds, then the sequence of (3.5) converges to the unique solution of (3.4):
Milestones, in the order the proof uses them
- Lemma 4.1 (p. 1968): for , .
- Lemma 4.3 (p. 1969): for a primal-dual optimal pair and each , .
- Eq. (4.4) (p. 1969): there are and with and for all .
- Eq. (4.5) (p. 1969): .
- Contraction step (p. 1969): .
- Eq. (4.6) (p. 1970): if for , then .
Significance
The result. Theorem 4.4 is the convergence guarantee for SVT beyond matrix completion. The componentwise error bounds of (3.8), whose SVT iteration is (3.9), are finitely many affine constraints and fall under it directly, as does any finite family of Lipschitz convex constraints, for instance a Frobenius-norm ball around the data. The conic variants of §3.3 ((3.11)–(3.13)) project the dual variable onto a cone rather than onto the nonnegative orthant and are not covered by the theorem as stated. Together with Theorem 3.1 of the same paper, which says that the solution of (3.4) tends to the minimum-nuclear-norm solution as , it justifies using SVT as a solver for nuclear-norm minimization under general convex constraints.
Formalizing it. The theorem is proved in the paper, with two steps delegated to the literature: Lemma 4.3 cites [31], and the concluding step reads "the conclusion is as before". Its proof is short but relies on convex-analytic facts that are standard on paper and missing, in this form, from Mathlib: subgradients of the nuclear norm, the subdifferential sum rule for finite convex functions, and nonexpansiveness of the projection onto the nonnegative orthant. No machine-checked proof of this theorem or of Uzawa-type convergence for nuclear-norm objectives is known to exist. The mission produces a complete, checked version of the argument, including the omitted closing step.
Difficulty
The obvious approach is to view (3.5) as projected gradient ascent on the dual function and quote the standard convergence theorem for gradient methods with Lipschitz gradients. That does not apply directly: for general convex the dual function need not be differentiable, is only a supergradient, and the Lipschitz hypothesis (4.2) is on , not on a dual gradient. The proof instead works with the primal-dual pair: it needs first-order optimality conditions (4.4), which require a subdifferential sum rule for with nonsmooth , and it needs the strong monotonicity of (Lemma 4.1), which depends on the description of subgradients of the nuclear norm. A second subtlety is that the theorem asserts convergence of the whole primal sequence to the unique solution, not to some solution along a subsequence, while nothing is claimed about convergence of the dual sequence.
Formalization scope
Matrices are Matrix (Fin n₁) (Fin n₂) ℝ, vectors in are Fin m → ℝ, and convergence of matrices is in Mathlib's product topology, which coincides with the Frobenius topology. The nuclear norm is the sum of Mathlib's LinearMap.singularValues of the matrix viewed as a map between Euclidean spaces. Each is a real-valued function with ConvexOn ℝ Set.univ. The iteration is a predicate on sequences indexed by ℕ: the paper's step produces X (k+1) and y (k+1) from y k with step size δ (k+1), and y 0 = 0. is required to minimize ; for and convex this minimizer exists and is unique, so the predicate is satisfiable and determines the sequence. The step-size condition is stated as for with and , which avoids the division (evaluated as in Lean when ); for it requires only bounded steps, matching the convention . Strong duality is the hypothesis that a saddle point exists; Slater's condition is not assumed. The paper's standing assumptions (, convex , and (4.2) where enters) appear as explicit hypotheses in every statement.
A formalization that assumes convergence or boundedness of the dual iterates, replaces the primal minimization by a closed-form thresholding step (valid only for affine ), or states only subsequential convergence would not be this theorem; each of these is excluded by the statements above.
A complete development needs: subgradients of the nuclear norm and strong monotonicity of ; existence and characterization of minimizers of strongly convex continuous functions on a finite-dimensional space; the subdifferential sum rule for finite convex functions; complementary slackness from the saddle-point inequalities; and nonexpansiveness of the entrywise positive part. These are reusable beyond this mission, especially for other Uzawa and augmented Lagrangian analyses. Contributions of any of these pieces as separate lemmas are welcome.
Selected references
- J.-F. Cai, E. J. Candès, Z. Shen, A Singular Value Thresholding Algorithm for Matrix Completion, SIAM J. Optim. 20(4):1956–1982, 2010. https://doi.org/10.1137/080738970
- E. J. Candès, B. Recht, Exact Matrix Completion via Convex Optimization, Found. Comput. Math. 9:717–772, 2009. https://doi.org/10.1007/s10208-009-9045-5
- K. J. Arrow, L. Hurwicz, H. Uzawa, Studies in Linear and Nonlinear Programming, Stanford University Press, 1958.
- S. Boyd, L. Vandenberghe, Convex Optimization, Cambridge University Press, 2004. https://doi.org/10.1017/CBO9780511804441