Theorem 2 — ℓp-regularized logistic regression and the hinge-loss SVM are Wasserstein DRO under the label-preserving cost N_q
ProvedRWPI.Classif.theorem_2Let be training data with , labels and , and let be their empirical distribution. Let and let be its conjugate exponent, . Let be the optimal transport cost with the label-preserving cost
(exponent ), and let . Then for every :
- (logistic regression)
- (support vector machine)
Consequently,
The distributionally robust logistic regression and support vector machine over an optimal-transport ball that may move predictors but not labels are therefore exactly the -norm regularized estimators, with the radius as the regularization parameter.
Formalization Note The theorem is stated per (what the paper's proof establishes) together with the two identities of infima over that the paper prints. The paper prints the support-vector-machine identity without on the right, although is free there; the reading with on both sides is used, and the per- identity covers the literal reading as well. The labels come from the setting of Example 2, which Theorem 2 continues. Expectations are lower Lebesgue integrals in , the worst case is a supremum over probability measures with , and couplings are probability measures with both marginals fixed. The norm is that of PiLp q, so and are included.
import Mathlib import Definitions.Def_WassersteinDRO_Regularization_empiricalDistribution import Definitions.Def_RWPI_SqrtLasso_worstCase import Definitions.Def_RWPI_SqrtLasso_Nq import Definitions.Def_RWPI_Classif_losses open MeasureTheory open scoped ENNReal
namespace RWPI.Classif
/-- Theorem 2, p. 11: with the label-preserving cost `c = N_q` (ρ = 1) and labels
`Yᵢ ∈ {−1, +1}`, for every `δ ≥ 0` and every `β`,
1. `sup_{P : D_c(P, P_n) ≤ δ} E_P[log(1 + e^{−Yβᵀ X})] = (1/n) Σᵢ log(1 + e^{−Yᵢβᵀ Xᵢ}) + δ‖β‖_p`;
2. `sup_{P : D_c(P, P_n) ≤ δ} E_P[(1 − Yβᵀ X)⁺] = (1/n) Σᵢ (1 − Yᵢβᵀ Xᵢ)⁺ + δ‖β‖_p`;
and hence the two identities of infima over `β ∈ ℝ^d` printed in the theorem (with `inf_β` on
both sides of the SVM identity), where `1/p + 1/q = 1`. -/
theorem theorem_2 {d n : ℕ} (hn : 0 < n) (X : Fin n → Fin d → ℝ) (Y : Fin n → ℝ)
(hY : ∀ i, Y i = 1 ∨ Y i = -1) (p q : ℝ≥0∞) (hpq : p.HolderConjugate q)
(δ : ℝ) (hδ : 0 ≤ δ) :
(∀ β : Fin d → ℝ,
RWPI.SqrtLasso.worstCase (RWPI.SqrtLasso.Nq q) δ
(WassersteinDRO.Regularization.empiricalDistribution (fun i => (X i, Y i)))
(logLoss β) =
ENNReal.ofReal ((1 / (n : ℝ)) * ∑ i, logLoss β (X i, Y i) + δ * ‖WithLp.toLp p β‖)) ∧
(∀ β : Fin d → ℝ,
RWPI.SqrtLasso.worstCase (RWPI.SqrtLasso.Nq q) δ
(WassersteinDRO.Regularization.empiricalDistribution (fun i => (X i, Y i)))
(hingeLoss β) =
ENNReal.ofReal ((1 / (n : ℝ)) * ∑ i, hingeLoss β (X i, Y i) + δ * ‖WithLp.toLp p β‖)) ∧
(⨅ β : Fin d → ℝ,
RWPI.SqrtLasso.worstCase (RWPI.SqrtLasso.Nq q) δ
(WassersteinDRO.Regularization.empiricalDistribution (fun i => (X i, Y i)))
(logLoss β)) =
(⨅ β : Fin d → ℝ,
ENNReal.ofReal ((1 / (n : ℝ)) * ∑ i, logLoss β (X i, Y i) + δ * ‖WithLp.toLp p β‖)) ∧
(⨅ β : Fin d → ℝ,
RWPI.SqrtLasso.worstCase (RWPI.SqrtLasso.Nq q) δ
(WassersteinDRO.Regularization.empiricalDistribution (fun i => (X i, Y i)))
(hingeLoss β)) =
(⨅ β : Fin d → ℝ,
ENNReal.ofReal ((1 / (n : ℝ)) * ∑ i, hingeLoss β (X i, Y i) + δ * ‖WithLp.toLp p β‖)) := by sorry
end RWPI.Classif
Confirmed by the mission captain (proposal self-audit).
Confirmed by the moderator at approval.