Algorithm 6, step 2(a), p. 356 — the population weighted least-squares fit of z_j with weights p_j(1−p_j) is E(y*_j − p_j|x)/(p_j(1 − p_j))
ProvedAddLogReg.Multiclass.algorithm_6_fitlogitboostmulticlass-classificationp2o-batch-pfp2ap2o-gran-per-chapterp2o-plan-paperp2o-v1
Let , let be a probability measure on and the current fit with model probabilities given by (40). In Algorithm 6, step 2(a), the working response and weight of class are
and the population version of the weighted least-squares fit of on is the weighted conditional mean . For every and ,
Since the weight depends on only, the fitted of Algorithm 6 coincides with the diagonal quasi-Newton step of the Derivation of Result 6.
Formalization Note "Population version" replaces the weighted least-squares regression by the weighted conditional expectation, as for Algorithm 3. Conditional expectations are integrals against ν.condKernel x.
Preamble
import Mathlib import Definitions.Def_AddLogReg_Multiclass_Setting open MeasureTheory ProbabilityTheory ENNReal
Formal statement
namespace AddLogReg.Multiclass
theorem algorithm_6_fit {J : ℕ} [NeZero J] (hJ : 2 ≤ J) {X : Type*} [MeasurableSpace X]
(ν : Measure (X × Fin J)) [IsProbabilityMeasure ν] (F : X → Fin J → ℝ) (x : X) (j : Fin J) :
wCondExpBy ν (fun x' _ => classWeight F x' j) (fun x' y => workingResponse F x' y j) x =
(∫ y, (ystar y j - softmax (F x) j) ∂(ν.condKernel x)) /
(softmax (F x) j * (1 - softmax (F x) j)) := by sorry
end AddLogReg.Multiclass
Source
Friedman, Hastie and Tibshirani, Additive logistic regression: a statistical view of boosting, Ann. Statist. 28 (2000), p. 356, Algorithm 6, step 2(a); p. 347 (weighted conditional expectation)
Human review
Confirmed by the mission captain (proposal self-audit).
Confirmed by the moderator at approval.