Lemma A.12 - Gaussian head misalignment

Notation: $n = \#\{\text{training examples}\}$, $d$ is the input dimension, $k$ the feature dimension, $X:\mathbb R^d\to\mathbb R^n$ the data map, $Y$ the labels, $B:\mathbb R^d\to\mathbb R^k$ the features, and $v\in\mathbb R^k$ the head. Adjoint means Euclidean transpose. The loss is $\widehat L(v,B)=\|XB^\top v-Y\|^2$, with no normalization. The probability model, when present, is explicitly specified below; deterministic flow statements involve no random data assumption.

For every natural number $k$, every $u\in\mathbb R^k$, and every real number $\sigma$, if $u\neq0$ and $\sigma>0$, then for almost every $v_0$ under the pushforward of standard Gaussian measure on Euclidean $\mathbb R^k$ by $z\mapsto\sigma z$, $\left|\langle v_0,u\rangle^2-\langle u,u\rangle^2\right|>0$. Equivalently, both equalities $\langle v_0,u\rangle=\|u\|^2$ and $\langle v_0,u\rangle=-\|u\|^2$ fail outside a set of measure zero. For $k=0$ the hypothesis $u\neq0$ is impossible, so the implication is vacuous. For $u=0$ or $\sigma\leq0$, no almost-everywhere conclusion is required.

Formalization note: Qualitative source-derived consequence of Lemma A.12; no numerical anti-concentration constant is claimed. Source: Kumar, Raghunathan, Jones, Ma, and Liang, Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution, ICLR 2022, https://arxiv.org/pdf/2202.10054v1. Appendix A.3.2, PDF pp. 34--35, Lemma A.12, equations (A.123), (A.126)--(A.128). Source-backed parent: Section 3.4, PDF p. 10, Proposition 3.7, equations (3.10)--(3.11); Appendix A.7, PDF pp. 45--47.