Masked AdamW: one concrete deterministic optimizer instance

{"text": "{\n "data": "adamM1 (definition). Given a real parameter $\\\\beta_1$, a natural number $d$ (implicit), a training state $S$ over $d$ coordinates \u2014 i.e. a quadruple $S = (w, m, v, t)$ where $w, m, v \\\\in \\\\mathbb{R}^d$ are the parameter vector and the two moment vectors of the state and $t \\\\in \\\\mathbb{N}$ is its step counter \u2014 a vector $g \\\\in \\\\mathbb{R}^d$, and a coordinate index $j \\\\in \\\\{0, \\\\dots, d-1\\\\}$, the definition produces the real number\n\n

\n\nwhere $m_j$ is the $j$-th coordinate of the state's first-moment vector $m$ and $g_j$ is the $j$-th coordinate of $g$. No hypothesis is placed on any argument: $\\\\beta_1$ is an arbitrary real, not restricted to $[0,1]$ (for example $\\\\beta_1 = 1$ makes the value $m_j$ and $\\\\beta_1 = 0$ makes it $g_j$), the formula is total real arithmetic, and the remaining components of $S$ ($w$, $v$, $t$) play no role. When $d = 0$ the index set is empty, so there is no coordinate $j$ to which the definition can be applied.\n\n**adamM2 (definition).** With exactly the same arguments \u2014 a real $\\\\beta_2$, an implicit natural $d$, a training state $S = (w, m, v, t)$, a vector $g \\\\in \\\\mathbb{R}^d$, an index $j$ \u2014 the definition produces the real number\n\n

\n\nwhere $v_j$ is the $j$-th coordinate of the state's second-moment vector $v$. Again there are no hypotheses, the arithmetic is total, and only the $v$-slot of $S$ is used. Although $g_j^2 \\\\ge 0$, the defined value can be an arbitrary real number, since neither $v_j$ nor $\\\\beta_2$ is constrained (e.g. $\\\\beta_2 > 1$ with $v_j$ small makes it negative); this value is later placed under a square root in adamWStep below.\n\n**adamWStep (definition).** Given five real parameters $\\\\beta_1, \\\\beta_2, \\\\eta, \\\\lambda, \\\\varepsilon$ (the code's lam is $\\\\lambda$; no hypothesis whatsoever is imposed on any of them \u2014 in particular $\\\\varepsilon$ is not required to be positive and $\\\\beta_1, \\\\beta_2$ are not required to lie in $[0,1)$), an implicit natural number $d$, a finite set $T$ of coordinate indices (a finite subset of $\\\\{0,\\\\dots,d-1\\\\}$), a training state $S = (w, m, v, t)$ as above, and an arbitrary vector $g \\\\in \\\\mathbb{R}^d$ (nothing requires $g$ to be a gradient or to satisfy any masking property), the definition produces a new training state $S' = (w', m', v', t')$ whose components are given coordinate-wise by:\n\n- step counter: $t' = t + 1$;\n- first moments: $m'_j = \\\\beta_1 m_j + (1 - \\\\beta_1) g_j$ \u2014 exactly the value adamM1 defines \u2014 for every index $j$;\n- second moments: $v'_j = \\\\beta_2 v_j + (1 - \\\\beta_2) g_j^2$ \u2014 exactly the value adamM2 defines \u2014 for every index $j$;\n- parameters: for indices $j \\\\in T$,\n

\n while for indices $j \\\\notin T$, $w'_j = w_j$ exactly (copied with no decay factor and no moment term).\n\nThe bias-corrected ratio inside $w'_j$ uses the freshly computed moments $m'_j, v'_j$ (the very values stored in the output state, computed from the incoming $S$ and $g$), and the correction exponents are $t + 1 \\\\ge 1$, the successor of the incoming step counter. Everything is total real arithmetic with no side conditions, so the following degenerate behaviors are literally included in the definition:\n\n- Division by $1 - \\\\beta_1^{t+1}$ or $1 - \\\\beta_2^{t+1}$ equal to zero. Real division is total and $x/0$ is defined to be $0$. The factor $1 - \\\\beta_1^{t+1}$ vanishes whenever $\\\\beta_1^{t+1} = 1$: for instance $\\\\beta_1 = 1$ (any $t$), or $\\\\beta_1 = -1$ with $t+1$ even. In that case $m'_j / (1 - \\\\beta_1^{t+1}) = 0$. Likewise $1 - \\\\beta_2^{t+1} = 0$ (e.g. $\\\\beta_2 = 1$, or $\\\\beta_2 = -1$ with $t+1$ even) makes the quotient under the root equal to $0$.\n- Square root of a negative number. The square root used is the total real square root: for a nonnegative argument it is the nonnegative root, and for a negative argument it is defined to be $0$. The argument $v'_j / (1 - \\\\beta_2^{t+1})$ can be negative (the sign of $v'_j$ is unconstrained, and the denominator can be negative when $\\\\beta_2^{t+1} > 1$), in which case the root is $0$.\n- Vanishing outer denominator. The denominator $\\\\sqrt{v'_j/(1-\\\\beta_2^{t+1})} + \\\\varepsilon$ can itself be $0$ (e.g. $\\\\varepsilon = 0$ together with a zero root, or a negative $\\\\varepsilon$ cancelling a positive root); then the whole fraction is $0$ by the division convention. In every case where either denominator is $0$, the subtracted term vanishes identically and the update of a $j \\\\in T$ degenerates to the pure rescaling $w'_j = (1 - \\\\eta\\\\lambda)\\\\, w_j$.\n- Coordinates outside $T$. For $j \\\\notin T$ the parameter coordinate is untouched, but the moment slots $m'_j, v'_j$ are still updated by the formulas above at every coordinate (the incoming $g$ enters those moment updates at all coordinates, unmasked by $T$), and the counter increments once regardless of $T$. So the effect of $T$ is confined to the parameter vector $w$.\n- Degenerate index sets. $T$ may be empty (then $w' = w$, while the moments and counter still advance) or the full index set; and for $d = 0$ all vectors are empty and the output is the unique empty-coordinate state with counter $t + 1$."\n}", "details": {"resolvedPath": "/home/ajax/.omp/agent/sessions/-math/2026-09-22T19-31-08-510Z_01a0ca99-be5e-7000-93c6-dac7c74cc004/RB-VathekAdamW.md", "contentType": "text/markdown", "totalLines": 3, "displayContent": {"text": "{\n "data": "adamM1 (definition). Given a real parameter $\\\\beta_1$, a natural number $d$ (implicit), a training state $S$ over $d$ coordinates \u2014 i.e. a quadruple $S = (w, m, v, t)$ where $w, m, v \\\\in \\\\mathbb{R}^d$ are the parameter vector and the two moment vectors of the state and $t \\\\in \\\\mathbb{N}$ is its step counter \u2014 a vector $g \\\\in \\\\mathbb{R}^d$, and a coordinate index $j \\\\in \\\\{0, \\\\dots, d-1\\\\}$, the definition produces the real number\n\n

\n\nwhere $m_j$ is the $j$-th coordinate of the state's first-moment vector $m$ and $g_j$ is the $j$-th coordinate of $g$. No hypothesis is placed on any argument: $\\\\beta_1$ is an arbitrary real, not restricted to $[0,1]$ (for example $\\\\beta_1 = 1$ makes the value $m_j$ and $\\\\beta_1 = 0$ makes it $g_j$), the formula is total real arithmetic, and the remaining components of $S$ ($w$, $v$, $t$) play no role. When $d = 0$ the index set is empty, so there is no coordinate $j$ to which the definition can be applied.\n\n**adamM2 (definition).** With exactly the same arguments \u2014 a real $\\\\beta_2$, an implicit natural $d$, a training state $S = (w, m, v, t)$, a vector $g \\\\in \\\\mathbb{R}^d$, an index $j$ \u2014 the definition produces the real number\n\n

\n\nwhere $v_j$ is the $j$-th coordinate of the state's second-moment vector $v$. Again there are no hypotheses, the arithmetic is total, and only the $v$-slot of $S$ is used. Although $g_j^2 \\\\ge 0$, the defined value can be an arbitrary real number, since neither $v_j$ nor $\\\\beta_2$ is constrained (e.g. $\\\\beta_2 > 1$ with $v_j$ small makes it negative); this value is later placed under a square root in adamWStep below.\n\n**adamWStep (definition).** Given five real parameters $\\\\beta_1, \\\\beta_2, \\\\eta, \\\\lambda, \\\\varepsilon$ (the code's lam is $\\\\lambda$; no hypothesis whatsoever is imposed on any of them \u2014 in particular $\\\\varepsilon$ is not required to be positive and $\\\\beta_1, \\\\beta_2$ are not required to lie in $[0,1)$), an implicit natural number $d$, a finite set $T$ of coordinate indices (a finite subset of $\\\\{0,\\\\dots,d-1\\\\}$), a training state $S = (w, m, v, t)$ as above, and an arbitrary vector $g \\\\in \\\\mathbb{R}^d$ (nothing requires $g$ to be a gradient or to satisfy any masking property), the definition produces a new training state $S' = (w', m', v', t')$ whose components are given coordinate-wise by:\n\n- step counter: $t' = t + 1$;\n- first moments: $m'_j = \\\\beta_1 m_j + (1 - \\\\beta_1) g_j$ \u2014 exactly the value adamM1 defines \u2014 for every index $j$;\n- second moments: $v'_j = \\\\beta_2 v_j + (1 - \\\\beta_2) g_j^2$ \u2014 exactly the value adamM2 defines \u2014 for every index $j$;\n- parameters: for indices $j \\\\in T$,\n

\n while for indices $j \\\\notin T$, $w'_j = w_j$ exactly (copied with no decay factor and no moment term).\n\nThe bias-corrected ratio inside $w'_j$ uses the freshly computed moments $m'_j, v'_j$ (the very values stored in the output state, computed from the incoming $S$ and $g$), and the correction exponents are $t + 1 \\\\ge 1$, the successor of the incoming step counter. Everything is total real arithmetic with no side conditions, so the following degenerate behaviors are literally included in the definition:\n\n- Division by $1 - \\\\beta_1^{t+1}$ or $1 - \\\\beta_2^{t+1}$ equal to zero. Real division is total and $x/0$ is defined to be $0$. The factor $1 - \\\\beta_1^{t+1}$ vanishes whenever $\\\\beta_1^{t+1} = 1$: for instance $\\\\beta_1 = 1$ (any $t$), or $\\\\beta_1 = -1$ with $t+1$ even. In that case $m'_j / (1 - \\\\beta_1^{t+1}) = 0$. Likewise $1 - \\\\beta_2^{t+1} = 0$ (e.g. $\\\\beta_2 = 1$, or $\\\\beta_2 = -1$ with $t+1$ even) makes the quotient under the root equal to $0$.\n- Square root of a negative number. The square root used is the total real square root: for a nonnegative argument it is the nonnegative root, and for a negative argument it is defined to be $0$. The argument $v'_j / (1 - \\\\beta_2^{t+1})$ can be negative (the sign of $v'_j$ is unconstrained, and the denominator can be negative when $\\\\beta_2^{t+1} > 1$), in which case the root is $0$.\n- Vanishing outer denominator. The denominator $\\\\sqrt{v'_j/(1-\\\\beta_2^{t+1})} + \\\\varepsilon$ can itself be $0$ (e.g. $\\\\varepsilon = 0$ together with a zero root, or a negative $\\\\varepsilon$ cancelling a positive root); then the whole fraction is $0$ by the division convention. In every case where either denominator is $0$, the subtracted term vanishes identically and the update of a $j \\\\in T$ degenerates to the pure rescaling $w'_j = (1 - \\\\eta\\\\lambda)\\\\, w_j$.\n- Coordinates outside $T$. For $j \\\\notin T$ the parameter coordinate is untouched, but the moment slots $m'_j, v'_j$ are still updated by the formulas above at every coordinate (the incoming $g$ enters those moment updates at all coordinates, unmasked by $T$), and the counter increments once regardless of $T$. So the effect of $T$ is confined to the parameter vector $w$.\n- Degenerate index sets. $T$ may be empty (then $w' = w$, while the moments and counter still advance) or the full index set; and for $d = 0$ all vectors are empty and the output is the unique empty-coordinate state with counter $t + 1$."\n}", "startLine": 1, "lineNumbers": [1, 2, 3]}, "meta": {"source": {"type": "internal", "value": "agent://RB-VathekAdamW"}}}}