Motivation
A single exact linear equation y=⟨x,s⟩ about an unknown unit vector s∈Rd carries arbitrarily fine information: d independent Gaussian equations determine s almost surely, provided all of them can be kept. A learner that reads the equations once, in order, and keeps only a bounded number of bits between them faces a different problem. The question asked here is how many noiseless measurements such a learner needs in order to reach a prescribed angular precision ϵ, as a function of its memory.
This is a question about memory–sample trade-offs: how much a bound on persistent memory forces a learning procedure to read more data. With unbounded real registers, the randomized Kaczmarz update zt=zt−1+∥xt∥2yt−⟨xt,zt−1⟩xt satisfies E∥zt−s∥2=(1−1/d)t on isotropic Gaussian rows, so order dlog(1/ϵ) samples suffice. That update stores real numbers and gives no finite-bit upper bound; the question is whether a learner with roughly d2 bits can do better than this scale.
Background
- 1937. Kaczmarz introduces the projection method for linear systems; Strohmer and Vershynin (2009) give the randomized version with exponential convergence.
- 2014. Shamir's finite-message framework for online and distributed learning includes bounded-memory online processing as a special case.
- 2015. Steinhardt and Duchi prove memory-dependent minimax rates for sparse linear regression with noise.
- 2016. Steinhardt, Valiant and Wager relate memory, communication and statistical queries and pose a quadratic-memory versus exponential-sample conjecture for parity learning; Raz proves a time–space lower bound for parity learning, and in 2017 extends the method to a large class of learning problems.
- 2019. Sharan, Sidford and Valiant prove that for Gaussian regression with tiny uniform additive noise, a learner with at most d2/4 bits needs Ω(dlogr) samples to reach Euclidean accuracy d−r, in a stated range of r, and ask (Section 1.1) whether the first-order sample dependence on precision is optimal under bounded memory. In the same year Dagan, Kur and Shamir prove quadratic-space lower bounds for two different linear-prediction tasks in the streaming model.
- 2026. The OpenAI preprint Memory and precision in noiseless Gaussian regression (dated September 27, 2026) states an ΩA(dlog(1/ϵ)) lower bound for exact observations and memory Ad2. It has not been peer reviewed, and the Lean statement of its main theorem is open on this platform.
Setting
Fix integers d≥2, M≥0, T≥0. A signal s lies on the unit sphere Sd−1⊂Rd. At step t∈{1,…,T} the learner receives the pair (xt,yt) with xt∼N(0,Id) independent and yt=⟨xt,s⟩ exactly (no noise).
A finite-state learner with M bits keeps a state in a set of at most 2M elements. A data-independent shared seed ω (drawn from a probability space (Ω,ρ)) may select its rules. Given the seed, the current state, the step index and the current pair, a transition chooses the next state and whether to stop; it may perform unrestricted computation but retains only the next state. The learner must stop by the deterministic horizon T. Its output s^∈Sd−1 is a function of the terminal state, the stopping index and the seed only; a discarded observation cannot be read again. Success at precision ϵ means angular error arccos⟨s^,s⟩≤ϵ.
Write σd for the uniform probability measure on Sd−1. Uniform success is the probability of success when S∼σd is drawn independently of the rows and the seed.
In Lean (OAI.NoiselessRegression), the state space is Fin (2 ^ M), a learner is a structure Learner d M T Ω with fields initialChoice, transition, output, the sample law is the product of stdGaussian on EuclideanSpace ℝ (Fin d), and uniformSuccess and success are the probabilities defined above. Randomness during the run is represented through the arbitrary seed space Ω.
Formalization targets
Goal: OAI.NoiselessRegression.main
The goal is the conjunction of two statements.
Fixed quadratic memory (Theorem 1.2): for every A>0 there are cA>0 and dA such that, for d≥dA, M≤Ad2, 0<ϵ≤1/10, and every admissible learner,
S∼σdPr{arccos⟨s^,S⟩≤ϵ}≥32⟹T≥cAdlog(1/ϵ).
Subquadratic memory (Corollaries 1.3 and 1.4): there is an absolute c>0 such that for every memory sequence M(d)=o(d2) there is d0 with the same conclusion T≥cdlog(1/ϵ) for d≥d0, under either uniform success at least 2/3 or success at least 2/3 for every fixed signal s.
No relation between ϵ and M is assumed; the constant is absolute in the subquadratic case.
Milestone: supporting estimates (OAI.MemoryPrecision.main)
A single bundled statement collecting the paper's projection-density and backward-block estimates (Theorem 3.1, Propositions A.1, A.4, A.5, B.6, B.7, Corollary 8.2), the cube-prior precision bound (Proposition 8.3), and the explicit success bound after a bounded number of samples (Proposition 5.3):
Pr{arccos⟨s^,S⟩≤ϵ}≤min{1,[2ϵeC(1+M/d2)⌈T/q⌉](d−1)/2},q=⌊(d−1)/8⌋.
Significance
The result. Theorem 1.2 answers, for exact observations and isotropic Gaussian rows, the precision question raised in Section 1.1 of Sharan–Sidford–Valiant: with O(d2) bits, the dlog(1/ϵ) scale achieved by real-valued Kaczmarz cannot be improved, uniformly over all 0<ϵ≤1/10. Because an exact-observation learner can simulate added noise, the preprint derives from it an ΩA(drlogd) lower bound in the noisy experiment of Sharan–Sidford–Valiant at accuracy d−r, which strengthens their Ω(dlogr) bound in that range.
Formalizing it. The result is proved only in an unrefereed preprint; no machine-checked proof exists. A formal proof would certify a lower bound whose model (randomized measurable rules, early stopping, shared seeds, measurability conventions) is delicate, and the conventions are already fixed in the Lean definitions. Remaining work: the full formal proof, and as a variant the every-signal version of Theorem 1.2 for fixed A (Corollary 1.4), which the Lean goal states only in the subquadratic case.
Difficulty
Conditioning the uniform prior on one exact observation confines the signal to a hyperplane section of the sphere, a law singular with respect to the original spherical measure; information-theoretic arguments that track a posterior density therefore break down after a single step. Arguments for noisy observations, such as Sharan–Sidford–Valiant's, rely on the noise to keep posteriors spread out and do not transfer to the more informative exact experiment. The bound must also hold uniformly in ϵ with no relation between ϵ and M, so a learner with Θ(d2) bits must be prevented from storing log(1/ϵ) bits per coordinate for very small ϵ.
Formalization scope
- Vectors live in
EuclideanSpace ℝ (Fin d); signals and outputs in Metric.sphere 0 1. The uniform sphere law is the normalized toSphere measure; rows are i.i.d. stdGaussian.
- States are
Fin (2 ^ M); the first component of each action is the stop flag. Runs that never stop are forced to terminate at index T.
Learner.Admissible requires a.e.-measurability of the initial choice, transitions (against Gaussian × Lebesgue on pairs), outputs, and of the full experiment under each fixed signal and under the uniform prior.
- Probabilities are
ℝ≥0∞-valued; the threshold is 2/3; angular error is Real.arccos ⟪ŝ, s⟫; log is natural.
- The seed space Ω is an arbitrary probability space in universe
u, so randomized learners are covered by placing their randomness in the seed.
The goal cannot be satisfied trivially: the hypotheses range over all admissible learners, and the success threshold 2/3 is achievable (with large T), so the implication has content.
A complete development needs Gaussian measures on Euclidean space, spherical measure and cap estimates, Gram determinants and affine distances, Lq densities of Gaussian projections, and measurable selection of Borel versions of kernels. The projection-density estimates are reusable beyond this mission. Contributions toward the bundled milestone, or toward any of its component estimates, are welcome.
Selected references
- OpenAI, Memory and precision in noiseless Gaussian regression, OpenAI Math Release preprint, September 27, 2026. https://github.com/openai/math/blob/adc7f1241b42e322a6451854ab7e4b4c146bf78a/preprints/Memory-and-precision-in-noiseless-Gaussian-regression-September-27-2026/paper.pdf
- V. Sharan, A. Sidford, G. Valiant, Memory-Sample Tradeoffs for Linear Regression with Small Error, STOC 2019. https://doi.org/10.1145/3313276.3316403 (full version https://arxiv.org/abs/1904.08544)
- R. Raz, Fast Learning Requires Good Memory: A Time-Space Lower Bound for Parity Learning, 2016. https://arxiv.org/abs/1602.05161
- R. Raz, A Time-Space Lower Bound for a Large Class of Learning Problems, FOCS 2017. https://doi.org/10.1109/FOCS.2017.73
- J. Steinhardt, G. Valiant, S. Wager, Memory, Communication, and Statistical Queries, COLT 2016. https://proceedings.mlr.press/v49/steinhardt16.html
- J. Steinhardt, J. Duchi, Minimax Rates for Memory-Bounded Sparse Linear Regression, COLT 2015. https://proceedings.mlr.press/v40/Steinhardt15.html
- O. Shamir, Fundamental Limits of Online and Distributed Algorithms for Statistical Learning and Estimation, NeurIPS 2014. https://proceedings.neurips.cc/paper_files/paper/2014/hash/cc427d934a7f6c0663e5923f49eba531-Abstract.html
- Y. Dagan, G. Kur, O. Shamir, Space Lower Bounds for Linear Prediction in the Streaming Model, COLT 2019. https://proceedings.mlr.press/v99/dagan19b.html
- T. Strohmer, R. Vershynin, A Randomized Kaczmarz Algorithm with Exponential Convergence, J. Fourier Anal. Appl. 15 (2009). https://doi.org/10.1007/s00041-008-9030-4