{"ID":2889716,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2507.20981","arxiv_id":"2507.20981","title":"Stochastic gradient with least-squares control variates","abstract":"The stochastic gradient descent (SGD) method is a widely used approach for solving stochastic optimization problems, but its convergence is typically slow. Existing variance reduction techniques, such as SAGA, improve convergence by leveraging stored gradient information; however, they are restricted to settings where the objective functional is a finite sum, and their performance degrades when the number of terms in the sum is large. In this work, we propose a novel approach which is well suited when the objective is given by an expectation over random variables with a continuous probability distribution. Our method constructs a control variate by fitting a linear model to past gradient evaluations using weighted discrete least-squares, effectively reducing variance while preserving computational efficiency. We establish theoretical sublinear convergence guarantees for strongly convex objectives and demonstrate the method's effectiveness through numerical experiments on random PDE-constrained optimization problems.","short_abstract":"The stochastic gradient descent (SGD) method is a widely used approach for solving stochastic optimization problems, but its convergence is typically slow. Existing variance reduction techniques, such as SAGA, improve convergence by leveraging stored gradient information; however, they are restricted to settings where...","url_abs":"https://arxiv.org/abs/2507.20981","url_pdf":"https://arxiv.org/pdf/2507.20981v2","authors":"[\"Fabio Nobile\",\"Matteo Raviola\",\"Nathan Schaeffer\"]","published":"2025-07-28T16:43:27Z","proceeding":"math.OC","tasks":"[\"math.OC\",\"math.NA\"]","methods":"[]","has_code":false}