{"ID":2886519,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2508.03688","arxiv_id":"2508.03688","title":"Learning quadratic neural networks in high dimensions: SGD dynamics and scaling laws","abstract":"We study the optimization and sample complexity of gradient-based training of a two-layer neural network with quadratic activation function in the high-dimensional regime, where the data is generated as $f_*(\\boldsymbol{x}) \\propto \\sum_{j=1}^{r}λ_j σ\\left(\\langle \\boldsymbol{θ_j}, \\boldsymbol{x}\\rangle\\right), \\boldsymbol{x} \\sim N(0,\\boldsymbol{I}_d)$, $σ$ is the 2nd Hermite polynomial, and $\\lbrace\\boldsymbolθ_j \\rbrace_{j=1}^{r} \\subset \\mathbb{R}^d$ are orthonormal signal directions. We consider the extensive-width regime $r \\asymp d^β$ for $β\\in [0, 1)$, and assume a power-law decay on the (non-negative) second-layer coefficients $λ_j\\asymp j^{-α}$ for $α\\geq 0$. We present a sharp analysis of the SGD dynamics in the feature learning regime, for both the population limit and the finite-sample (online) discretization, and derive scaling laws for the prediction risk that highlight the power-law dependencies on the optimization time, sample size, and model width. Our analysis combines a precise characterization of the associated matrix Riccati differential equation with novel matrix monotonicity arguments to establish convergence guarantees for the infinite-dimensional effective dynamics.","short_abstract":"We study the optimization and sample complexity of gradient-based training of a two-layer neural network with quadratic activation function in the high-dimensional regime, where the data is generated as $f_*(\\boldsymbol{x}) \\propto \\sum_{j=1}^{r}λ_j σ\\left(\\langle \\boldsymbol{θ_j}, \\boldsymbol{x}\\rangle\\right), \\boldsy...","url_abs":"https://arxiv.org/abs/2508.03688","url_pdf":"https://arxiv.org/pdf/2508.03688v3","authors":"[\"Gérard Ben Arous\",\"Murat A. Erdogdu\",\"Nuri Mert Vural\",\"Denny Wu\"]","published":"2025-08-05T17:57:56Z","proceeding":"stat.ML","tasks":"[\"stat.ML\",\"cs.LG\"]","methods":"[]","has_code":false}