{"ID":6023597,"CreatedAt":"2026-07-08T01:00:23.257252134Z","UpdatedAt":"2026-07-10T14:11:27.630055639Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2607.06314","arxiv_id":"2607.06314","title":"Gradient descent with exponentially increasing stepsizes and restarts","abstract":"Let $f:\\mathbb{R}^d \\rightarrow \\mathbb{R}$. We consider gradient descent $x_{n+1} = x_n - τ_n \\nabla f(x_n)$, where the stepsize $τ_n = τ\\cdot e^{rn}$ is exponentially growing (with $τ\u003e 0$ and $0 \u003c r \\ll 1$). This diverges for almost all initial values. We show that restarting the algorithm whenever $\\|x_{n+1} - x_n\\| \\geq e^r\\|x_n - x_{n-1}\\|$ has good properties: it works very well in practice; we determine the limiting convergence rate in the case of convergence to a non-degenerate local minimum: it improves on classic gradient descent even though computational cost is comparable. The precise choice of $0 \u003c r \\ll 1$ does not matter much and the method is virtually independent of an initial stepsize $τ$ that is too small: while the convergence rate for gradient descent decays linearly as $τ\\rightarrow 0$, it decays as $1/\\log(1/τ)$ in this modified version; numerical examples illustrate the results.","short_abstract":"Let $f:\\mathbb{R}^d \\rightarrow \\mathbb{R}$. We consider gradient descent $x_{n+1} = x_n - τ_n \\nabla f(x_n)$, where the stepsize $τ_n = τ\\cdot e^{rn}$ is exponentially growing (with $τ\u003e 0$ and $0 \u003c r \\ll 1$). This diverges for almost all initial values. We show that restarting the algorithm whenever $\\|x_{n+1} - x_n\\|...","url_abs":"https://arxiv.org/abs/2607.06314","url_pdf":"https://arxiv.org/pdf/2607.06314v1","authors":"[\"François Clément\",\"Stefan Steinerberger\"]","published":"2026-07-07T14:16:29Z","proceeding":"math.OC","tasks":"[\"math.OC\"]","methods":"[]","has_code":false}