{"ID":5935643,"CreatedAt":"2026-07-07T01:22:02.77346169Z","UpdatedAt":"2026-07-07T02:10:06.972658124Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2607.03519","arxiv_id":"2607.03519","title":"On the Convergence of Adam, Revisited","abstract":"We show that projected Adam for online optimization with arbitrary moment decay parameters $β_1,β_2\\in[0,1)$ can have average regret bounded away from zero. A similar result of Reddi-Kale-Kumar from 2018 required $β_1\u003c\\sqrt{β_2}$. Similar to their result, we use a three-periodic sequence of linear functions on $[-1,1]$ with slopes $c,-1,-1$, though we use $c$ slightly larger than $2$. This nonzero average regret result extends to Adam variants such as AdamW, RMSProp, NAdam, Adan, AdaMax, Muon, and to an i.i.d. variant of the three-periodic sequence of slopes for Adam.","short_abstract":"We show that projected Adam for online optimization with arbitrary moment decay parameters $β_1,β_2\\in[0,1)$ can have average regret bounded away from zero. A similar result of Reddi-Kale-Kumar from 2018 required $β_1\u003c\\sqrt{β_2}$. Similar to their result, we use a three-periodic sequence of linear functions on $[-1,1]$...","url_abs":"https://arxiv.org/abs/2607.03519","url_pdf":"https://arxiv.org/pdf/2607.03519v1","authors":"[\"Steven Heilman\",\"Sampad Mohanty\"]","published":"2026-07-03T17:45:56Z","proceeding":"cs.LG","tasks":"[\"cs.LG\",\"math.OC\",\"stat.ML\"]","methods":"[]","has_code":false}