{"ID":2848315,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2510.25060","arxiv_id":"2510.25060","title":"Nonlinear Dynamics In Optimization Landscape of Shallow Neural Networks with Tunable Leaky ReLU","abstract":"In this work, we study the nonlinear dynamics of a shallow neural network trained with mean-squared loss and leaky ReLU activation. Under Gaussian inputs and equal layer width k, (1) we establish, based on the equivariant gradient degree, a theoretical framework, applicable to any number of neurons k\u003e= 4, to detect bifurcation of critical points with associated symmetries from global minimum as leaky parameter $α$ varies. Typically, our analysis reveals that a multi-mode degeneracy consistently occurs at the critical number 0, independent of k. (2) As a by-product, we further show that such bifurcations are width-independent, arise only for nonnegative $α$ and that the global minimum undergoes no further symmetry-breaking instability throughout the engineering regime $α$ in range (0,1). An explicit example with k=5 is presented to illustrate the framework and exhibit the resulting bifurcation together with their symmetries.","short_abstract":"In this work, we study the nonlinear dynamics of a shallow neural network trained with mean-squared loss and leaky ReLU activation. Under Gaussian inputs and equal layer width k, (1) we establish, based on the equivariant gradient degree, a theoretical framework, applicable to any number of neurons k\u003e= 4, to detect bif...","url_abs":"https://arxiv.org/abs/2510.25060","url_pdf":"https://arxiv.org/pdf/2510.25060v1","authors":"[\"Jingzhou Liu\"]","published":"2025-10-29T01:00:07Z","proceeding":"math.OC","tasks":"[\"math.OC\",\"cs.LG\",\"math.DS\"]","methods":"[]","has_code":false}