{"ID":6267754,"CreatedAt":"2026-07-10T01:11:38.759438437Z","UpdatedAt":"2026-07-11T19:24:34.872436639Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2607.07884","arxiv_id":"2607.07884","title":"Optimal Learning Rate Scaling Depends on Data in Deep Scalar Linear Networks","abstract":"In this short note we consider the gradient descent dynamics of deep scalar linear networks, $f(x) = \\prod_{l=1}^L w_l x$, which enjoy exact time-course solutions for any integer depth. We show that even in this minimal model, the optimal depth-wise learning rate scaling depends on data, whereas data-agnostic scaling rules fail to transfer across depths. Under the data-dependent optimal scaling, the learning dynamics is independent of data and weakly dependent on depth, resulting in a constant linear convergence rate across all depths including infinity. We further show similar data-dependent effects in deep scalar linear networks with residual connections.","short_abstract":"In this short note we consider the gradient descent dynamics of deep scalar linear networks, $f(x) = \\prod_{l=1}^L w_l x$, which enjoy exact time-course solutions for any integer depth. We show that even in this minimal model, the optimal depth-wise learning rate scaling depends on data, whereas data-agnostic scaling r...","url_abs":"https://arxiv.org/abs/2607.07884","url_pdf":"https://arxiv.org/pdf/2607.07884v1","authors":"[\"Yedi Zhang\",\"Peter E. Latham\",\"Leena Chennuru Vankadara\",\"Andrew Saxe\"]","published":"2026-07-08T19:39:11Z","proceeding":"cs.LG","tasks":"[\"cs.LG\"]","methods":"[]","has_code":false}