{"ID":6536180,"CreatedAt":"2026-07-14T01:21:01.169441415Z","UpdatedAt":"2026-07-15T03:28:55.185153975Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2607.10681","arxiv_id":"2607.10681","title":"LayerNorm as Implicit Gain Control in Looped Transformers","abstract":"In pre-LayerNorm looped transformers, LayerNorm inside the recurrent block acts as an implicit gain controller: by coupling the block's local Lipschitz constant inversely to the activation scale, it renders the recurrence Jacobian non-normal -- asymptotically contractive at every verified fixed point even where its operator norm exceeds 1 -- so the true stability budget is the spectral margin, not an operator-norm bound. That margin depletes as the carry $ρ\\to 1$, and a minority of initializations never converge to a fixed point at all, so the diagonal carry constraint $ρ(\\bar{A}) \u003c 1$ is necessary but not sufficient for convergence of the full recurrence. Training experiments across six tasks, including a controlled ablation, reveal that the linear carry is not the depth-memory mechanism: gradient descent routes memory through the block's more expressive nonlinear recurrence and leaves the stability-constrained carry at rest -- the carry's role is stabilization, not memory. We characterize the boundary of this claim: on tasks with axis-aligned per-channel structure, gradient descent does recruit the carry. All results are derived analytically and verified in a from-scratch, CPU-scale implementation; verification at larger scale is needed.","short_abstract":"In pre-LayerNorm looped transformers, LayerNorm inside the recurrent block acts as an implicit gain controller: by coupling the block's local Lipschitz constant inversely to the activation scale, it renders the recurrence Jacobian non-normal -- asymptotically contractive at every verified fixed point even where its ope...","url_abs":"https://arxiv.org/abs/2607.10681","url_pdf":"https://arxiv.org/pdf/2607.10681v1","authors":"[\"Matthias M. M. Buehlmaier\"]","published":"2026-07-12T09:57:23Z","proceeding":"cs.LG","tasks":"[\"cs.LG\",\"cs.NE\"]","methods":"[\"Transformer\"]","has_code":false}