{"ID":2858370,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2510.08570","arxiv_id":"2510.08570","title":"Who Said Neural Networks Aren't Linear?","abstract":"Neural networks are famously nonlinear. However, linearity is defined relative to a pair of vector spaces, $f:X \\to Y$. Leveraging the algebraic concept of transport of structure, we propose a method to explicitly identify non-standard vector spaces where a neural network acts as a linear operator. When sandwiching a linear operator $A$ between two invertible neural networks, $f(x)=g_y^{-1}(A g_x(x))$, the corresponding vector spaces $X$ and $Y$ are induced by newly defined addition and scaling actions derived from $g_x$ and $g_y$. We term this kind of architecture a Linearizer. This framework makes the entire arsenal of linear algebra, including SVD, pseudo-inverse, orthogonal projection and more, applicable to nonlinear mappings. Furthermore, we show that the composition of two Linearizers that share a neural network is also a Linearizer. We leverage this property and demonstrate that training diffusion models using our architecture makes the hundreds of sampling steps collapse into a single step. We further utilize our framework to enforce idempotency (i.e. $f(f(x))=f(x)$) on networks leading to a globally projective generative model and to demonstrate modular style transfer.","short_abstract":"Neural networks are famously nonlinear. However, linearity is defined relative to a pair of vector spaces, $f:X \\to Y$. Leveraging the algebraic concept of transport of structure, we propose a method to explicitly identify non-standard vector spaces where a neural network acts as a linear operator. When sandwiching a l...","url_abs":"https://arxiv.org/abs/2510.08570","url_pdf":"https://arxiv.org/pdf/2510.08570v2","authors":"[\"Nimrod Berman\",\"Assaf Hallak\",\"Assaf Shocher\"]","published":"2025-10-09T17:59:57Z","proceeding":"cs.LG","tasks":"[\"cs.LG\"]","methods":"[\"Diffusion Model\"]","has_code":false}