{"ID":2824340,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2512.23448","arxiv_id":"2512.23448","title":"Dynamic Subspace Composition: Efficient Adaptation via Contractive Basis Expansion","abstract":"Mixture of Experts (MoE) models scale capacity but often suffer from representation collapse and gradient instability. We propose Dynamic Subspace Composition (DSC), a framework that approximates context-dependent weights via a state-dependent, sparse expansion of a shared basis bank. Formally, DSC models the weight update as a residual trajectory within a Star- Shaped Domain, employing a Magnitude-Gated Simplex Interpolation to ensure continuity at the identity. Unlike standard Mixture-of-LoRAs, which incurs O(M rd) parameter complexity by retrieving independent rank-r matrices, DSC constructs a compositional rank-K approximation from decoupled unit-norm basis vectors. This reduces parameter complexity to O(M d) and memory traffic to O(Kd), while Frame-Theoretic regularization and spectral constraints provide rigorous worst-case bounds on the dynamic update. The code is available at https://github. com/VladimerKhasia/DSC","short_abstract":"Mixture of Experts (MoE) models scale capacity but often suffer from representation collapse and gradient instability. We propose Dynamic Subspace Composition (DSC), a framework that approximates context-dependent weights via a state-dependent, sparse expansion of a shared basis bank. Formally, DSC models the weight up...","url_abs":"https://arxiv.org/abs/2512.23448","url_pdf":"https://arxiv.org/pdf/2512.23448v1","authors":"[\"Vladimer Khasia\"]","published":"2025-12-29T13:11:01Z","proceeding":"cs.LG","tasks":"[\"cs.LG\"]","methods":"[\"Mixture of Experts\",\"LoRA\"]","project_urls":"[\"https://github\"]","has_code":false}