{"ID":5438761,"CreatedAt":"2026-07-01T01:17:58.482524686Z","UpdatedAt":"2026-07-03T09:10:46.706950747Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2606.31397","arxiv_id":"2606.31397","title":"Mixture-of-Control: State-Aware Fine-Tuning for Transformer-based Models","abstract":"State-based fine-tuning has emerged as a compelling alternative to weight-based adaptation for transformers, updating lightweight controls into states rather than model weights, offering substantial memory savings while retaining parameter efficiency. However, most existing state-based methods typically apply only per-block control updates, which limits inter-block information exchange and restricts representational adaptation. Meanwhile, prior mechanisms that enable cross-block communication often introduce considerable computational overhead, reducing their practicality for efficient fine-tuning. We introduce Mixture-of-Control (MoC), a lightweight fine-tuning framework that adaptively integrates local and global control signals to enhance representation learning. MoC treats block-wise control states as experts in a sparse mixture-of-experts process, enabling efficient communication across transformer blocks. Empirical results across diverse transformer-based benchmarks demonstrate that MoC outperforms state-based methods while maintaining a comparable memory and computational efficiency.","short_abstract":"State-based fine-tuning has emerged as a compelling alternative to weight-based adaptation for transformers, updating lightweight controls into states rather than model weights, offering substantial memory savings while retaining parameter efficiency. However, most existing state-based methods typically apply only per-...","url_abs":"https://arxiv.org/abs/2606.31397","url_pdf":"https://arxiv.org/pdf/2606.31397v1","authors":"[\"Duc Anh Nguyen\",\"Tien Ngoc Luu\",\"Tung Pham\",\"Toan Tran\"]","published":"2026-06-30T09:25:37Z","proceeding":"cs.LG","tasks":"[\"cs.LG\",\"cs.AI\"]","methods":"[\"Transformer\"]","has_code":false}