{"ID":3004900,"CreatedAt":"2026-06-03T03:09:48.883664427Z","UpdatedAt":"2026-06-05T10:54:25.708081962Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2606.03483","arxiv_id":"2606.03483","title":"Analyzing Stream Collapse in Hyper-Connections: From Diagnosis to Mitigation","abstract":"Hyper-Connections (HC) replace the single Transformer residual stream with multiple streams, introducing a permutation symmetry over stream indices. We study how this symmetry is resolved in practice: whether streams specialize in a balanced way or exhibit dominant-stream usage. Using fine-grained diagnostics for HC-based language models, we trace how multi-stream representations are actually used. We find that after an early seeding stage, residual mixing often remains close to identity, limiting a core HC mechanism for exchanging information between streams. Moreover, both signal and interpretable features concentrate in a dominant stream, and the nominally multi-stream residual connection can underutilize its capacity, behaving closer to a single-stream residual pathway. Finally, we show that breaking symmetry at stream initialization reduces dominant behavior and improves performance across \\textit{m}HC variants. Our code is publicly available.","short_abstract":"Hyper-Connections (HC) replace the single Transformer residual stream with multiple streams, introducing a permutation symmetry over stream indices. We study how this symmetry is resolved in practice: whether streams specialize in a balanced way or exhibit dominant-stream usage. Using fine-grained diagnostics for HC-ba...","url_abs":"https://arxiv.org/abs/2606.03483","url_pdf":"https://arxiv.org/pdf/2606.03483v1","authors":"[\"Ekaterina Alimaskina\",\"Gleb Molodtsov\",\"Aleksandr Beznosikov\"]","published":"2026-06-02T11:00:49Z","proceeding":"cs.LG","tasks":"[\"cs.LG\",\"cs.AI\"]","methods":"[\"Transformer\",\"Language Model\"]","has_code":false}