{"ID":2838225,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2511.17970","arxiv_id":"2511.17970","title":"Controllability Analysis of State Space-based Language Model","abstract":"State-space models (SSMs), particularly Mamba, have become powerful architectures for sequence modeling, yet their internal dynamics remain poorly understood compared to attention-based models. We introduce and validate the Influence Score, a controllability-based metric derived from the discretized state-space parameters of Mamba and computed through a backward recurrence analogous to system observability. The score quantifies how strongly a token at position k affects all later states and outputs. We evaluate this measure across three Mamba variants: mamba-130m, mamba-2.8b, and mamba-2.8b-slimpj, using six experiments that test its sensitivity to temperature, prompt complexity, token type, layer depth, token position, and input perturbations. The results show three main insights: (1) the Influence Score increases with model size and training data, reflecting model capacity; (2) Mamba exhibits consistent architectural patterns, including recency bias and concentrated influence in mid-to-late layers; and (3) emergent behaviors appear only at scale, with mamba-2.8b-slimpj uniquely prioritizing content words and reducing internal influence in the presence of noise. These findings establish the Influence Score as a practical diagnostic tool for interpreting and comparing SSM-based language models.","short_abstract":"State-space models (SSMs), particularly Mamba, have become powerful architectures for sequence modeling, yet their internal dynamics remain poorly understood compared to attention-based models. We introduce and validate the Influence Score, a controllability-based metric derived from the discretized state-space paramet...","url_abs":"https://arxiv.org/abs/2511.17970","url_pdf":"https://arxiv.org/pdf/2511.17970v1","authors":"[\"Mohamed Mabrok\",\"Yalda Zafari\"]","published":"2025-11-22T08:13:17Z","proceeding":"cs.LG","tasks":"[\"cs.LG\"]","methods":"[\"Language Model\"]","has_code":false}