{"ID":2862765,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2509.26207","arxiv_id":"2509.26207","title":"The silence of the weights: a structural pruning strategy for attention-based audio signal architectures with second order metrics","abstract":"Transformer-based models have become the state of the art across multiple domains, from natural language processing to machine listening, thanks to the attention mechanisms. However, the attention layers require a large number of parameters and high-end hardware for both training and inference. We propose a novel channel-pruning technique explicitly targeted at the attention mechanism, decoupling the pruning of each head and the four layers in the attention block: query, key, value, and output projection matrices, employing a second-order metric to score the network's parameters. We compare our technique against head-pruning strategies and magnitude-driven scoring metrics, investigating the effects of pruning on Audio Spectrogram Transformer (AST) and Whisper. Our results show that even after pruning 50\\% of the parameters in the attention block, performance is largely preserved.","short_abstract":"Transformer-based models have become the state of the art across multiple domains, from natural language processing to machine listening, thanks to the attention mechanisms. However, the attention layers require a large number of parameters and high-end hardware for both training and inference. We propose a novel chann...","url_abs":"https://arxiv.org/abs/2509.26207","url_pdf":"https://arxiv.org/pdf/2509.26207v2","authors":"[\"Andrea Diecidue\",\"Carlo Alberto Barbano\",\"Piero Fraternali\",\"Mathieu Fontaine\",\"Enzo Tartaglione\"]","published":"2025-09-30T13:10:19Z","proceeding":"cs.SD","tasks":"[\"cs.SD\",\"cs.LG\"]","methods":"[\"Transformer\"]","has_code":false}