{"ID":2886814,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2508.02228","arxiv_id":"2508.02228","title":"Large Language Model Guided Decoding for Self-Supervised Speech Recognition","abstract":"Self-supervised automatic speech recognition (SSL-ASR) is an ASR approach that uses speech encoders pretrained on large amounts of unlabeled audio (e.g., wav2vec2.0 or HuBERT) and then fine-tunes them with limited labeled data to perform transcription. Decoding is usually performed with a CTC decoder, whose hypotheses are scored and refined using an external language model (LM), typically an n-gram or neural LM, which guides beam search to produce the final transcription. Using Large Language Models (LLMs) as external LMs remains a challenge, as their word probabilities are overly confident. The proposed method integrates an LLM with an SSL acoustic model by using the LLM's decoding mechanism to generate a set of candidate next tokens. For each candidate, the SSL model provides an acoustic score by aligning it to the input acoustics of the SSL model. A combined acoustic and LLM score is then calculated based on decomposing the MAP estimator of words given the acoustic signal. The tokens with the highest combined scores are maintained in a beam, which is then used to proceed to the next decoding step. We illustrate the effectiveness of our method through a comprehensive comparison with the current state-of-the-art LLM-based decoding, post-processing, and error-correcting methods across multiple datasets. Our approach proves particularly effective when processing challenging inputs such as complex speech sentences, acronyms, and domain-specific vocabulary.","short_abstract":"Self-supervised automatic speech recognition (SSL-ASR) is an ASR approach that uses speech encoders pretrained on large amounts of unlabeled audio (e.g., wav2vec2.0 or HuBERT) and then fine-tunes them with limited labeled data to perform transcription. Decoding is usually performed with a CTC decoder, whose hypotheses...","url_abs":"https://arxiv.org/abs/2508.02228","url_pdf":"https://arxiv.org/pdf/2508.02228v2","authors":"[\"Eyal Cohen\",\"Bhiksha Raj\",\"Joseph Keshet\"]","published":"2025-08-04T09:25:48Z","proceeding":"eess.AS","tasks":"[\"eess.AS\"]","methods":"[\"Large Language Model\",\"Language Model\"]","has_code":false}