{"ID":5438616,"CreatedAt":"2026-07-01T01:17:58.482524686Z","UpdatedAt":"2026-07-03T03:45:45.236501583Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2606.31112","arxiv_id":"2606.31112","title":"What Counts as an Error? Dual-Reference Benchmarking for Atypical ASR","abstract":"ASR systems have been often reported to underperform on atypical speech. An often conflated compounding factor is the existence of two valid transcription references: verbatim (actual produced speech, including repetitions/prolongations) and intended (the canonical form of the text with disfluencies removed) in atypical speech recognition depending on context and use-case. Most ASR evaluations conflate this duality into a single ground truth and reward systems that delete disfluencies, ignoring verbatim faithfulness. We benchmark 11 ASR models from encoder-decoder, CTC and transducer families using both verbatim and intended references on atypical stuttered speech as a case study. Our quantitative assessment underlines the disparity in model performance and rankings using the two transcript styles. Through this analysis, we highlight the importance of selecting a suitable transcription reference for valid model selection depending on the use-case, particularly for atypical ASR.","short_abstract":"ASR systems have been often reported to underperform on atypical speech. An often conflated compounding factor is the existence of two valid transcription references: verbatim (actual produced speech, including repetitions/prolongations) and intended (the canonical form of the text with disfluencies removed) in atypica...","url_abs":"https://arxiv.org/abs/2606.31112","url_pdf":"https://arxiv.org/pdf/2606.31112v1","authors":"[\"Hawau Olamide Toyin\",\"Srinivasan Umesh\",\"Hanan Aldarmaki\"]","published":"2026-06-30T04:15:39Z","proceeding":"cs.CL","tasks":"[\"cs.CL\",\"cs.HC\"]","methods":"[]","has_code":false}