{"ID":3005023,"CreatedAt":"2026-06-03T03:09:48.883664427Z","UpdatedAt":"2026-06-05T06:46:15.197025399Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2606.03241","arxiv_id":"2606.03241","title":"Benchmarking Speech-to-Speech Translation Models","abstract":"Speech-to-speech translation (S2ST) has advanced rapidly, but offline evaluation lacks a unified protocol: studies report non-overlapping metric subsets, preventing direct comparisons. We introduce COMPASS, a unified and reproducible benchmarking framework integrating 46 metrics across eight dimensions, and deploy it on 1,248 model-language configurations from FLEURS and CVSS, spanning cascaded and end-to-end architectures over ten language pairs. Architectures exhibit complementary strengths: best-vs-worst gaps exceed 30\\% on naturalness and speaker preservation but remain within a few points on translation quality, so single-metric rankings systematically misrepresent system quality. Correlation filtering reduces 46 metrics to 10 per direction, with three axes requiring different metrics across X$\\to$EN and EN$\\to$X (e.g., TER/UTMOS vs. ChrF++/NISQA-MOS); these subsets preserve rankings (Spearman's $ρ\u003e0.80$) while cutting evaluation time by $\\approx 2.5\\times$. Human validation across dubbing, podcasts, and medical domains shows standalone MOS predictors fail to predict listener preference, while top domain-specific metrics correlate with human judgment ($ρ\\geq 0.90$). We release COMPASS as a foundation for domain-aware S2ST evaluation.","short_abstract":"Speech-to-speech translation (S2ST) has advanced rapidly, but offline evaluation lacks a unified protocol: studies report non-overlapping metric subsets, preventing direct comparisons. We introduce COMPASS, a unified and reproducible benchmarking framework integrating 46 metrics across eight dimensions, and deploy it o...","url_abs":"https://arxiv.org/abs/2606.03241","url_pdf":"https://arxiv.org/pdf/2606.03241v1","authors":"[\"Alkis Koudounas\",\"Hayato Futami\",\"Quentin Jodelet\",\"Osamu Take\",\"Shinji Watanabe\",\"Emiru Tsunoo\"]","published":"2026-06-02T07:01:33Z","proceeding":"cs.CL","tasks":"[\"cs.CL\",\"eess.AS\"]","methods":"[]","has_code":false}