{"ID":2883106,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2508.09228","arxiv_id":"2508.09228","title":"Objective Soups: Multilingual Multi-Task Modeling for Speech Processing","abstract":"Training a single model for multilingual, multi-task speech processing (MSP) is severely hampered by conflicting objectives between tasks like speech recognition and translation. While multi-objective optimization (MOO) aims to align gradient updates, its effectiveness diminishes as the number of tasks grows, making it difficult to find a common descent direction. This raises a fundamental question: should highly conflicting objectives be optimized jointly or separated into a hierarchical structure? To address this question, this paper investigates three multi-objective MSP formulations, which we refer to as \\textbf{objective soup recipes}. These formulations apply multi-objective optimization at different optimization levels to mitigate potential conflicts among all objectives. To ensure efficiency, we introduce a lightweight layer-selection mechanism that computes the conflict-avoiding gradient using only the most problematic layers, minimizing computational and memory overhead. Extensive experiments on CoVoST v2, LibriSpeech, and AISHELL-1 reveal that a bi-level recipe separating recognition and translation tasks consistently outperforms standard flat optimization. Our work demonstrates that hierarchical MOO is a more effective and scalable approach for building state-of-the-art MSP models. Our code has been released at https://github.com/afmsaif/Objective_Soups.","short_abstract":"Training a single model for multilingual, multi-task speech processing (MSP) is severely hampered by conflicting objectives between tasks like speech recognition and translation. While multi-objective optimization (MOO) aims to align gradient updates, its effectiveness diminishes as the number of tasks grows, making it...","url_abs":"https://arxiv.org/abs/2508.09228","url_pdf":"https://arxiv.org/pdf/2508.09228v1","authors":"[\"A F M Saif\",\"Lisha Chen\",\"Xiaodong Cui\",\"Songtao Lu\",\"Brian Kingsbury\",\"Tianyi Chen\"]","published":"2025-08-12T07:01:09Z","proceeding":"eess.AS","tasks":"[\"eess.AS\",\"cs.LG\",\"math.OC\",\"stat.ML\"]","methods":"[]","has_code":false,"code_links":[{"ID":610952,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_id":2883106,"paper_url":"https://arxiv.org/abs/2508.09228","paper_title":"Objective Soups: Multilingual Multi-Task Modeling for Speech Processing","repo_url":"https://github.com/afmsaif/Objective_Soups","is_official":false,"mentioned_in_paper":false,"mentioned_in_github":true,"github_stars":0}]}