{"ID":5552827,"CreatedAt":"2026-07-02T01:54:51.863792489Z","UpdatedAt":"2026-07-03T20:14:26.82372516Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2607.00171","arxiv_id":"2607.00171","title":"ALEE: Any-Language Evaluation of Embeddings via English-Centric Minimal Pairs","abstract":"Text embeddings are standard for semantic similarity tasks, yet their evaluation remains an open challenge. Current benchmarks are static, cover only a limited set of languages, are often domain-specific, susceptible to overfitting, and poorly representative of low-resource languages. To address these limitations, we introduce ALEE, a framework that extends Sentence Smith (Li et al., 2025) to the cross-lingual and paragraph level. ALEE uses Abstract Meaning Representations (AMR) to generate English minimal pairs with controlled, fine-grained semantic shifts, which are paired with translations in target languages. This approach enables targeted diagnostics for models in any language with English parallel data. We conduct a large-scale empirical study across a diverse set of embedding models and 275+ languages spanning three parallel datasets. On ALEE, performance varies substantially across languages, text lengths, and linguistic phenomena, exposing persistent gaps in cross-lingual semantic representation that track language prevalence in training resources and subword tokenization. We release ALEE at https://github.com/Andrian0s/any-lang-embed-eval","short_abstract":"Text embeddings are standard for semantic similarity tasks, yet their evaluation remains an open challenge. Current benchmarks are static, cover only a limited set of languages, are often domain-specific, susceptible to overfitting, and poorly representative of low-resource languages. To address these limitations, we i...","url_abs":"https://arxiv.org/abs/2607.00171","url_pdf":"https://arxiv.org/pdf/2607.00171v1","authors":"[\"Andrianos Michail\",\"Stylianos Psychias\",\"Michelle Wastl\",\"Simon Clematide\",\"Rico Sennrich\",\"Juri Opitz\"]","published":"2026-06-30T20:45:17Z","proceeding":"cs.CL","tasks":"[\"cs.CL\"]","methods":"[]","has_code":false,"code_links":[{"ID":613861,"CreatedAt":"2026-07-02T01:54:51.863792489Z","UpdatedAt":"2026-07-02T01:54:51.863792489Z","DeletedAt":null,"paper_id":5552827,"paper_url":"https://arxiv.org/abs/2607.00171","paper_title":"ALEE: Any-Language Evaluation of Embeddings via English-Centric Minimal Pairs","repo_url":"https://github.com/Andrian0s/any-lang-embed-eval","is_official":false,"mentioned_in_paper":false,"mentioned_in_github":true,"github_stars":0}]}