{"ID":2885998,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2508.14051","arxiv_id":"2508.14051","title":"Benchmarking Sociolinguistic Diversity in Swahili NLP: A Taxonomy-Guided Approach","abstract":"We introduce the first taxonomy-guided evaluation of Swahili NLP, addressing gaps in sociolinguistic diversity. Drawing on health-related psychometric tasks, we collect a dataset of 2,170 free-text responses from Kenyan speakers. The data exhibits tribal influences, urban vernacular, code-mixing, and loanwords. We develop a structured taxonomy and use it as a lens for examining model prediction errors across pre-trained and instruction-tuned language models. Our findings advance culturally grounded evaluation frameworks and highlight the role of sociolinguistic variation in shaping model performance.","short_abstract":"We introduce the first taxonomy-guided evaluation of Swahili NLP, addressing gaps in sociolinguistic diversity. Drawing on health-related psychometric tasks, we collect a dataset of 2,170 free-text responses from Kenyan speakers. The data exhibits tribal influences, urban vernacular, code-mixing, and loanwords. We deve...","url_abs":"https://arxiv.org/abs/2508.14051","url_pdf":"https://arxiv.org/pdf/2508.14051v1","authors":"[\"Kezia Oketch\",\"John P. Lalor\",\"Ahmed Abbasi\"]","published":"2025-08-06T20:10:11Z","proceeding":"cs.CL","tasks":"[\"cs.CL\",\"cs.CY\"]","methods":"[\"Language Model\"]","has_code":false}