{"ID":2879823,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2508.15478","arxiv_id":"2508.15478","title":"SLM-Bench: A Comprehensive Benchmark of Small Language Models on Environmental Impacts--Extended Version","abstract":"Small Language Models (SLMs) offer computational efficiency and accessibility, yet a systematic evaluation of their performance and environmental impact remains lacking. We introduce SLM-Bench, the first benchmark specifically designed to assess SLMs across multiple dimensions, including accuracy, computational efficiency, and sustainability metrics. SLM-Bench evaluates 15 SLMs on 9 NLP tasks using 23 datasets spanning 14 domains. The evaluation is conducted on 4 hardware configurations, providing a rigorous comparison of their effectiveness. Unlike prior benchmarks, SLM-Bench quantifies 11 metrics across correctness, computation, and consumption, enabling a holistic assessment of efficiency trade-offs. Our evaluation considers controlled hardware conditions, ensuring fair comparisons across models. We develop an open-source benchmarking pipeline with standardized evaluation protocols to facilitate reproducibility and further research. Our findings highlight the diverse trade-offs among SLMs, where some models excel in accuracy while others achieve superior energy efficiency. SLM-Bench sets a new standard for SLM evaluation, bridging the gap between resource efficiency and real-world applicability.","short_abstract":"Small Language Models (SLMs) offer computational efficiency and accessibility, yet a systematic evaluation of their performance and environmental impact remains lacking. We introduce SLM-Bench, the first benchmark specifically designed to assess SLMs across multiple dimensions, including accuracy, computational efficie...","url_abs":"https://arxiv.org/abs/2508.15478","url_pdf":"https://arxiv.org/pdf/2508.15478v2","authors":"[\"Nghiem Thanh Pham\",\"Tung Kieu\",\"Duc-Manh Nguyen\",\"Son Ha Xuan\",\"Nghia Duong-Trung\",\"Danh Le-Phuoc\"]","published":"2025-08-21T11:56:05Z","proceeding":"cs.CL","tasks":"[\"cs.CL\",\"cs.CY\",\"cs.PF\"]","methods":"[\"Language Model\"]","has_code":false}