{"ID":2892499,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2507.17773","arxiv_id":"2507.17773","title":"MultiKernelBench: A Multi-Platform Benchmark for Kernel Generation","abstract":"The automatic generation of deep learning (DL) kernels using large language models (LLMs) has emerged as a promising approach to reduce the manual effort and hardware-specific expertise required for writing high-performance operator implementations. However, existing benchmarks for evaluating LLMs in this domain suffer from limited hardware support, coarse-grained kernel categorization, and imbalanced task coverage. To address these limitations, we introduce MultiKernelBench, the first comprehensive, multi-platform benchmark for LLM-based DL kernel generation. MultiKernelBench spans 285 tasks across 14 well-defined kernel categories and supports three major hardware platforms: Nvidia GPUs, Huawei NPUs, and Google TPUs. To enable future extensibility, we design a modular backend abstraction layer that decouples platform-specific logic from the core benchmarking infrastructure, allowing easy integration of new hardware platforms. We further propose a simple yet effective category-aware one-shot prompting method that improves generation quality by providing in-category exemplars. Through systematic evaluations of seven state-of-the-art LLMs, we reveal significant variation in task difficulty, poor generalization to platforms with less training exposure, and the effectiveness of targeted prompting strategies. MultiKernelBench is publicly available at https://github.com/wzzll123/MultiKernelBench.","short_abstract":"The automatic generation of deep learning (DL) kernels using large language models (LLMs) has emerged as a promising approach to reduce the manual effort and hardware-specific expertise required for writing high-performance operator implementations. However, existing benchmarks for evaluating LLMs in this domain suffer...","url_abs":"https://arxiv.org/abs/2507.17773","url_pdf":"https://arxiv.org/pdf/2507.17773v2","authors":"[\"Zhongzhen Wen\",\"Yinghui Zhang\",\"Zhong Li\",\"Zhongxin Liu\",\"Linna Xie\",\"Tian Zhang\"]","published":"2025-07-20T00:58:33Z","proceeding":"cs.DC","tasks":"[\"cs.DC\",\"cs.LG\",\"cs.PF\",\"cs.SE\"]","methods":"[\"Large Language Model\",\"Language Model\"]","has_code":false,"code_links":[{"ID":611990,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_id":2892499,"paper_url":"https://arxiv.org/abs/2507.17773","paper_title":"MultiKernelBench: A Multi-Platform Benchmark for Kernel Generation","repo_url":"https://github.com/wzzll123/MultiKernelBench","is_official":false,"mentioned_in_paper":false,"mentioned_in_github":true,"github_stars":0}]}