{"ID":2863256,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2509.24239","arxiv_id":"2509.24239","title":"ChessArena: A Chess Testbed for Evaluating Strategic Reasoning Capabilities of Large Language Models","abstract":"Recent large language models (LLMs) have shown strong reasoning capabilities. However, a critical question remains: do these models possess genuine strategic reasoning, or do they primarily excel at pattern recognition? To address this, we present ChessArena, a chess-based testbed for evaluating LLMs. Chess demands strategic reasoning, precise rule adherence, and the ability to track complex game states. ChessArena is a competitive framework where LLMs play against each other under four play modes. We evaluate 13 LLMs across over 800 games, testing basic understanding, move selection, and puzzle solving. Results reveal significant shortcomings: no model beats Maia-1100 (human amateur level), and some lose to random play. We also present a strong baseline: our fine-tuned Qwen3-8B substantially improves performance, approaching much larger state-of-the-art reasoning models.","short_abstract":"Recent large language models (LLMs) have shown strong reasoning capabilities. However, a critical question remains: do these models possess genuine strategic reasoning, or do they primarily excel at pattern recognition? To address this, we present ChessArena, a chess-based testbed for evaluating LLMs. Chess demands str...","url_abs":"https://arxiv.org/abs/2509.24239","url_pdf":"https://arxiv.org/pdf/2509.24239v4","authors":"[\"Jincheng Liu\",\"Sijun He\",\"Jingjing Wu\",\"Xiangsen Wang\",\"Yang Chen\",\"Zhaoqi Kuang\",\"Siqi Bao\",\"Yuan Yao\"]","published":"2025-09-29T03:24:48Z","proceeding":"cs.LG","tasks":"[\"cs.LG\",\"cs.AI\"]","methods":"[\"Large Language Model\",\"Language Model\"]","has_code":false}