{"ID":2896269,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2507.07988","arxiv_id":"2507.07988","title":"Automating Expert-Level Medical Reasoning Evaluation of Large Language Models","abstract":"As large language models (LLMs) become increasingly integrated into clinical decision-making, ensuring transparent and trustworthy reasoning is essential. However, existing evaluation strategies of LLMs' medical reasoning capability either suffer from unsatisfactory assessment or poor scalability, and a rigorous benchmark remains lacking. To address this, we introduce MedThink-Bench, a benchmark designed for rigorous, explainable, and scalable assessment of LLMs' medical reasoning. MedThink-Bench comprises 500 challenging questions across ten medical domains, each annotated with expert-crafted step-by-step rationales. Building on this, we propose LLM-w-Ref, a novel evaluation framework that leverages fine-grained rationales and LLM-as-a-Judge mechanisms to assess intermediate reasoning with expert-level fidelity while maintaining scalability. Experiments show that LLM-w-Ref exhibits a strong positive correlation with expert judgments. Benchmarking twelve state-of-the-art LLMs, we find that smaller models (e.g., MedGemma-27B) can surpass larger proprietary counterparts (e.g., OpenAI-o3). Overall, MedThink-Bench offers a foundational tool for evaluating LLMs' medical reasoning, advancing their safe and responsible deployment in clinical practice.","short_abstract":"As large language models (LLMs) become increasingly integrated into clinical decision-making, ensuring transparent and trustworthy reasoning is essential. However, existing evaluation strategies of LLMs' medical reasoning capability either suffer from unsatisfactory assessment or poor scalability, and a rigorous benchm...","url_abs":"https://arxiv.org/abs/2507.07988","url_pdf":"https://arxiv.org/pdf/2507.07988v1","authors":"[\"Shuang Zhou\",\"Wenya Xie\",\"Jiaxi Li\",\"Zaifu Zhan\",\"Meijia Song\",\"Han Yang\",\"Cheyenna Espinoza\",\"Lindsay Welton\",\"Xinnie Mai\",\"Yanwei Jin\",\"Zidu Xu\",\"Yuen-Hei Chung\",\"Yiyun Xing\",\"Meng-Han Tsai\",\"Emma Schaffer\",\"Yucheng Shi\",\"Ninghao Liu\",\"Zirui Liu\",\"Rui Zhang\"]","published":"2025-07-10T17:58:26Z","proceeding":"cs.CL","tasks":"[\"cs.CL\"]","methods":"[\"Large Language Model\",\"Language Model\"]","has_code":false}