{"ID":2881587,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2508.13201","arxiv_id":"2508.13201","title":"Benchmarking LLM-based agents for single-cell omics analysis","abstract":"Background: The surge in single-cell omics data exposes limitations in traditional, manually defined analysis workflows. AI agents offer a paradigm shift, enabling adaptive planning, executable code generation, traceable decisions, and real-time knowledge fusion. However, the lack of a comprehensive benchmark critically hinders progress. Results: We introduce a novel benchmarking evaluation system to rigorously assess agent capabilities in single-cell omics analysis. This system comprises: a unified platform compatible with diverse agent frameworks and LLMs; multidimensional metrics assessing cognitive program synthesis, collaboration, execution efficiency, bioinformatics knowledge integration, and task completion quality; and 50 diverse real-world single-cell omics analysis tasks spanning multi-omics, species, and sequencing technologies. Our evaluation reveals that Grok3-beta achieves state-of-the-art performance among tested agent frameworks. Multi-agent frameworks significantly enhance collaboration and execution efficiency over single-agent approaches through specialized role division. Attribution analyses of agent capabilities identify that high-quality code generation is crucial for task success, and self-reflection has the most significant overall impact, followed by retrieval-augmented generation (RAG) and planning. Conclusions: This work highlights persistent challenges in code generation, long-context handling, and context-aware knowledge retrieval, providing a critical empirical foundation and best practices for developing robust AI agents in computational biology.","short_abstract":"Background: The surge in single-cell omics data exposes limitations in traditional, manually defined analysis workflows. AI agents offer a paradigm shift, enabling adaptive planning, executable code generation, traceable decisions, and real-time knowledge fusion. However, the lack of a comprehensive benchmark criticall...","url_abs":"https://arxiv.org/abs/2508.13201","url_pdf":"https://arxiv.org/pdf/2508.13201v3","authors":"[\"Yang Liu\",\"Lu Zhou\",\"Xiawei Du\",\"Ruikun He\",\"Xuguang Zhang\",\"Rongbo Shen\",\"Yixue Li\"]","published":"2025-08-16T04:26:18Z","proceeding":"q-bio.GN","tasks":"[\"q-bio.GN\",\"cs.AI\",\"cs.MA\"]","methods":"[\"RAG\",\"Large Language Model\"]","has_code":false}