{"ID":2878935,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2508.17393","arxiv_id":"2508.17393","title":"Agent-Testing Agent: A Meta-Agent for Automated Testing and Evaluation of Conversational AI Agents","abstract":"LLM agents are increasingly deployed to plan, retrieve, and write with tools, yet evaluation still leans on static benchmarks and small human studies. We present the Agent-Testing Agent (ATA), a meta-agent that combines static code analysis, designer interrogation, literature mining, and persona-driven adversarial test generation whose difficulty adapts via judge feedback. Each dialogue is scored with an LLM-as-a-Judge (LAAJ) rubric and used to steer subsequent tests toward the agent's weakest capabilities. On a travel planner and a Wikipedia writer, the ATA surfaces more diverse and severe failures than expert annotators while matching severity, and finishes in 20--30 minutes versus ten-annotator rounds that took days. Ablating code analysis and web search increases variance and miscalibration, underscoring the value of evidence-grounded test generation. The ATA outputs quantitative metrics and qualitative bug reports for developers. We release the full methodology and open-source implementation for reproducible agent testing: https://github.com/KhalilMrini/Agent-Testing-Agent","short_abstract":"LLM agents are increasingly deployed to plan, retrieve, and write with tools, yet evaluation still leans on static benchmarks and small human studies. We present the Agent-Testing Agent (ATA), a meta-agent that combines static code analysis, designer interrogation, literature mining, and persona-driven adversarial test...","url_abs":"https://arxiv.org/abs/2508.17393","url_pdf":"https://arxiv.org/pdf/2508.17393v1","authors":"[\"Sameer Komoravolu\",\"Khalil Mrini\"]","published":"2025-08-24T15:02:13Z","proceeding":"cs.CL","tasks":"[\"cs.CL\",\"cs.AI\"]","methods":"[\"Large Language Model\"]","has_code":false,"code_links":[{"ID":610528,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_id":2878935,"paper_url":"https://arxiv.org/abs/2508.17393","paper_title":"Agent-Testing Agent: A Meta-Agent for Automated Testing and Evaluation of Conversational AI Agents","repo_url":"https://github.com/KhalilMrini/Agent-Testing-Agent","is_official":false,"mentioned_in_paper":false,"mentioned_in_github":true,"github_stars":0}]}