{"ID":2892094,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2507.15286","arxiv_id":"2507.15286","title":"Beyond Easy Wins: A Text Hardness-Aware Benchmark for LLM-generated Text Detection","abstract":"We present a novel evaluation paradigm for AI text detectors that prioritizes real-world and equitable assessment. Current approaches predominantly report conventional metrics like AUROC, overlooking that even modest false positive rates constitute a critical impediment to practical deployment of detection systems. Furthermore, real-world deployment necessitates predetermined threshold configuration, making detector stability (i.e. the maintenance of consistent performance across diverse domains and adversarial scenarios), a critical factor. These aspects have been largely ignored in previous research and benchmarks. Our benchmark, SHIELD, addresses these limitations by integrating both reliability and stability factors into a unified evaluation metric designed for practical assessment. Furthermore, we develop a post-hoc, model-agnostic humanification framework that modifies AI text to more closely resemble human authorship, incorporating a controllable hardness parameter. This hardness-aware approach effectively challenges current SOTA zero-shot detection methods in maintaining both reliability and stability. (Data and code: https://github.com/navid-aub/SHIELD-Benchmark)","short_abstract":"We present a novel evaluation paradigm for AI text detectors that prioritizes real-world and equitable assessment. Current approaches predominantly report conventional metrics like AUROC, overlooking that even modest false positive rates constitute a critical impediment to practical deployment of detection systems. Fur...","url_abs":"https://arxiv.org/abs/2507.15286","url_pdf":"https://arxiv.org/pdf/2507.15286v1","authors":"[\"Navid Ayoobi\",\"Sadat Shahriar\",\"Arjun Mukherjee\"]","published":"2025-07-21T06:37:27Z","proceeding":"cs.CL","tasks":"[\"cs.CL\"]","methods":"[\"Large Language Model\"]","has_code":false,"code_links":[{"ID":611954,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_id":2892094,"paper_url":"https://arxiv.org/abs/2507.15286","paper_title":"Beyond Easy Wins: A Text Hardness-Aware Benchmark for LLM-generated Text Detection","repo_url":"https://github.com/navid-aub/SHIELD-Benchmark","is_official":false,"mentioned_in_paper":false,"mentioned_in_github":true,"github_stars":0}]}