{"ID":2862741,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2509.26165","arxiv_id":"2509.26165","title":"Human-MME: A Holistic Evaluation Benchmark for Human-Centric Multimodal Large Language Models","abstract":"Multimodal Large Language Models (MLLMs) have demonstrated significant advances in visual understanding tasks. However, their capacity to comprehend human-centric scenes has rarely been explored, primarily due to the absence of comprehensive evaluation benchmarks that take into account both the human-oriented granular level and higher-dimensional causal reasoning ability. Such high-quality evaluation benchmarks face tough obstacles, given the physical complexity of the human body and the difficulty of annotating granular structures. In this paper, we propose Human-MME, a curated benchmark designed to provide a more holistic evaluation of MLLMs in human-centric scene understanding. Compared with other existing benchmarks, our work provides three key features: 1. Diversity in human scene, spanning 4 primary visual domains with 15 secondary domains and 43 sub-fields to ensure broad scenario coverage. 2. Progressive and diverse evaluation dimensions, evaluating the human-based activities progressively from the human-oriented granular perception to the higher-dimensional reasoning, consisting of eight dimensions with 19,945 real-world image question pairs and an evaluation suite. 3. High-quality annotations with rich data paradigms, constructing the automated annotation pipeline and human-annotation platform, supporting rigorous manual labeling to facilitate precise and reliable model assessment. Our benchmark extends the single-target understanding to the multi-person and multi-image mutual understanding by constructing the choice, short-answer, grounding, ranking and judgment question components, and complex questions of their combination. The extensive experiments on 17 state-of-the-art MLLMs effectively expose the limitations and guide future MLLMs research toward better human-centric image understanding. All data and code are available at https://github.com/Yuan-Hou/Human-MME.","short_abstract":"Multimodal Large Language Models (MLLMs) have demonstrated significant advances in visual understanding tasks. However, their capacity to comprehend human-centric scenes has rarely been explored, primarily due to the absence of comprehensive evaluation benchmarks that take into account both the human-oriented granular...","url_abs":"https://arxiv.org/abs/2509.26165","url_pdf":"https://arxiv.org/pdf/2509.26165v3","authors":"[\"Yuansen Liu\",\"Haiming Tang\",\"Jinlong Peng\",\"Jiangning Zhang\",\"Xiaozhong Ji\",\"Qingdong He\",\"Wenbin Wu\",\"Donghao Luo\",\"Zhenye Gan\",\"Junwei Zhu\",\"Yunhang Shen\",\"Chaoyou Fu\",\"Chengjie Wang\",\"Xiaobin Hu\",\"Shuicheng Yan\"]","published":"2025-09-30T12:20:57Z","proceeding":"cs.CV","tasks":"[\"cs.CV\"]","methods":"[\"Large Language Model\",\"Language Model\"]","has_code":false,"code_links":[{"ID":608937,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_id":2862741,"paper_url":"https://arxiv.org/abs/2509.26165","paper_title":"Human-MME: A Holistic Evaluation Benchmark for Human-Centric Multimodal Large Language Models","repo_url":"https://github.com/Yuan-Hou/Human-MME","is_official":false,"mentioned_in_paper":false,"mentioned_in_github":true,"github_stars":0}]}