{"ID":2863269,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2509.24253","arxiv_id":"2509.24253","title":"MRAG-Suite: A Diagnostic Evaluation Platform for Visual Retrieval-Augmented Generation","abstract":"Multimodal Retrieval-Augmented Generation (Visual RAG) significantly advances question answering by integrating visual and textual evidence. Yet, current evaluations fail to systematically account for query difficulty and ambiguity. We propose MRAG-Suite, a diagnostic evaluation platform integrating diverse multimodal benchmarks (WebQA, Chart-RAG, Visual-RAG, MRAG-Bench). We introduce difficulty-based and ambiguity-aware filtering strategies, alongside MM-RAGChecker, a claim-level diagnostic tool. Our results demonstrate substantial accuracy reductions under difficult and ambiguous queries, highlighting prevalent hallucinations. MM-RAGChecker effectively diagnoses these issues, guiding future improvements in Visual RAG systems.","short_abstract":"Multimodal Retrieval-Augmented Generation (Visual RAG) significantly advances question answering by integrating visual and textual evidence. Yet, current evaluations fail to systematically account for query difficulty and ambiguity. We propose MRAG-Suite, a diagnostic evaluation platform integrating diverse multimodal...","url_abs":"https://arxiv.org/abs/2509.24253","url_pdf":"https://arxiv.org/pdf/2509.24253v3","authors":"[\"Yuelyu Ji\",\"Wuwei Lan\",\"Patrick NG\"]","published":"2025-09-29T03:55:28Z","proceeding":"cs.CL","tasks":"[\"cs.CL\"]","methods":"[\"RAG\"]","has_code":false}