{"ID":2880877,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2508.14279","arxiv_id":"2508.14279","title":"GRILE: A Benchmark for Grammar Reasoning and Explanation in Romanian LLMs","abstract":"LLMs (Large language models) have revolutionized NLP (Natural Language Processing), yet their pedagogical value for low-resource languages remains unclear. We present GRILE (Grammar Romanian Inference and Language Explanations) , the first open benchmark of 1,151 multiple-choice questions harvested from Romanian high-stakes exams (National Evaluation, Baccalaureate, university admissions). GRILE enables us to probe two complementary abilities of seven state-of-the-art multilingual and Romanian-specific LLMs: (i) selecting the correct answer, and (ii) producing linguistically accurate explanations. While Gemini 2.5 Pro reaches 83% accuracy, most open-weight models stay below 65%, and 48% of their explanations contain factual or pedagogical flaws according to expert review. A detailed error analysis pinpoints systematic weaknesses in morphology and in applying the latest DOOM3 orthographic norms. All data, code and a public web demo are released to catalyze future research. Our findings expose open challenges for trustworthy educational NLP in low-resource settings and establish GRILE as a new test-bed for controllable explanation generation and evaluation.","short_abstract":"LLMs (Large language models) have revolutionized NLP (Natural Language Processing), yet their pedagogical value for low-resource languages remains unclear. We present GRILE (Grammar Romanian Inference and Language Explanations) , the first open benchmark of 1,151 multiple-choice questions harvested from Romanian high-s...","url_abs":"https://arxiv.org/abs/2508.14279","url_pdf":"https://arxiv.org/pdf/2508.14279v1","authors":"[\"Adrian-Marius Dumitran\",\"Alexandra-Mihaela Danila\",\"Angela-Liliana Dumitran\"]","published":"2025-08-19T21:27:06Z","proceeding":"cs.CL","tasks":"[\"cs.CL\",\"cs.CY\",\"cs.LG\"]","methods":"[\"Large Language Model\",\"Language Model\"]","has_code":false}