{"ID":2835101,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_url":"https://arxiv.org/abs/2512.00333","arxiv_id":"2512.00333","title":"IndicParam: Benchmark to evaluate LLMs on low-resource Indic Languages","abstract":"While large language models excel on high-resource multilingual tasks, low- and extremely low-resource Indic languages remain severely under-evaluated. We present IndicParam, a human-curated benchmark of over 13,000 multiple-choice questions covering 11 such languages (Nepali, Gujarati, Marathi, Odia as low-resource; Dogri, Maithili, Rajasthani, Sanskrit, Bodo, Santali, Konkani as extremely low-resource) plus Sanskrit-English code-mixed set. We evaluated 20 LLMs, both proprietary and open-weights, which reveals that even the top-performing \\texttt{Gemini-2.5} reaches 58\\% average accuracy, followed by \\texttt{GPT-5} (45) and \\texttt{DeepSeek-3.2} (43.1). We additionally label each question as knowledge-oriented or purely linguistic to discriminate factual recall from grammatical proficiency. Further, we assess the ability of LLMs to handle diverse question formats-such as list-based matching, assertion-reason pairs, and sequence ordering-alongside conventional multiple-choice questions. \\benchmark\\ provides insights into limitations of cross-lingual transfer and establishes a challenging benchmark for Indic languages. The dataset is available at https://huggingface.co/datasets/bharatgenai/IndicParam. Scripts to run benchmark are present at https://github.com/ayushbits/IndicParam.","short_abstract":"While large language models excel on high-resource multilingual tasks, low- and extremely low-resource Indic languages remain severely under-evaluated. We present IndicParam, a human-curated benchmark of over 13,000 multiple-choice questions covering 11 such languages (Nepali, Gujarati, Marathi, Odia as low-resource; D...","url_abs":"https://arxiv.org/abs/2512.00333","url_pdf":"https://arxiv.org/pdf/2512.00333v2","authors":"[\"Ayush Maheshwari\",\"Kaushal Sharma\",\"Vivek Patel\",\"Aditya Maheshwari\"]","published":"2025-11-29T05:49:50Z","proceeding":"cs.CL","tasks":"[\"cs.CL\"]","methods":"[\"Large Language Model\",\"Language Model\"]","has_code":false,"code_links":[{"ID":606485,"CreatedAt":"2026-06-01T04:54:23.091178241Z","UpdatedAt":"2026-06-01T04:54:23.091178241Z","DeletedAt":null,"paper_id":2835101,"paper_url":"https://arxiv.org/abs/2512.00333","paper_title":"IndicParam: Benchmark to evaluate LLMs on low-resource Indic Languages","repo_url":"https://github.com/ayushbits/IndicParam","is_official":false,"mentioned_in_paper":false,"mentioned_in_github":true,"github_stars":0}]}