IslamicMMLU Leaderboard

[Walid Magdy]

Dear SigArabs,

It is my pleasure to announce the IslamicMMLU Leaderboard on Huggingface.

The leaderboard uses a set of 10,000+ MCQ questions to evaluate LLMs on Islamic knowledge. The questions examine the LLMs' knowledge on 3 main topics: 1) Quran, 2) Hadith, and 3) Fiqh (Islamic jurisprudence).

It also measures the Fiqh school preference of each model from the main four Sunni madhabs (Shafie, Hanafi, Hanbali, and Maliki).

Currently, the leaderboard ranks 26 models according to their Islamic knowledge, and it accepts the addition of any new models through APIs. So, feel free to add any models you want to the leaderboard.

In a nutshell about the best models, the top 10 models are all frontier models, where Gemini family comes in the top 3, followed by GPT, followed by Claude.

Regarding the Arabic specific models, Fanar-Sadiq comes as the top Arabic model (at rank 11), followed by Fanar, Jais, Allam, and Fanar C-2. 

The effort is based on the efforts of amazing students at the University of Edinburgh, including undergrad (Ali Abdelaal), master’s (Nader Al Haffar), and PhD (Mahmoud Fawzi) students. In addition, we have a special thanks to QCRI who kindly provided a link to a professor of comparative Fiqh from Azhar University who kindly revised samples of the questions confirming its high accuracy.

For more details about the efforts, the types of questions, the distractors within answers, methodologies used, and full list of results and analysis, please check our preprint paper on arXiv.

IslamicMMLU is not a final dataset or a static leaderboard, it is just a start. I really hope everyone to contribute to this effort with one of the following:

1. Include more models to the leaderboard
2. Create questions in domains of Islamic knowledge, such as Sira, History, Aqeeda …etc.
3. Help coverage of more schools of Fiqh, such as Shia madhabs
4. Share the word about the leaderboard with everyone who might be interested in the domain.

May Allah make this work useful for everyone using LLMs for Islamic purposes.

Jazakum Allah khairan,

Walid

Walid Magdy,

Professor,

The School of Informatics,

The University of Edinburgh

The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Is e buidheann carthannais a th’ ann an Oilthigh Dhùn Èideann, clàraichte an Alba, àireamh clàraidh SC005336.

[Hamdy S. Hussein]

Great work, dear Walid!

Congratulations to the whole team 👍.

Hamdy S. Hussein

Principal Software Engineer

Qatar Computing Research Institute

+974 445 41679

www.hbku.edu.qa

---

From: 'Walid Magdy' via SIGARAB: Special Interest Group on Arabic Natural Language Processing <sig...@googlegroups.com>
Sent: Thursday, March 26, 2026 10:08 AM
To: sig...@googlegroups.com <sig...@googlegroups.com>
Subject: [SIGARAB] IslamicMMLU Leaderboard

 

--
You received this message because you are subscribed to the Google Groups "SIGARAB: Special Interest Group on Arabic Natural Language Processing" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sigarab+u...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/sigarab/DU0PR05MB92381562E4CADC2EA30DD4FAA949A%40DU0PR05MB9238.eurprd05.prod.outlook.com.

[Mona Diab]

Ma shaa Allah 

Congratulations on creating such a valuable platform and data set

Cheers 

Mona

On Mar 26, 2026, at 00:08, 'Walid Magdy' via SIGARAB: Special Interest Group on Arabic Natural Language Processing <sig...@googlegroups.com> wrote:



--