CFP: IslamicEval shared-task, at ArabicNLP 2025
Walid Magdy
السلام عليكم ورحمة الله وبركاته،
We are happy to announce the IslamicEval Shared-Task as part of ArabicNLP 2025.
LLMs are increasingly utilized in Arabic NLP applications, with recent models such as Jais, Allam, and Fanar demonstrating strong performance across various tasks. However, a key challenge remains: Hallucination!
This issue is particularly critical in religious contexts, where hallucination in Arabic LLMs can affect Quran and Hadith, through misquotations or fabricated content, posing significant ethical, theological, and social risks by spreading misinformation and undermining trust in AI systems.
To address this critical challenge, we introduce the first IslamicEval shared task, featuring a set of subtasks designed to tackle hallucination in LLM responses to Islamic questions. Our goal is to provide the NLP community with the resources needed to develop tools that can detect and correct hallucinated outputs related to Islamic texts in the age of LLMs. In this first edition, we offer four subtasks that move us in this direction:
Subtask 1A: Identifying Intended Ayahs and Hadiths: Given an LLM-generated response, participants must mark the character spans corresponding to intended Ayahs (Quranic verses) or Hadiths (Prophetic sayings).
Subtask 1B: Validating Content Accuracy: For each identified Ayah or Hadith, participants will label it as correct or incorrect based on trusted Islamic sources. Diacritic errors count as mistakes.
Subtask 1C: Correcting Errors: Participants will correct any inaccurate Ayahs or Hadiths using the provided official sources. Full Ayahs or Hadiths are required, including accurate writing and diacritics.
Subtask 2: Qur’an and Hadith QA: Given a question about an Islamic topic, systems must return a ranked list of relevant Ayahs or Hadiths (from Sahih Bukhari) that answer the question.
Important Dates
16 June 2025: Registration opens + Dataset released
20 July 2025: Registration closes + Test set released
For more details and participation, please visit
the shared-task webpage:
https://sites.google.com/view/islamiceval-2025
نفع الله بكم وبعلمكم فيما يحبه ويرضاه
IslamicEval Organigers
Walid Magdy,
The University of Edinburgh
Walid Magdy
السلام عليكم ورحمة الله وبركاته،
We are happy to announce the following on the IslamicEval Shared-Task:
- Test set of the shared task and submission website are now released.
- Subtask 1A: Identifying Intended Ayahs and Hadiths: https://www.codabench.org/competitions/9820/
- Subtask 1B: Validating Content Accuracy: https://www.codabench.org/competitions/9822/
- Subtask 1C: Correcting Errors: https://www.codabench.org/competitions/9824/
- Subtask 2: Qur’an and Hadith QA: https://www.codabench.org/competitions/9939/
- We extended the registration deadline till 6th of August, for those interested in the task.
- Deadline for submitting the runs is the 8th of August.
Full details can be found on the shared task website here.
We look forward for your participation 😊
IslamicEval Organisers