Current state of the art for contextual Arabic lemmatization in search?
82 views
Skip to first unread message
Mustafa J
Jul 18, 2026, 11:16:05 AMJul 18
to sig...@googlegroups.com
Dear SIGARAB,
We are improving full-text search over a large corpus of Classical Arabic, heritage texts, and MSA. We need a dependable lemma for each query and corpus token, with richer morphology such as roots available as an optional search primitive. Accuracy is the priority, but the system must also be reproducible and practical for indexing millions of documents and serving short queries.
On a small search-oriented sample, we compared CAMeL Tools MLE and unfactored BERT with SinaLab Alma. CAMeL BERT was only modestly better overall. It makes clear repairs—for example, MLE reads `أبيه` as the proper name Abbé, while BERT returns `أب` (father)—but its contextual behavior is not consistently better. For example:
- `أَعُوذُ` → `عاذ` (correct)
- `قُلْ أَعُوذُ` → generated proper-noun backoff `اعوذ`
- `قُلْ أَعُوذُ بِرَبِّ` → `عاذ` again
The valid `عاذ` analysis remains available but is ranked second in the failing case. We reproduced the same ranking with CAMeL Tools 1.5.7 and 1.6.0 using the current MSA unfactored model and with the pretrained ranking cache disabled.
What is currently the strongest model or system for contextual Arabic lemma selection for this use case, especially across Classical Arabic and MSA? Is CAMeL's unfactored BERT still the best maintained option, or are there newer open or licensed models we should evaluate? We would also appreciate recommendations for representative benchmarks, particularly ones that test clitics, diacritics, historical prose, and lemma selection in context.
We can share the frozen examples, complete ranked analyses, and comparison artifacts.
Best,
Mustafa
We are improving full-text search over a large corpus of Classical Arabic, heritage texts, and MSA. We need a dependable lemma for each query and corpus token, with richer morphology such as roots available as an optional search primitive. Accuracy is the priority, but the system must also be reproducible and practical for indexing millions of documents and serving short queries.
On a small search-oriented sample, we compared CAMeL Tools MLE and unfactored BERT with SinaLab Alma. CAMeL BERT was only modestly better overall. It makes clear repairs—for example, MLE reads `أبيه` as the proper name Abbé, while BERT returns `أب` (father)—but its contextual behavior is not consistently better. For example:
- `أَعُوذُ` → `عاذ` (correct)
- `قُلْ أَعُوذُ` → generated proper-noun backoff `اعوذ`
- `قُلْ أَعُوذُ بِرَبِّ` → `عاذ` again
The valid `عاذ` analysis remains available but is ranked second in the failing case. We reproduced the same ranking with CAMeL Tools 1.5.7 and 1.6.0 using the current MSA unfactored model and with the pretrained ranking cache disabled.
What is currently the strongest model or system for contextual Arabic lemma selection for this use case, especially across Classical Arabic and MSA? Is CAMeL's unfactored BERT still the best maintained option, or are there newer open or licensed models we should evaluate? We would also appreciate recommendations for representative benchmarks, particularly ones that test clitics, diacritics, historical prose, and lemma selection in context.
We can share the frozen examples, complete ranked analyses, and comparison artifacts.
Best,
Mustafa
Nizar Habash
Jul 18, 2026, 12:56:42 PMJul 18
to Mustafa J, sig...@googlegroups.com
Hi Mustafa. Everyone.
The best results I am aware of are the works of Saaed and Habash (2025a,b) which are not yet integrated in the public release of Camel Tools. GitHub links are in the papers. We are targeting this Fall for the new release.
(1) Mostafa Saeed and Nizar Habash. 2025. Lemmatization as a Classification Task: Results from Arabic across Multiple Genres. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 30014–30029, Suzhou, China.
Finally, there is a far more superior analyzer to the basic version provided in camel tools: camel morph MSA which covers most of the known gaps in Aramorph and SAMA. Thus can be used as is in camel tools but we have not yet announced the release as we are making some adjustments to integrate properly.
(3) Christian Khairallah, Salam Khalifa, Reham Marzouk, Mayar Nassar, and Nizar Habash. 2024. Camel Morph MSA: A Large-Scale Open-Source Morphological Analyzer for Modern Standard Arabic. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 2683–2691, Torino, Italia.
Best,
Nizar
Nizar
--
You received this message because you are subscribed to the Google Groups "SIGARAB: Special Interest Group on Arabic Natural Language Processing" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sigarab+u...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/sigarab/CANXyXnvbSWQsNywLr%3DrcVMeXpdk9B7Ysc%3D1tBm1aBjmntFLHuQ%40mail.gmail.com.
Kareem Darwish
Jul 18, 2026, 4:20:56 PMJul 18
to Nizar Habash, Mustafa J, sig...@googlegroups.com
For search, it would be good to test with bpe and with LLM embeddings. I have a very strong feeling that bpe in combination with embeddings (or just bpe) would significantly outperform lemmas. My two cents.
Kareem
From: 'Nizar Habash' via SIGARAB: Special Interest Group on Arabic Natural Language Processing <sig...@googlegroups.com>
Sent: Saturday, 18 July 2026 19:56:27
To: Mustafa J <musta...@gmail.com>
Cc: sig...@googlegroups.com <sig...@googlegroups.com>
Subject: Re: [SIGARAB] Current state of the art for contextual Arabic lemmatization in search?
Sent: Saturday, 18 July 2026 19:56:27
To: Mustafa J <musta...@gmail.com>
Cc: sig...@googlegroups.com <sig...@googlegroups.com>
Subject: Re: [SIGARAB] Current state of the art for contextual Arabic lemmatization in search?
To view this discussion visit
https://groups.google.com/d/msgid/sigarab/CAFfBGVm%2BjS9K%2BUY_CdS5N1YHR%2B2zqgGcQxzHwehBL%2B05E2KZhQ%40mail.gmail.com.
Mustafa J
Jul 21, 2026, 2:26:55 PMJul 21
to Nizar Habash, sig...@googlegroups.com
Thank you for pointing us to this work. We compared CAMEL Morph MSA v1.0 with CAMeL Tools’ current calima-msa-r13 database using a fixed set of search queries and passages from a classical Arabic text collection.
Because we do not yet have a contextual selector for Morph, we compared how often each analyzer included the correct search lemma among its suggestions. Morph found the correct lemma more often:
- Search queries: 84/88 (95.5%) → 86/88 (97.7%)
- Words in context: 425/468 (90.8%) → 440/468 (94.0%)
- Unique word/lemma pairs in context: 277/315 (87.9%) → 290/315 (92.1%)
Useful additions included الطاعات → طاعة, الفتويين → فتوى, العقيقة → عقيقة, and وقرن → قر.
That broader coverage also increased the average candidate count from 1.54 to 2.97, so we looked into using the released contextual model with CAMEL Morph. Our reading of the code is that the model relies on a mapping built from CALIMA-S31. It can use CAMEL Morph analyses that are already in that mapping, but analyses missing from the mapping have no cluster and cannot be selected.
Because we do not yet have a contextual selector for Morph, we compared how often each analyzer included the correct search lemma among its suggestions. Morph found the correct lemma more often:
- Search queries: 84/88 (95.5%) → 86/88 (97.7%)
- Words in context: 425/468 (90.8%) → 440/468 (94.0%)
- Unique word/lemma pairs in context: 277/315 (87.9%) → 290/315 (92.1%)
Useful additions included الطاعات → طاعة, الفتويين → فتوى, العقيقة → عقيقة, and وقرن → قر.
That broader coverage also increased the average candidate count from 1.54 to 2.97, so we looked into using the released contextual model with CAMEL Morph. Our reading of the code is that the model relies on a mapping built from CALIMA-S31. It can use CAMEL Morph analyses that are already in that mapping, but analyses missing from the mapping have no cluster and cannot be selected.
Is that correct? If so, is a Morph-compatible mapping or contextual selector available now, or will one be included in the forthcoming CAMeL Tools release?
—Mustafa
Nizar Habash
Jul 23, 2026, 3:43:48 AMJul 23
to Mustafa J, sig...@googlegroups.com
Hi Mustafa -
* Thanks for the evaluation of Camel Morph MSA 1.0 on your task. Glad to see th positive value.
* Please do not refer to it as "Morph"... it's Camel-Morph-MSA-1.0 (CM-MSA-1.0 if you insist on abbreviating) -- "Morph" is hyper vague :-D.
* Your understanding of the selection is correct. We will be releasing the update with the next release.
Best
Nizar
--
Mustafa J
Jul 25, 2026, 10:19:36 AM (12 days ago) Jul 25
to Nizar Habash, sig...@googlegroups.com
Ah, my apologies Dr Nizar, thanks for the etiquette tip. :]
Looking forward to the release!
—Mustafa
Reply all
Reply to author
Forward
0 new messages