Arabic Stemmer

Arabic Stemmer (Stem)

A Stem is a computational algorithm that returns words with the same root to a common form by removing their prefixes and suffixes. For example, the Arabic words «یعلمون»، «سیعلمان»، «یعلمن»، «فیعلمونه» and «أیعلم؟» — all derived from the root «علم» (knowledge) — share the common stem «یعلم».

The Necessity of a Stemmer Specific to Arabic

Achieving desirable results in syntax and semantics layers depends on accurate processing at the morphological analysis layer. Research has shown that using the Stem pattern for morphological analysis of words is more effective in intelligent processes than other patterns such as Root and Lemma.

The complexities of the Arabic language have meant that none of the stemmers used in other languages, with their different approaches, can achieve exemplary success in Arabic.

Challenges of Building a Stemmer in Arabic

• Incorrect removal of affixes (e.g., «والده»)
• Ambiguity caused by diacritics (e.g., «بکر»)
• Single-answer ambiguity (e.g., «لهم»)
• Structural changes in Arabic words (e.g., «قلت و مررت»)
• Irregular structures and auditory (Sama'i) forms

Stages of Developing Noor Stemmer

1. Word Refinement - Separating non-Arabic words using linguistic tags (e.g., «سبزوار») - Supporting different writing versions of the Quran (e.g., «یسمریّ»، «بقیت الله»، «امرأت نوح»)

2. Affix Calculation - Implementing specific rules such as non-repetition of prefixes and suffixes, non-occurrence of the prefix «س» with other prefixes, and non-combination of sound plural suffixes with each other

3. Root Matching Pattern

1. Usage Verification Pattern

1. Matching Dictionary Lexical Entries

1. Implementation of I'lal (Weak Letter) and Idgham (Assimilation) Rules - For example, the rule of removing the third letter in words where the second and third root letters are identical

1. Using Lexical Network to Construct Stems from Broken Plurals - These plurals constitute approximately 10% of Arabic texts and about 41% of all Arabic plurals.