Persian Part-of-Speech Tagging

Persian Part-of-Speech (POS) Tagging / Morphological Tagging

In recent years, computational linguistics has become a major concern for researchers in computer science and linguistics. The use of computers and intelligent tools has made it possible to perform many text-related tasks with remarkable speed and accuracy. Furthermore, it has provided the power to enter areas that were previously difficult to imagine, such as intelligent translation, semantic search engines, and many other applications. Any language in the world can be the subject of linguistic processing.

What is Part-of-Speech Tagging?

Part-of-Speech (POS) tagging means obtaining the morphological type (word class) of each word in a text. For example, assuming we have the following tagset:

Table 1-1: A simple example of word morphological type tags

The equivalent tags for words in the phrase "اولین سیاره خارج از منظومه شمسی دیده شد" (The first planet outside the solar system was seen) would be as follows:

Implementation Method: Hidden Markov Model (HMM) Tagger

Using the Hidden Markov Model for POS tagging can be considered an approach based on probability theory. This process is implemented as follows:

The question is: For a given sequence of words, what tag sequence is the best tag sequence for that word sequence?

If we represent the input text (a sequence of morphological units) as W = (wi)1<i<n, and represent a sequence of tags from the tagset as T = (ti)1<i<n, our goal is to calculate the following:

The above relation represents the probability that the tag sequence T belongs to the word sequence W. Using Bayes' rule and removing P(W), the relation can be transformed as follows:

P(T|W) represents the probability of occurrence of this tag sequence (probabilities related to tag transitions) and can be calculated using the N-gram model:

Normalization method:

Such that:

The calculation method is as follows:

Training Data and Results

For this project's experiments, we used the Bijankhan Corpus, a standard Persian text corpus. This corpus contains approximately 2.6 million (2,597,937) tagged words from the Hamshahri corpus.

The tagset used in this corpus consists of 40 tags, each representing a specific morphological type. For testing, we selected 2 million words as training data and the remaining as test data. Among the first 2 million words, 39 out of 40 tags were observed, and the tagset was modified accordingly.

Evaluation Results

Table 3-2: Precision, Recall, and F-measure for the entire project