Improving the tagging of unknown words in Persian texts using association rules

This article addresses and investigates one of the major challenges in computational linguistics: Part-of-Speech (POS) tagging of unknown words. POS tagging, being one of the most fundamental requirements for intelligent text processing, is highly dependent on the language of the text being processed. Therefore, developing a high-accuracy POS tagger for the Persian language has been a top priority for the authors. The technique employed to solve the problem of unknown words is a hybrid approach combining the Hidden Markov Model (HMM) algorithm with association rules. The HMM algorithm has been widely utilized in numerous previous POS taggers [2,3] and is considered among the best methods used in tagging systems. According to the experiments conducted in this study, applying association rules increased the accuracy of tagging unknown Persian words to 81.2%. Meanwhile, the overall cumulative accuracy of the proposed tagger reaches 98%.