Named Entity Recognition (NER)

Named Entity Recognition (NER) Engine of Noor Computer Research Center for Islamic Sciences

Entity recognition, indexing, and utilizing it as an infrastructure for other machine processes are among the most important areas requiring Named Entity Recognition (NER). Names of persons, individuals, places, institutions, and similar entities are a basic need for researchers in many fields.

For these reasons, and given the high cost of manual NER by humans, the development of a Named Entity Recognition engine at the Noor Computer Research Center for Islamic Sciences (Noor) has been prioritized.

What is Named Entity Recognition?

Today, there are multiple definitions for the term "Named Entity." In some sources, up to 19 different definitions have been provided. Generally, named entities are the names of desired entities in a given text (such as names of persons, places, drugs, diseases, etc.).

Wide Range of Applications

Named Entity Recognition has extensive applications including:
- Question answering
- Information retrieval
- Information extraction
- Trend analysis
- Document classification
- Summarization
- Automatic text tagging
- Machine translation
- Profile and entity extraction
- And many other potential applications

Traditional and Modern Approaches

To date, three traditional approaches for named entity recognition have been proposed, and modern NER systems primarily use a combination of these methods:

1. Dictionary-based methods
2. Rule-based methods
3. Machine learning-based methods

Noor Center NER Engine

The NER system of the Noor Computer Research Center for Islamic Sciences has been developed using the latest machine learning technology—Deep Learning—in both Persian and Arabic languages.

Deep Learning is a novel approach in artificial intelligence and machine learning that has become the dominant approach in various fields over the past decade. The use of AI technology allows the system to function without relying on vocabulary. This enables a single word appearing in two different contexts in two different texts to be interpreted correctly.

Figure 1: Sample of NER engine performance

The Noor Center NER engine has been developed in both Arabic and Persian. The system is designed so that, with the generation of training data alone, it can perform NER for other languages.[1]

Advantages

The use of computational processing in NER, along with AI, provides the following advantages:

• Processing massive volumes of data in very short time
• Detection of unseen named entities by considering surrounding context as evidence
• Ability to develop this engine for other languages at very low cost
• Not relying solely on vocabulary for named entity recognition

Development Steps and Applications

Based on the center's core products and their vision, the following steps are envisioned for deploying the NER engine:

1. Integrating the NER engine with the Noormags search engine to provide better results when encountering named entities
2. Creating entity pages on the Noorlib website for each book
3. Assisting in developing ontologies needed for Noor's intelligent projects

Improvement Stages

To improve the use of NER engine output, two essential steps must be taken:

1. Disambiguation
Disambiguation of named entities means that if a text contains various forms of names for a specific entity, these diverse names should be connected to prepare for the next step—entity linking.

2. Coreference Resolution
Two different individuals or two different locations may have similar names. In such cases, the ideal output is to link disambiguated entities to a unique identifier to avoid homonym errors in the information processing and retrieval process.

Conclusion

Using this engine in producing indexes, dictionaries, content analysis of information, qualitative research, and information retrieval can reduce the cost of human operations and naturally accelerate the research process.

Adding this product's output to other Noor Center products, such as the Noormags website or even Noor's desktop software, can complete the research ecosystem of Noor software.

Appendices

Appendix 1: Sample system output in Arabic

Appendix 2: Sample system output in Persian

Reference

[1] To view a demo of this product and other intelligent products at Noor Center, please visit: https://ai.inoor.ir