Retriever Evaluation

Retriever Evaluation Service

Retriever Evaluation Service

Using this tool, the retrieval results in the RAG (Retrieval-Augmented Generation) process are measured.

How It Works

In this tool, for each query, the top 10 results are stored. Then, the retrieval results from the RAG system are evaluated using one of two methods:

1. Human Expert Annotation: A human expert reviews the results and determines their relevance to the query
2. LLM-based Evaluation: A Large Language Model acts as the evaluator and assesses the results

Evaluation Output

This tool determines how relevant and close the retrieved contents are to the target query.

Applications

• RAG System Quality Assessment: Evaluating the performance of the retrieval component in Retrieval-Augmented Generation systems
• Retrieval Improvement: Identifying weaknesses in the retrieval process for optimization
• Comparing Different Methods: Evaluating and comparing different retrieval algorithms and methods
• Result Validation: Ensuring the relevance of retrieved content to the user's query

Benefits

• Automated evaluation using LLMs or precise evaluation with human experts
• Top-10 result storage for each query
• Determining the relevance and closeness of content to the query
• Continuous improvement of retrieval quality in RAG systems