To ensure Ragie delivers optimal performance in real-world applications, we regularly benchmark it against complex datasets across various domains to evaluate how it would perform in a production environment. In this article, we’ll walk you through how we conducted a focused evaluation of our RAG pipeline using LegalBench-RAG, a benchmark designed to evaluate the retrieval component of RAG systems in the legal domain. 

The LegalBench-RAG focuses on precision (retrieving the most relevant data) and recall (retrieving all relevant data) to measure how effectively RAG systems handle the complexities of legal texts. 

It includes four categories of legal documents:

1. NDA related documents (ContractNLI)
2. Private Contracts (Contract Understanding Atticus Dataset - CUAD)
3. M&A documents of public companies (M&A Understanding Dataset - MAUD)
4. Privacy policies of consumer apps (Privacy QA)

This benchmarking exercise provides valuable insights into Ragie’s effectiveness in handling legal queries, where accuracy, contextual understanding, and relevance are essential. Through this evaluation, we also demonstrate how Ragie offers both precision and recall that surpass industry standards.

Our Methodology for Evaluating Ragie Against LegalBench-RAG

While we evaluated using the same metrics, we needed to measure them slightly differently due to Ragie's chunking approach, which more closely matches production RAG systems. This differing approach to chunking also highlights the importance of optimal chunking strategies in real-world scenarios.

The Results: Ragie’s Performance vs. LegalBench Benchmarks

We evaluated Ragie's recall and precision across three configurations with k values of 1 to 64 using the recursive text splitter method.

Benchmarking with Hybrid retrieval:

With Hybrid retrieval (semantic + keyword) enabled, which is the default behavior in Ragie—the results in the table below show Ragie’s precision across different k values:

‍

Ragie’s recall across different k values with Hybrid retrieval enabled:

Benchmarking with Hybrid retrieval and Re-ranking:

With Hybrid retrieval and re-ranking enabled, the results in the table below show Ragie’s precision across different k values:

‍

With Hybrid retrieval and re-ranking enabled, the results in the table below show Ragie’s recall across different k values:

Benchmarking with Hierarchical retrieval:

‍With Hierarchical retrieval enabled, the results in the table below show Ragie’s precision across different k values (max_chunks_per_doc = top_k):

‍

Ragie’s recall across different k values (max_chunks_per_doc = top k) with Hierarchical retrieval:

Key Insights from Ragie's Benchmark Analysis

1. Production-level Precision and Recall:
   Ragie significantly outperforms the benchmark in terms of both precision and recall. This means developers can rely on Ragie to retrieve highly accurate results consistently, ensuring high-quality performance in production. 

2. High Recall Accuracy:
   With recall accuracy reaching 99.4%, Ragie excels in retrieving the relevant information, leaving minimal room for missed results. This ensures a highly reliable system for handling complex data queries.
3. Efficiency at Lower k Values:
   Even with low k values, Ragie’s precision and recall are significantly better than the benchmarks. This capability allows developers to use smaller context windows, making production systems faster and more cost-effective without sacrificing performance.

4. Precision Improves with Reranking:
   When the reranking feature is enabled, Ragie achieves a 124% improvement in precision, although it introduces a slight trade-off with recall. This enhancement ensures optimal performance in scenarios where precision is more important than recall.

5. Hierarchical Retrieval Boosts Accuracy:
   Enabling hierarchical settings improves precision further by 26%. 
   ‍
6. Ragie is Highly Customizable:
   Our default setting is the best balance between precision and recall for most apps. However, developers can choose from hybrid, rerank, and hierarchical to pick the best for their needs.

Conclusion

Ragie’s performance against the LegalBench-RAG underscores its readiness for production use in legal and other domain-specific applications. We’re not entirely surprised by the evaluation results because we’ve seen Ragie deliver impressive results in real-world legal use cases. For example, one of our customers (Ellis, an immigration law firm) increased their legal drafting speed by 10x using Ragie's RAG system.