Getting Started with Ragas: Evaluate Your RAG Pipeline in 5 Minutes

Getting Started with Ragas: Evaluate Your RAG Pipeline in 5 Minutes

Ragas makes LLM evaluation straightforward. In this tutorial, you'll learn how to install Ragas, create a synthetic test dataset, and run your first evaluation — all in under 5 minutes. By the end, you'll have a clear picture of how your RAG pipeline is performing across key quality metrics.

Step 1: Installation

Ragas installs like any Python package. Create a virtual environment and install:

pip install ragas
# Or if you want the latest development version:
pip install git+https://github.com/explodinggradients/ragas.git

That's it. Ragas works with Python 3.8+ and has minimal dependencies. You'll also need an LLM provider (OpenAI, Anthropic, or any OpenAI-compatible endpoint) to act as the judge for evaluation.

Step 2: Prepare Your Data

Ragas expects evaluation data in a simple format. Here's a minimal example:

from ragas import evaluate
from datasets import Dataset

# Sample data: questions, answers, and retrieved contexts
data = {
    "question": [
        "What is the capital of France?",
        "How does RAG improve LLM accuracy?",
    ],
    "answer": [
        "The capital of France is Paris.",
        "RAG improves accuracy by grounding LLM outputs in retrieved context documents.",
    ],
    "contexts": [
        ["Paris is the capital and most populous city of France."],
        ["Retrieval-Augmented Generation (RAG) combines retrieval from a knowledge base with generative models to produce factually grounded responses."],
    ],
    "ground_truth": [
        "Paris",
        "RAG improves accuracy through contextual grounding.",
    ],
}

dataset = Dataset.from_dict(data)

Step 3: Run Your First Evaluation

With your dataset ready, evaluation is a single function call:

results = evaluate(
    dataset,
    metrics=[
        "faithfulness",
        "answer_relevancy",
        "context_precision",
        "context_recall",
    ],
)

print(results)
# Output:
# {'faithfulness': 0.95, 'answer_relevancy': 0.88, 
#  'context_precision': 0.92, 'context_recall': 0.85}

Step 4: Generate Synthetic Test Data

One of Ragas' most powerful features is automatic test data generation. Given your document corpus, it creates diverse evaluation questions:

from ragas.testset import TestsetGenerator

# Your documents as a list of strings
documents = [
    "Your first document content here...",
    "Your second document content here...",
]

generator = TestsetGenerator()
testset = generator.generate(documents, test_size=10)

print(f"Generated {len(testset)} test samples")
# The testset includes questions, answers, and reference contexts

Understanding the Metrics

Ragas provides a rich vocabulary for understanding your LLM application's performance:

• Faithfulness (0-1): How factually accurate the answer is relative to the provided context. High scores mean the model doesn't hallucinate.
• Answer Relevancy (0-1): How relevant the answer is to the question asked. Low scores suggest the model misunderstood the query.
• Context Precision (0-1): Whether the retrieved context contains only relevant information. Important for efficient retrieval.
• Context Recall (0-1): Whether all relevant information was retrieved. Critical for comprehensive answers.

Conclusion

In just a few minutes, you've set up Ragas, created an evaluation dataset, and measured your RAG pipeline's performance across four key dimensions. Ragas integrates seamlessly into CI/CD workflows, allowing you to track these metrics after every change to your retrieval system, prompts, or model configuration. Start evaluating today and ship better LLM applications with confidence.