Getting Started with Haystack on Docker: Build Your First RAG Pipeline

Getting Started with Haystack

Haystack is the leading open-source framework for building production-ready RAG (Retrieval-Augmented Generation) pipelines. In this tutorial, we will walk through setting up Haystack with Docker and building your first question-answering system over your own documents.

Prerequisites

Before starting, make sure you have Docker installed on your system. You will also need an API key for an LLM provider (such as OpenAI or Cohere) or access to a local model running via Ollama or vLLM. Haystack works with both cloud and local models.

Step 1: Pull and Run Haystack

The fastest way to get started is using the official Docker image:

docker pull deepset/haystack:latest

docker run -d --name haystack \

  -p 8080:8080 \

  -v $(pwd)/data:/data \

  deepset/haystack:latest

This starts Haystack on port 8080 with a persistent data directory for your documents and indexes.

Step 2: Install the Haystack Python Client

Inside your application container or environment, install the Haystack SDK:

pip install haystack-ai

Step 3: Build Your First RAG Pipeline

Create a Python script that indexes documents and answers questions:

from haystack import Pipeline

from haystack.components.retrievers import InMemoryEmbeddingRetriever

from haystack.components.generators import OpenAIGenerator

from haystack.components.embedders import SentenceTransformersTextEmbedder

from haystack.document_stores.in_memory import InMemoryDocumentStore

document_store = InMemoryDocumentStore()

pipeline = Pipeline()

pipeline.add_component('embedder', SentenceTransformersTextEmbedder())

pipeline.add_component('retriever', InMemoryEmbeddingRetriever(document_store))

pipeline.add_component('llm', OpenAIGenerator())

pipeline.connect('embedder.embedding', 'retriever.query_embedding')

pipeline.connect('retriever', 'llm')

Step 4: Index Documents and Query

Add your documents to the store and run queries through the pipeline. Haystack handles chunking, embedding, retrieval, and generation automatically within your defined pipeline graph.

Going Further

Once your basic pipeline works, explore Haystack's more advanced features: hybrid retrieval (combining dense and sparse search), re-ranking with cross-encoders for higher accuracy, and custom components for domain-specific processing. Haystack also supports streaming responses, async execution, and OpenTelemetry tracing for production monitoring.

Why Self-Host Haystack?

Running Haystack on your own infrastructure gives you full control over data privacy, model choice, and pipeline customization. You can connect it to any vector database (Weaviate, Milvus, Qdrant, Chroma, Pinecone), use any LLM provider or local model, and scale components independently. No vendor lock-in, no data leaving your servers.

Conclusion

Haystack makes it straightforward to build production-grade RAG pipelines that are modular, testable, and self-hosted. With Docker, you can have your first pipeline running in minutes. The framework's component-based architecture means you can start simple and add complexity as your requirements grow.