AnythingLLM Installation Guide

Ubuntu 24.04 • Docker • Ollama • Full RAG Setup

What is AnythingLLM?

AnythingLLM is the ultimate AI workspace — a full-stack application that lets you turn any document, website, or codebase into RAG (Retrieval-Augmented Generation) data you can chat with. It works with any LLM, local or cloud-based, and runs entirely on your own infrastructure. Combine documents, PDFs, videos, and audio into a single AI-powered knowledge base with a clean, modern interface.

Key capabilities:

• Multi-LLM support — Ollama, OpenAI, Anthropic, Azure, LM Studio, and 20+ more providers
• Local-first RAG — embedding and inference never leave your machine
• Multi-user workspaces — separate workspaces with isolated document stores
• Custom agents — built-in tools for web search, code execution, and more

Prerequisites

• 2 vCPU / 4 GB RAM minimum (8 GB+ recommended for large documents or multiple users)
• Docker — community edition (25.x+) installed (docker --version to verify)
• Docker Compose (V2, included with Docker Desktop or docker compose plugin)
• Port 3001 — must be free and accessible (or proxied via Nginx)
• 10 GB+ free disk space for Docker images, storage volume, and LLM models
• Ubuntu 24.04 LTS (Noble Numbat) — freshly updated (sudo apt update && sudo apt upgrade -y)

1. Install Docker

Install the official Docker repository packages on Ubuntu 24.04:

# Add Docker's official GPG key and repository
sudo apt-get update
sudo apt-get install -y ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

# Add your user to the docker group (log out and back in after)
sudo usermod -aG docker $USER

# Verify installation
docker --version
docker compose version

2. Create the Project Directory

mkdir -p ~/anythingllm/storage
cd ~/anythingllm

3. Docker Compose Configuration

Create docker-compose.yml in ~/anythingllm/. This is the recommended production-grade setup with Ollama as the LLM provider and local embeddings:

version: "3.8"
services:
  anythingllm:
    image: ghcr.io/mintplexlabs/anything-llm:latest
    container_name: anythingllm
    ports:
      - "3001:3001"
    volumes:
      - ./storage:/app/storage
    environment:
      - STORAGE_DIR=/app/storage
      - JWT_SECRET="change-me-to-a-random-string"
      - LLM_PROVIDER=ollama
      - OLLAMA_BASE_URL=http://host.docker.internal:11434
      - OLLAMA_MODEL_PREF=llama3.2
      - EMBEDDING_PROVIDER=ollama
      - OPEN_MODEL_PREF=nomic-embed-text
    extra_hosts:
      - "host.docker.internal:host-gateway"
    restart: unless-stopped

Minimal Configuration (without Ollama env vars)

If you don't need Ollama presets, the bare minimum is:

version: "3.8"
services:
  anythingllm:
    image: ghcr.io/mintplexlabs/anything-llm:latest
    container_name: anythingllm
    ports:
      - "3001:3001"
    volumes:
      - ./storage:/app/storage
    environment:
      - STORAGE_DIR=/app/storage
      - JWT_SECRET="a-strong-random-secret"
    restart: unless-stopped

You can then configure LLM and embedding providers through the web UI at first login.

4. Start the Container

docker compose up -d
docker compose logs -f    # Watch startup (Ctrl+C to detach)

Once you see Server running on port 3001, open http://your-server-ip:3001 in your browser.

5. Nginx Reverse Proxy (Optional but Recommended)

Expose AnythingLLM on a subdomain with SSL:

# Install Nginx
sudo apt-get install -y nginx certbot python3-certbot-nginx

# Create site config
sudo nano /etc/nginx/sites-available/anythingllm

Paste the following (replace anything.example.com):

server {
    listen 80;
    server_name anything.example.com;

    client_max_body_size 100M;

    location / {
        proxy_pass http://127.0.0.1:3001;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_buffering off;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }
}

# Enable site and get SSL
sudo ln -s /etc/nginx/sites-available/anythingllm /etc/nginx/sites-enabled/
sudo nginx -t
sudo systemctl reload nginx

# Obtain Let's Encrypt certificate
sudo certbot --nginx -d anything.example.com

6. Environment Variables Reference

7. RAG Configuration with Local Embeddings

AnythingLLM supports fully local RAG — documents are chunked, embedded, and indexed on your machine without any external API calls.

Connecting to Ollama

Ensure Ollama is running on the host:

# Install Ollama (if not already installed)
curl -fsSL https://ollama.com/install.sh | sh

# Pull a chat model and an embedding model
ollama pull llama3.2
ollama pull nomic-embed-text

# Ollama serves on 0.0.0.0:11434 by default — verify
ollama list

Configuring RAG in the UI

1. Open AnythingLLM at http://localhost:3001 (or your domain).
2. Create an admin account on first launch.
3. Go to Settings → LLM Preference and select Ollama.
4. Set the Ollama Base URL to http://host.docker.internal:11434 (or the IP of your Ollama host).
5. Choose the chat model (e.g. llama3.2) and save.
6. Go to Settings → Embedding Preference and select Ollama.
7. Set the embedding model to nomic-embed-text (or llama3.2).
8. Save — the system will re-embed any existing documents with the new provider.

Document Vector Settings

In Settings → Vector Database you can adjust:

• Chunk size — 1000 tokens default; reduce to 500 for precise answers, increase to 2000 for broader context
• Chunk overlap — 200 tokens default; helps maintain context across chunk boundaries
• Document similarity threshold — minimum relevance score (0.0–1.0) for a chunk to be included in context

8. Backup

The entire AnythingLLM state — documents, vector index, user accounts, settings — lives in the ./storage directory. Back it up regularly:

# Stop the container before backup for consistency
docker compose down

# Create a timestamped backup
tar -czf anythingllm-backup-$(date +%Y%m%d-%H%M%S).tar.gz storage/

# Restart the container
docker compose up -d

To restore, stop the container, extract the tarball over ./storage/, and restart:

tar -xzf anythingllm-backup-20250115-120000.tar.gz
docker compose up -d

9. Updating AnythingLLM

cd ~/anythingllm
docker compose pull                 # Pull the latest image
docker compose up -d                 # Recreate container if image changed
docker image prune                   # Remove old dangling images

Check the GitHub Releases page for changelogs and breaking changes before upgrading major versions.

10. Troubleshooting

JWT Errors After Restart

Symptom: After restarting the container, existing sessions show "Invalid or expired token" or users are logged out.

Cause: The JWT_SECRET changed between runs. AnythingLLM uses it to sign session tokens; changing it invalidates all existing sessions.

Fix: Set a fixed JWT_SECRET in docker-compose.yml and never change it. Users simply log in again. If you intentionally rotated the secret, that's expected behavior.

Ollama Connection Refused

Symptom: "Failed to connect to Ollama" or "Connection refused" when selecting the Ollama LLM provider.

Checklist:

1. Verify Ollama is running: curl http://localhost:11434/api/tags on the host.
2. Ensure the bridge network allows host access: host.docker.internal works on Docker Desktop and with extra_hosts as shown above.
3. If using Docker on a remote host, replace host.docker.internal with the actual IP (e.g. http://192.168.1.50:11434).
4. Check Ollama binds to 0.0.0.0, not 127.0.0.1: ss -tlnp | grep 11434 should show 0.0.0.0:11434.

Embedding Dimension Mismatch

Symptom: "Vector dimension mismatch" or "Embedding size mismatch" errors when querying documents.

Cause: You switched embedding models (or providers) after documents were already indexed. A different model produces vectors of a different dimension, corrupting the vector database.

Fix: Go to Settings → Vector Database and click Reset Vector Database (this removes all existing embeddings but keeps your documents). Then re-upload or re-process documents so they are embedded with the new model.

Large Document Processing

Symptom: Large PDFs (100+ pages) time out or fail to process.

Solutions:

• Increase the client_max_body_size in Nginx to 500M+ (see Section 5).
• Add DISABLE_LOGS=true to the container env vars to reduce I/O overhead during processing.
• Split very large documents into smaller files (50–100 pages each) before uploading.
• Ensure the server has adequate RAM — embedding is memory-intensive. 8 GB is recommended for documents over 1000 pages.
• Reduce chunk size in Vector Database settings to 500 tokens so each chunk processes faster.

General Debugging

Check container logs for detailed error messages:

docker compose logs -f --tail=100

To rebuild without cache if something is stuck:

docker compose down
docker compose up -d --force-recreate

AnythingLLM — GitHub Repository • Official Website • Run This AI