LangWatch Guide: LLM Evaluation and Agent Testing Platform

🤔 The Problem: Testing LLMs Without Going Insane

If you've ever shipped an LLM-powered feature to production, you know the feeling. It works great on your three test prompts. Then a real user shows up and suddenly your agent is hallucinating SQL queries, refusing to translate "hello," or burning $50 in API calls on a single conversation.

I've been there. I'd run a few manual tests, think "looks good," deploy, and then spend the next day debugging why my agent broke on an edge case I never imagined. The worst part? No visibility into what was actually happening inside the agent's decision loop.

Enter LangWatch. An open-source platform that gives you full visibility into LLM agent behavior — trace every decision, run simulations against realistic scenarios, evaluate systematically, and optimize before your users find the bugs for you. It's Apache 2.0 licensed, self-hostable, and integrates with anything via OpenTelemetry.

🧠 What Makes LangWatch Different?

🔬 Agent Simulations That Actually Work

LangWatch lets you run end-to-end simulations against your full agent stack — tools, state, user simulator, judge — before going to production. You define scenarios (e.g., "user wants a refund for an item that shipped 3 days ago"), and LangWatch runs the whole thing, tracking every decision your agent makes. When something breaks, you see exactly where and why, down to the individual tool call.

I set up a simulation with 12 scenarios in about 20 minutes. Found two bugs in my agent's routing logic that would've definitely hit production. The trace view showed me the exact chain of thought that led to the wrong decision. That alone saved me a weekend of debugging.

🔄 Trace → Dataset → Evaluate → Optimize

This is where LangWatch shines. Every production trace becomes a dataset entry. Every dataset entry feeds evaluations. Every evaluation result tells you what to optimize. It's one continuous loop, not five disconnected tools.

🛡️ AI Gateway: Governance Without the Bloat

LangWatch ships with an AI Gateway — a Go binary that acts as an OpenAI/Anthropic-compatible proxy. Virtual keys, hierarchical budgets, inline guardrails, automatic fallback across providers, and Anthropic cache_control passthrough. The hot-path overhead is ~700 nanoseconds. I tested this: dropped it in front of our existing OpenAI calls by changing the base URL, and within 5 minutes I had usage limits per team, cost alerts, and automatic fallback to a cheaper model when the primary hit its budget. No code changes.

💡 What I Wish I Knew Before Starting

The Docker setup needs patience the first time. LangWatch depends on PostgreSQL, Redis, and ClickHouse — so the docker-compose pulls about 2GB of images initially. But after that first pull, restart is instant. Give it 5-10 minutes on a fresh install.

The npx CLI is actually faster for local dev. Run npx @langwatch/server and it handles everything — installs uv, Postgres, Redis, ClickHouse, the gateway binary — all into ~/.langwatch/. Opens at localhost:5560. Clean reset is rm -rf ~/.langwatch. I use this for testing, Docker for production.

🎯 Who Is LangWatch For?

Yes, if: You're building LLM-powered agents or features and need structured testing before production. You already have or plan to use OpenTelemetry. Your team needs shared visibility into agent behavior and evaluation results.

Maybe not, if: You're just wrapping a single LLM call with no tool use or multi-step reasoning — LangWatch's simulation and tracing features are overkill for a simple chatbot. Also, the evaluation workflow assumes you have some labeled data or scenarios; if you're starting from zero, expect a setup day.