LangWatch Tutorial: Set Up LLM Agent Testing in 15 Minutes

🛠️ Tutorial: Set Up LangWatch and Run Your First Agent Simulation in 15 Minutes

I'll walk you through getting LangWatch running with Docker, creating your first project, and running an agent simulation that catches bugs before they hit production. No prior setup needed — just Docker and a terminal.

Time estimate: ~15 minutes (first-time Docker pull adds 5-10 minutes). Take a coffee break during the image downloads.

Step 1: Start LangWatch with Docker

Clone the repo and spin it up:

git clone https://github.com/langwatch/langwatch.git
cd langwatch
cp langwatch/.env.example langwatch/.env
docker compose up -d --wait --build

Once the containers are up, open http://localhost:5560 in your browser. If you see a login page — congrats, it worked. If not, run docker compose logs langwatch to check for errors.

Step 2: Create Your First Project

Click "Create Project" → name it "Test Agent" → copy the API key that appears. You'll need this key to send traces from your agent.

In the dashboard, notice a few things:

• Traces tab: Empty now, but this is where every agent decision shows up
• Datasets tab: For storing and labeling evaluation data
• Scenarios tab: This is where you define agent simulations
• Evaluations tab: Run evals against your datasets or scenarios

Don't worry if it looks sparse — it will fill up as soon as you send your first trace or run a simulation.

Step 3: Run Your First Agent Simulation

Go to Scenarios → "New Scenario". Define a simple test:

Click "Run Simulation". LangWatch will simulate a user interacting with your agent through this scenario. Every tool call, every LLM response, every decision — it's all captured in the trace view.

What I discovered on my first run: My agent tried to refund an order that had already shipped. The trace showed it called the wrong tool — it used cancel_order instead of check_refund_policy. LangWatch flagged the mismatch between the actual and expected behavior. Caught it before any real user hit it.

Step 4: Send a Real Trace (Optional)

To monitor a production agent, install the LangWatch SDK:

pip install langwatch
# or: npm install langwatch

Then instrument your agent:

import langwatch

langwatch.init(api_key="lw_...")  # your project API key

with langwatch.trace("user-query") as span:
    result = my_agent.process(query)
    span.log(result)

Traces appear in real-time in the LangWatch dashboard. Each trace shows the full decision chain: which LLM calls were made, which tools were invoked, latency per step, token usage, and cost.

📊 Performance Expectations

🎯 Common Pitfalls

✅ Verification Checklist

After following these steps, you should see:

1. http://localhost:5560 → LangWatch dashboard ✅
2. Created a project with an API key ✅
3. Ran at least one scenario simulation ✅
4. Viewed the trace for that simulation ✅

If all four are checked — you're up and running. If something failed, check docker compose logs for errors or hop into the LangWatch Discord.