TensorZero: The Open-Source LLMOps Platform That Combines Gateway, Observability & Optimization
TensorZero unifies an LLM gateway, observability, evaluation, and experimentation in one open-source platform. Learn how it optimizes prompts and models with real production traffic.
🚀 Want to deploy TensorZero yourself?
Docker configs, system requirements, and installation guides — all on one page.
View TensorZero Tool Page →What is TensorZero?
TensorZero is an open-source LLMOps platform that combines four critical pieces of the AI stack into one system: an LLM gateway, observability, evaluation, and experimentation. Instead of stitching together a proxy, a tracing tool, and a prompt-management spreadsheet, you get a single endpoint that records everything and lets you optimize against real production traffic.
With 11.7K GitHub stars and an Apache-2.0 license, TensorZero has quickly become a favorite for teams that want to move from "prompting by intuition" to systematic, data-driven LLM improvement. It is written in Rust for low latency and high throughput, and it works with OpenAI, Anthropic, Google, Mistral, vLLM, and any OpenAI-compatible endpoint.
Why a Gateway + Observability + Optimization in One?
Most LLM teams suffer from the same problem: the gateway team, the eval team, and the prompt team all work in different tools, so nobody can answer "which prompt actually performs best in production?" TensorZero solves this by treating every request as data. Each inference is recorded with its input, output, and metadata — then used to run experiments, A/B tests, and fine-tunes automatically.
- 🔄 Unified gateway for all major LLM providers
- 📊 Full request/response tracing and observability
- 🧪 A/B testing and gradual rollout of prompts & models
- 🎯 Fine-tuning (SFT + RL) driven by real traffic
- 📈 Variant scoring, caching, and cost controls
Use Cases
TensorZero shines for production AI features: customer support assistants, code generation, structured extraction, and routing agents. Because optimization is built into the gateway, teams can ship a new prompt variant to 5% of traffic, measure quality automatically, and roll it out — without a single manual eval spreadsheet.
Deployment Options
You can run TensorZero with the official tensorzero/gateway Docker image (234K+ pulls) plus the tensorzero/ui dashboard. It stores configuration in a Git repo and event data in ClickHouse, making it fully self-hosted and portable. For a quick start, a docker-compose setup with the gateway on port 8080 is all you need.
🚀 Ready to run TensorZero?
System requirements and ready-made Docker configs are waiting.
View TensorZero Tool Page →