Envoy AI Gateway: Unified Access to Generative AI Services

What is Envoy AI Gateway?

Envoy AI Gateway is an open-source, CNCF-hosted project that provides unified access to generative AI services. It is built directly on top of Envoy Gateway, which means you inherit the battle-tested data plane that powers some of the largest production fleets on the internet. Instead of wiring each application to a different LLM provider SDK, you expose a single consistent interface — and the gateway handles the rest.

Why teams choose it

• Provider-agnostic routing: Send traffic to OpenAI, Anthropic, AWS Bedrock, Azure OpenAI, or self-hosted models through the same API surface.
• Token-based rate limiting: Control spend with limits measured in tokens, not just requests — far more accurate for LLM workloads.
• Enterprise policy controls: Centralized governance for authentication, quotas, and routing rules across your whole organization.
• Observability: Get per-model and per-route metrics, so you always know what your AI traffic is doing.
• Zero vendor lock-in: Swap providers by changing a config line, not your codebase.

Control plane and data plane

The project follows the classic Envoy split: a control plane manages configuration and policies, while the data plane (Envoy itself) handles every request at high performance. This separation lets platform teams manage AI access centrally while keeping latency at the edge.

Use cases

• Platform teams exposing LLM access to many internal applications.
• Fintech, healthcare, and regulated industries that need audit-grade AI traffic controls.
• Organizations running hybrid AI — mixing cloud providers with self-hosted models like Ollama.
• Teams consolidating multiple provider accounts into one billing and policy surface.

Getting started

You can run Envoy AI Gateway in two modes: as a standalone Docker container (great for local testing) or as a Kubernetes Gateway API implementation (great for production). The standalone mode is perfect for trying it out today — a single container gives you the full gateway experience.

Check out the tool page for the official Docker Compose configuration, system requirements, and deployment notes.