Portkey AI Gateway: The Ultimate Self-Hosted API Gateway for LLMs
Route requests to 1,600+ LLMs through a single, self-hosted gateway with automatic retries, caching, guardrails, and observability.
What Is Portkey AI Gateway?
Portkey AI Gateway is an open-source, blazing-fast API gateway built specifically for the AI era. It acts as a single unified endpoint that routes your requests to over 1,600 large language models from providers like OpenAI, Anthropic, Google, Cohere, Mistral, and countless open-source models. Instead of managing dozens of SDKs, API keys, and rate limits separately, you point your application at Portkey and let it handle the complexity.
With 12,000+ GitHub stars and an MIT license, Portkey has become the go-to self-hosted gateway for teams that need production-grade reliability without vendor lock-in. It is written in TypeScript and Node.js, is incredibly lightweight (512 MB minimum RAM), and deploys in seconds with Docker.
Key Features
- Universal Routing: Send any LLM request through a single OpenAI-compatible endpoint. Portkey automatically maps the request format to the target provider.
- Automatic Retries and Fallbacks: If a provider returns a 5xx error or rate-limit response, Portkey can retry with exponential backoff or fall back to a secondary model automatically.
- Semantic Caching: Cache LLM responses based on embedding similarity, not just exact match. Drastically reduces costs for repeated queries.
- Rate Limiting and Guardrails: Protect your backend from abuse with configurable rate limits, and enforce content safety rules before requests reach the LLM.
- Observability: Every request is logged with latency, tokens used, cost, and model info. Integrates with your existing monitoring stack.
Why Self-Host Portkey?
Running Portkey on your own infrastructure gives you complete control over your AI traffic. Your API keys never leave your network, your request logs stay private, and you can configure every aspect of routing, caching, and fallback behavior to match your exact needs.
Use Cases
- Multi-Model Applications: Build apps that use GPT-4 for complex reasoning, Claude for long-context tasks, and open-source models for cost-sensitive workloads -- all through one API.
- High-Availability Deployments: Set up fallback chains so your app never goes down when one provider has an outage.
- Cost Management: Route cheaper models for simple queries, expensive models for complex ones. Track spend per user, per team, per feature.
- AI Guardrails: Inspect and filter inputs and outputs for PII, toxic content, or policy violations before they reach users or models.
Conclusion
Portkey AI Gateway is an essential piece of infrastructure for any team building production AI applications. It eliminates provider lock-in, improves reliability, and gives you deep visibility into your LLM usage -- all in a lightweight, self-hosted package. Whether you are a solo developer experimenting with different models or a large team managing hundreds of API calls per second, Portkey scales with you.