TabbyAPI: The Official ExLlamaV2 Server for Lightweight LLM Inference

What Is TabbyAPI?

TabbyAPI is the official API backend server for ExLlamaV2 and ExLlamaV3, two of the most popular inference engines for local LLMs. Built with FastAPI, it provides a lightweight, OpenAI-compatible API for generating text with large language models. It is designed for hobbyist and small-scale use, prioritizing ease of use and low resource overhead over massive concurrent throughput.

With 1,257 GitHub stars, TabbyAPI is the go-to choice for users who want an ExLlama-powered API server without the complexity of larger frameworks. It supports loading and unloading models at runtime, LoRA adapters, tool calling, and HuggingFace model downloads — all through a clean REST API.

Why TabbyAPI?

ExLlamaV2/V3 are known for being among the fastest inference engines for Llama-family models on consumer GPUs. TabbyAPI wraps this power in a production-quality API server with minimal overhead. Unlike heavier solutions like vLLM, TabbyAPI is designed to run well even on a single GPU, making it ideal for personal use or small communities.

Key Features

• OpenAI-compatible API: Drop-in replacement for any client that supports OpenAI's chat completions format
• ExLlamaV2 and V3 backends: Lightning-fast inference with Flash Attention, quantization support, and fused kernels
• Model hot-swapping: Load and unload models at runtime without restarting the server
• LoRA adapter support: Load multiple LoRAs and switch between them per-request
• Tool calling: Native support for function/tool calling — no modified Jinja templates needed
• HuggingFace integration: Directly download and load models from HuggingFace
• Docker support: Official CUDA-enabled Docker image via GitHub Container Registry
• GGUF via YALS: Sister project YALS extends GGUF model support

Quick Start with Docker

docker pull ghcr.io/theroyallab/tabbyapi:latest
docker run --gpus all --name tabbyapi -p 5000:5000 \
  -v /path/to/models:/app/models \
  ghcr.io/theroyallab/tabbyapi:latest

Replace /path/to/models with your local models directory. The API will be available at http://localhost:5000.

Using the API

Once running, you can use any OpenAI-compatible client:

curl http://localhost:5000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ModelName",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

TabbyAPI vs Other Servers

vs vLLM (on this platform): vLLM uses PagedAttention for high concurrency, while TabbyAPI uses ExLlamaV2 for raw single-user throughput. TabbyAPI is lighter and easier to configure for personal use; vLLM scales better for many concurrent users.

vs Aphrodite Engine (on this platform): Aphrodite is built on vLLM with extended sampler support, while TabbyAPI focuses on ExLlama backends. If you love ExLlama's performance and want a simple API server, TabbyAPI is the natural choice.

vs Ollama (on this platform): Ollama provides a higher-level abstraction with model management, while TabbyAPI gives you direct control over ExLlama's knobs. TabbyAPI is better for power users who want to tune generation parameters.

Who Should Use It?

• Users who already use ExLlamaV2/V3 and want a proper API server
• Hobbyists running LLMs on a single GPU
• Developers building applications that need an OpenAI-compatible local API
• Anyone who wants tool calling capabilities with local models

Conclusion

TabbyAPI is the definitive API server for the ExLlama ecosystem. Lightweight, well-documented, and straightforward to deploy, it gives you access to one of the fastest local LLM inference engines through a clean, standards-compliant API. If you're running ExLlama models locally, TabbyAPI is the server you've been looking for.