How to Deploy Text Generation Inference (TGI) with Docker

What is Text Generation Inference?

Text Generation Inference (TGI) is Hugging Face's production-grade LLM serving solution. Built with Rust for speed and Python for flexibility, TGI provides blazing-fast inference for large language models with an OpenAI-compatible API. It supports tensor parallelism, continuous batching, streaming tokens, and multiple quantization methods (bitsandbytes, GPTQ, AWQ).

Prerequisites

A Linux server with NVIDIA GPU (optional for CPU-only)
Docker and Docker Compose installed
At least 8GB RAM (16GB+ recommended)
NVIDIA Container Toolkit if using GPUs

Step 1: Pull the Docker Image

docker pull ghcr.io/huggingface/text-generation-inference:latest

Step 2: Run TGI with a Model

Launch the server with a model from Hugging Face Hub:

docker run --gpus all -p 8080:80 \
  -v $PWD/data:/data \
  ghcr.io/huggingface/text-generation-inference:latest \
  --model-id mistralai/Mistral-7B-Instruct-v0.2

Step 3: Use the OpenAI-Compatible API

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"tgi","messages":[{"role":"user","content":"Hello!"}]}'

Docker Compose for Production

version: '3.8'
services:
  tgi:
    image: ghcr.io/huggingface/text-generation-inference:latest
    restart: unless-stopped
    ports:
      - 8080:80
    volumes:
      - ./data:/data
    environment:
      - MODEL_ID=mistralai/Mistral-7B-Instruct-v0.2
      - NUM_SHARD=1
      - MAX_INPUT_TOKENS=2048
      - MAX_BATCH_PREFILL_TOKENS=4096

Key Features

Continuous Batching: Dynamically batches requests for maximum GPU utilization
Tensor Parallelism: Distributes models across multiple GPUs with zero code changes
Quantization: Built-in support for bitsandbytes 4/8-bit, GPTQ, AWQ, and EETQ
Streaming: Server-Sent Events (SSE) for real-time token-by-token output
Safety: Content moderation and rate limiting built in

Conclusion

TGI is the gold standard for self-hosted LLM serving within the Hugging Face ecosystem. With over 10,800 GitHub stars and production use by enterprises worldwide, it delivers exceptional performance through its Rust-powered core. Deploy it today for complete control over your AI infrastructure.