How to Deploy FastChat with Docker: Your Own LLM API in 10 Minutes

## FastChat: From zero to your own chatbot in 10 minutes I'm going to walk you through deploying FastChat with Docker. No prior LLM experience needed — just Docker and a terminal. I'll show you exactly what I did the first time, including the mistakes I made so you don't repeat them. ## Before we start What you'll need: - **Docker** — run `docker --version` to check. If it's not there, [install it](https://docs.docker.com/get-docker/). - **4GB RAM minimum** — FastChat + a 7B model need about 4-6GB - **A GPU helps but isn't required** — without one, inference will be slow (like, 30 seconds per response slow). With a GPU, it's near real-time. - **About 10 minutes** — 5 for setup, 5 for the model to download on first run ## Step 1: Pull the image ```bash docker pull localagi/fastchat:latest ``` This image is about 8GB — it bundles the FastChat framework plus dependencies. On a 100Mbps connection, expect about 10 minutes. Grab a coffee. ## Step 2: Write the Docker Compose file Create a `docker-compose.yml`: ```yaml services: fastchat: image: localagi/fastchat:latest restart: unless-stopped ports: - "8080:8080" # Web UI and API volumes: - ./models:/app/models # Cache downloaded models - ./data:/app/data # Conversation history deploy: resources: reservations: devices: - driver: nvidia count: 1 capabilities: [gpu] ``` I use `restart: unless-stopped` so if my server reboots, FastChat comes back automatically. The volumes are important: `./models` caches model weights so you don't redownload them every restart (trust me, you don't want to wait for that more than once). The GPU section at the bottom (under `deploy`) is optional — remove it if you're running on CPU. But if you have a GPU, this is how Docker makes it available to the container. ## Step 3: Fire it up ```bash docker compose up -d docker compose logs -f ``` The first time, you'll see a lot of Python loading output. Wait until you see something like: ``` INFO: Uvicorn running on http://0.0.0.0:8080 ``` This might take 30-60 seconds depending on your hardware. If you see "No module named 'torch'" or similar errors, the container is still downloading dependencies — give it time. ## Step 4: Verify it's alive ```bash curl http://localhost:8080/v1/models ``` You should see a JSON response listing available models. Or just open `http://localhost:8080` in your browser — you'll see the Gradio chat interface. ## Step 5: Actually use it Send your first chat request via the API: ```bash curl -X POST http://localhost:8080/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "lmsys/vicuna-7b-v1.5", "messages": [ {"role": "user", "content": "Hello! Tell me a short joke."} ] }' ``` If you get a JSON response with the model's reply — congrats, you're running your own LLM API. If you get an error about the model not being found, check the logs: the Docker image might use a different default model name. ## When things go wrong (and they will) ### 🚫 "Port already in use" ```bash # Find what's on port 8080 sudo lsof -i :8080 # Or just use a different port ``` Change `"8080:8080"` to `"8081:8080"` in your compose file and try again. FastChat will still listen on 8080 inside the container, but you'll access it from 8081. ### 🐢 CPU inference is painfully slow Yeah, this one's unavoidable without a GPU. A single 7B model response can take 20-40 seconds on CPU. Two options: 1. Use a smaller model (3B or 1.5B) — faster but dumber 2. Add the GPU config to your compose file (see Step 2) ### 💥 Container starts and immediately stops This usually means the model download failed. Remove the container and try again: ```bash docker compose down docker compose up -d ``` If it keeps happening, download the model manually first, then mount it: ```bash # Download model to local directory python -c "from huggingface_hub import snapshot_download; snapshot_download(repo_id='lmsys/vicuna-7b-v1.5', local_dir='./models/lmsys/vicuna-7b-v1.5')" ``` ## What about performance? On my homelab (RTX 3060 12GB, 6-core CPU): | Metric | Value | |---|---| | Cold start (first launch) | ~2 min (model download + loading) | | Warm start (subsequent) | ~15 seconds | | Time to first token | ~500ms | | Generation speed | ~30 tokens/sec | | RAM usage | ~5.5GB (container) | | Disk usage | ~8GB (image) + ~4GB (model cache) | On CPU only (no GPU), expect generation speeds of about 2-5 tokens/second — usable for testing, frustrating for real use. ## Final thoughts FastChat through Docker is the quickest path to running your own LLM API. It's not the most optimized serving solution (vLLM is faster, llama.cpp is lighter), but it's the most complete one — API + UI + evaluation all in a single container. If you hit a wall, check the logs first. Seriously. 80% of "FastChat isn't working" posts are solved by `docker compose logs -f`. P.S. Once you have it running, try loading a different model. The `localagi/fastchat` image supports most HuggingFace models. Just point it at a different `--model-path` and watch it work.