Getting Started with llama-cpp-python OpenAI-Compatible Server

Quick Start: Run Your First LLM with llama-cpp-python

llama-cpp-python's OpenAI-compatible server is the fastest way to get a local LLM running. This tutorial will walk you through installing, configuring, and querying your first model. By the end, you'll have a fully functional local AI endpoint that works with any OpenAI-compatible client.

Installation

Install via pip (the server is included):

pip install llama-cpp-python

For GPU acceleration, set build flags before installing:

# CUDA
CMAKE_ARGS="-DGGML_CUDA=on" pip install llama-cpp-python

# Metal (Mac)
CMAKE_ARGS="-DGGML_METAL=on" pip install llama-cpp-python

# Vulkan
CMAKE_ARGS="-DGGML_VULKAN=on" pip install llama-cpp-python

Download a Model

llama-cpp-python uses GGUF format models. Download one from Hugging Face:

# Create a models directory
mkdir -p models

# Download a small model (e.g., Llama 3.2 3B Instruct)
wget -O models/llama-3.2-3b-instruct.Q4_K_M.gguf \
  https://huggingface.co/bartowski/Llama-3.2-3B-Instruct-GGUF/resolve/main/Llama-3.2-3B-Instruct-Q4_K_M.gguf

Start the Server

python3 -m llama_cpp.server --model models/llama-3.2-3b-instruct.Q4_K_M.gguf \
  --n_gpu_layers -1 \
  --host 0.0.0.0 \
  --port 8080

The server starts on port 8080 with an OpenAI-compatible API. The --n_gpu_layers -1 flag offloads all layers to GPU if available.

Query Your Model

Use curl or any OpenAI SDK:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama-3.2-3b-instruct",
    "messages": [{"role": "user", "content": "Hello! What can you do?"}]
  }'

Docker Deployment

Or run via Docker (from our verified image):

docker pull ghcr.io/abetlen/llama-cpp-python:latest

docker run -d --name llama-server \
  -p 8080:8080 \
  -v $(pwd)/models:/models \
  ghcr.io/abetlen/llama-cpp-python:latest \
  --model /models/llama-3.2-3b-instruct.Q4_K_M.gguf \
  --host 0.0.0.0 \
  --port 8080

Using with Open WebUI or Chat Clients

Point any OpenAI-compatible chat UI (Open WebUI, SillyTavern, NextChat) to http://localhost:8080/v1. Configure the API key as anything (it's ignored) and select your model. You now have a fully private, self-hosted AI assistant.

Conclusion

llama-cpp-python transforms any machine into a local AI server. With its OpenAI-compatible API, vision support, and multi-model serving, it's the most practical way to run open models on your own hardware. Start small with a 3B model and scale up as needed.