How to Deploy vLLM Ascend with Docker: Step-by-Step Tutorial

In this tutorial you'll get vLLM Ascend serving an LLM on Huawei Ascend hardware using the official Docker image. The whole flow — from pull to a working OpenAI-compatible endpoint — takes about ten minutes on a machine with an Ascend 910B or 310P NPU.

Prerequisites

• An Ascend 910B / 310P NPU with the CANN toolkit installed and npu-smi info reporting your device
• Docker with the Ascend container runtime (Ascend Docker Runtime)
• 8 GB+ RAM minimum; 32 GB recommended for larger models

Step 1 — Pull the image

docker pull ascendai/vllm-ascend:latest

The image bundles vLLM, the vllm-ascend plugin, and the Ascend runtime — no manual pip installs needed. It has been pulled over 315,000 times, so it is battle-tested in production setups.

Step 2 — Run the server

docker run --rm \
  --device=/dev/davinci0 \
  --device=/dev/davinci_manager \
  --device=/dev/hisi_hdc \
  -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
  -v /usr/local/dcmi:/usr/local/dcmi \
  -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
  -p 8080:8080 \
  ascendai/vllm-ascend:latest \
  --model Qwen/Qwen2.5-7B-Instruct --max-model-len 8192

This exposes the standard vLLM server on port 8080. You can swap in any model from Hugging Face that fits your NPU memory.

Step 3 — Query the endpoint

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "Qwen/Qwen2.5-7B-Instruct", "messages": [{"role": "user", "content": "Hello from Ascend!"}]}'

Because the API is OpenAI-compatible, your existing clients — LangChain, LiteLLM, Open WebUI — work without modification. Just point base_url at http://localhost:8080/v1.

Performance notes