Deploying OpenVINO with Docker for Cross-Platform AI Inference

Why Docker for OpenVINO?

Intel's OpenVINO toolkit has numerous dependencies — OpenCL runtime, TBB, model optimizers, and framework-specific converters. The official Docker images bundle everything you need in a single container, eliminating setup headaches and ensuring reproducible inference across development, staging, and production environments.

Available Docker Images

Intel provides several official OpenVINO Docker images on Docker Hub under the openvino namespace:

openvino/ubuntu22_dev:latest — Development image with full toolkit, Python bindings, and samples
openvino/ubuntu22_runtime:latest — Lightweight runtime-only image for production deployments
openvino/ubuntu22_dev:2024.x — Version-pinned images for reproducible builds

Quick Start with Docker Compose

services:
  openvino:
    image: openvino/ubuntu22_dev:latest
    restart: unless-stopped
    ports:
      - 8080:8080
    volumes:
      - ./data/openvino:/data
      - ./models:/models
    environment:
      - INFERENCE_BACKEND=CPU

docker compose up -d

Running Inference in the Container

Once the container is running, you can execute OpenVINO commands inside it:

docker exec -it openvino-openvino-1 bash
benchmark_app -m /models/model.xml -d CPU

Using GPU Acceleration

For Intel integrated or discrete GPUs, pass through the device:

services:
  openvino:
    image: openvino/ubuntu22_dev:latest
    devices:
      - /dev/dri:/dev/dri
    environment:
      - INFERENCE_BACKEND=GPU

Converting Models

To convert a PyTorch or TensorFlow model to OpenVINO IR format:

docker exec openvino-openvino-1 bash -c "mo --input_model /models/model.pth --output_dir /models/ir"

Production Tips

Use the runtime image (not dev) for smaller deployments — saves ~2GB
Pre-convert models to IR format during your CI pipeline, not at deployment time
Enable INT8 quantization for 2-4x throughput improvement on compatible hardware
Use the HETERO plugin to split workloads across CPU + GPU automatically