Running AI Models Using Docker

# Running AI Models Using Docker

Introduction

Docker provides a portable, reproducible way to run AI models across different environments. This tutorial covers running AI models in Docker using Ollama, vLLM, TensorFlow Serving, and other tools, with production deployment patterns.

Learning Objectives

By the end of this tutorial, you will be able to:

• Run AI models in Docker containers
• Deploy Ollama and vLLM with Docker
• Use TensorFlow Serving for model deployment
• Configure GPU access in Docker
• Manage model volumes and persistence
• Deploy AI models to production with Docker Compose

Technical Background

Docker AI Model Deployment Architecture

flowchart TB
    Client[Client App] --> API[API Gateway]
    API --> LB[Load Balancer]
    LB --> Ollama[Ollama Container]
    LB --> vLLM[vLLM Container]
    LB --> TFS[TensorFlow Serving Container]
    
    Ollama --> Models[(Model Volume)]
    vLLM --> Models
    TFS --> Models
    
    subgraph Docker Host
    Ollama
    vLLM
    TFS
    end
    
    subgraph GPU
    GPU[NVIDIA GPU]
    end
    
    Ollama --> GPU
    vLLM --> GPU

Container Options for AI

| Tool | Best For | GPU Support | Complexity | |------|----------|-------------|------------| | Ollama | Local LLM inference | Yes | Low | | vLLM | High-throughput LLM serving | Yes | Medium | | TensorFlow Serving | TF model production | Yes | High | | PyTorch Serve | PyTorch model production | Yes | Medium | | Triton Inference Server | Multi-framework | Yes | High |

Prerequisites

• Docker and Docker Compose installed
• NVIDIA GPU (for GPU acceleration)
• NVIDIA Container Toolkit
• 50+ GB storage for models

Hardware and Software Requirements

| Component | Requirement | |-----------|-------------| | Docker | v24+ | | Docker Compose | v2+ | | NVIDIA Container Toolkit | Installed | | NVIDIA Driver | 525+ | | Storage | 100+ GB |

Step-by-Step Implementation

Phase 1: Install NVIDIA Container Toolkit

# Add NVIDIA package repository
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg

curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt update
sudo apt install -y nvidia-container-toolkit
sudo systemctl restart docker

# Verify GPU access
docker run --rm --gpus all nvidia/cuda:12.0-base-ubuntu22.04 nvidia-smi

Phase 2: Run Ollama in Docker

# Run Ollama with GPU support
docker run -d \
  --name ollama \
  --gpus all \
  -p 11434:11434 \
  -v ollama-models:/root/.ollama \
  --restart unless-stopped \
  ollama/ollama

# Pull and run a model
docker exec ollama ollama pull llama3.2
docker exec ollama ollama run llama3.2 "Hello, how are you?"

# Test API
curl http://localhost:11434/api/generate -d '{
  "model": "llama3.2",
  "prompt": "Explain Docker containers",
  "stream": false
}'

Phase 3: Run vLLM in Docker

# Run vLLM with OpenAI-compatible API
docker run -d \
  --name vllm \
  --gpus all \
  -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --restart unless-stopped \
  vllm/vllm-openai:latest \
  --model meta-llama/Meta-Llama-3.1-8B-Instruct \
  --tensor-parallel-size 1 \
  --max-model-len 4096

# Test API
curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
  "messages": [{"role": "user", "content": "Hello!"}],
  "temperature": 0.7
}'

Phase 4: Run TensorFlow Serving in Docker

# Export a SavedModel (example)
# python save_model.py

# Run TensorFlow Serving
docker run -d \
  --name tensorflow-serving \
  -p 8501:8501 \
  -v /path/to/saved_model:/models/my_model \
  -e MODEL_NAME=my_model \
  --restart unless-stopped \
  tensorflow/serving

# Test API
curl -d '{"instances": [[1.0, 2.0, 3.0]]}' -X POST http://localhost:8501/v1/models/my_model:predict

Phase 5: Docker Compose for Multi-Model Deployment

# docker-compose.yml
version: '3.8'
services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    ports:
      - "11434:11434"
    volumes:
      - ollama-data:/root/.ollama
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    restart: unless-stopped

  vllm:
    image: vllm/vllm-openai:latest
    container_name: vllm
    ports:
      - "8000:8000"
    volumes:
      - hf-cache:/root/.cache/huggingface
    environment:
      - MODEL_NAME=meta-llama/Meta-Llama-3.1-8B-Instruct
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    restart: unless-stopped
    command: --model meta-llama/Meta-Llama-3.1-8B-Instruct --tensor-parallel-size 1

  nginx:
    image: nginx:alpine
    container_name: nginx
    ports:
      - "80:80"
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf:ro
    depends_on:
      - ollama
      - vllm
    restart: unless-stopped

networks:
  ai-network:
    driver: bridge

volumes:
  ollama-data:
  hf-cache:

Phase 6: NGINX Reverse Proxy Configuration

# nginx.conf
events {
    worker_connections 1024;
}

http {
    upstream ollama_backend {
        server ollama:11434;
    }

    upstream vllm_backend {
        server vllm:8000;
    }

    server {
        listen 80;

        location /ollama/ {
            proxy_pass http://ollama_backend/;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";
            proxy_read_timeout 300s;
            proxy_send_timeout 300s;
        }

        location /v1/ {
            proxy_pass http://vllm_backend/v1/;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";
            proxy_read_timeout 300s;
            proxy_send_timeout 300s;
        }
    }
}

Phase 7: Model Volume Management

# Create named volumes
docker volume create ollama-models
docker volume create hf-models

# Inspect volume location
docker volume inspect ollama-models

# Backup model volume
docker run --rm \
  -v ollama-models:/source \
  -v $(pwd):/backup \
  alpine \
  tar czf /backup/ollama-models-backup.tar.gz -C /source .

# Restore model volume
docker run --rm \
  -v ollama-models:/target \
  -v $(pwd):/backup \
  alpine \
  tar xzf /backup/ollama-models-backup.tar.gz -C /target

Phase 8: Resource Limits and Optimization

# Resource limits in Docker Compose
deploy:
  resources:
    limits:
      cpus: '2.0'
      memory: 16G
    reservations:
      devices:
        - driver: nvidia
          count: 1
          capabilities: [gpu]
      reservations:
        cpus: '1.0'
        memory: 8G

Expected Output

• AI models running in Docker containers
• Ollama and vLLM APIs accessible
• TensorFlow Serving deployed
• Multi-model deployment with Docker Compose
• NGINX reverse proxy routing requests
• Model volumes persisted and backed up

Verification Procedures

| Check | Command | Expected Result | |-------|---------|-----------------| | Ollama running | `docker ps | grep ollama` | Container running | | Ollama API | `curl localhost:11434/api/tags` | Models listed | | vLLM API | `curl localhost:8000/v1/models` | Models listed | | GPU in container | `docker run --rm --gpus all nvidia/cuda nvidia-smi` | GPU info shown |

Troubleshooting Scenarios

| Issue | Diagnosis | Resolution | |-------|-----------|------------| | Container can't access GPU | NVIDIA Container Toolkit missing | Install and configure toolkit | | Port already in use | Another service on port | Change port mapping | | Model download fails | Network or auth issue | Check network; authenticate to registry | | Out of GPU memory | Model too large | Use smaller model or quantization | | Docker volume permission denied | Volume ownership mismatch | Fix volume permissions |

Best Practices

• **Use named volumes:** Persist model data across container restarts
• **Pin image versions:** Use specific tags for reproducibility
• **Set resource limits:** Prevent containers from consuming all resources
• **Use health checks:** Detect unhealthy containers
• **Backup model volumes:** Regularly back up model data
• **Use reverse proxy:** Add TLS and authentication for external access

Security Considerations

• **Container isolation:** Use Docker namespaces and cgroups
• **Image scanning:** Scan images for vulnerabilities
• **Secret management:** Don't store API keys in images
• **Network segmentation:** Isolate AI containers from other services
• **Non-root user:** Run containers as non-root where possible

Conclusion

Docker provides a powerful platform for running AI models in production. By combining Docker with Ollama, vLLM, and TensorFlow Serving, teams can deploy portable, scalable AI inference services with GPU acceleration. Docker Compose simplifies multi-service deployment, while named volumes ensure model data persistence.

References

• Docker: https://docs.docker.com/
• Ollama Docker: https://github.com/ollama/ollama/blob/main/docs/docker.md
• vLLM: https://docs.vllm.ai/
• TensorFlow Serving: https://www.tensorflow.org/tfx/guide/serving
• NVIDIA Container Toolkit: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/