Building a Private AI Infrastructure

# Building a Private AI Infrastructure

Introduction

A private AI infrastructure enables organizations to run AI workloads on-premises with full control over data, models, and costs. This tutorial covers the complete architecture, hardware, software, deployment, and operations of a private AI infrastructure.

Learning Objectives

By the end of this tutorial, you will be able to:

• Design private AI infrastructure architecture
• Select hardware for AI workloads
• Deploy model serving infrastructure
• Implement security and access controls
• Monitor and maintain AI infrastructure
• Optimize costs and performance

Technical Background

Private AI Architecture

flowchart TB
    Users[Users] --> LB[Load Balancer]
    LB --> API[API Gateway]
    API --> Auth[Authentication]
    Auth --> Model[Model Serving Layer]
    
    Model --> Ollama[Ollama]
    Model --> vLLM[vLLM]
    Model --> Custom[Custom Models]
    
    Model --> GPU[GPU Cluster]
    Model --> Storage[Model Storage]
    
    Storage --> Backup[Backup/DR]
    
    Monitoring[Monitoring] --> Alerts[Alerting]
    Logging[Logging] --> Analysis[Log Analysis]
    
    subgraph Private AI Infrastructure
    API
    Auth
    Model
    GPU
    Storage
    end
    
    subgraph Operations
    Monitoring
    Logging
    end

Infrastructure Components

| Component | Purpose | Options | |-----------|---------|---------| | Compute | Model inference/training | GPU servers | | Storage | Model weights, datasets | NVMe, NAS, S3 | | Network | API access, model loading | 10/25/100 GbE | | Orchestration | Container management | Kubernetes, Docker | | API Gateway | Request routing, auth | NGINX, Kong, Traefik | | Monitoring | Metrics, alerts | Prometheus, Grafana |

Prerequisites

• Server hardware with GPU(s)
• Ubuntu 22.04 LTS or Proxmox VE
• Docker and Kubernetes
• 1 TB+ storage
• Network infrastructure

Hardware and Software Requirements

| Component | Development | Production | |-----------|-------------|------------| | CPU | 16 cores | 32+ cores | | RAM | 64 GB | 256+ GB | | GPU | 1x RTX 4090 | 4x A100/H100 | | Storage | 2 TB NVMe | 8+ TB NVMe | | Network | 1 GbE | 10/25 GbE |

Step-by-Step Implementation

Phase 1: Infrastructure Design

#### Capacity Planning

| Workload | Users | Models | GPU Servers | Storage | |----------|-------|--------|-------------|---------| | Development | 1-5 | 1-3 | 1-2 | 2 TB | | Team | 5-20 | 3-10 | 2-4 | 8 TB | | Enterprise | 20+ | 10+ | 4+ | 32+ TB |

Phase 2: Hardware Deployment

# Server provisioning checklist
# 1. Install Ubuntu Server 22.04 LTS
# 2. Configure RAID for storage
# 3. Install NVIDIA drivers
# 4. Install Docker and Kubernetes
# 5. Configure network (static IPs, VLANs)
# 6. Set up monitoring

Phase 3: Kubernetes Deployment

# k8s/ai-namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: ai-platform
---
apiVersion: v1
kind: StorageClass
metadata:
  name: fast-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: ollama
  namespace: ai-platform
spec:
  serviceName: ollama
  replicas: 2
  selector:
    matchLabels:
      app: ollama
  template:
    metadata:
      labels:
        app: ollama
    spec:
      containers:
      - name: ollama
        image: ollama/ollama:latest
        ports:
        - containerPort: 11434
        resources:
          limits:
            nvidia.com/gpu: 1
        volumeMounts:
        - name: models
          mountPath: /root/.ollama
      nodeSelector:
        gpu: "true"
  volumeClaimTemplates:
  - metadata:
      name: models
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: fast-storage
      resources:
        requests:
          storage: 500Gi

Phase 4: Model Management

# Create model registry
# Pull models to local registry
docker pull ollama/ollama
docker tag ollama/ollama:latest registry.local:5000/ollama:latest
docker push registry.local:5000/ollama:latest

# Pull AI models
ollama pull llama3.1:8b
ollama pull deepseek-coder:6.7b
ollama pull codellama:13b

Phase 5: API Gateway and Authentication

# kong.yaml - API Gateway configuration
_format_version: "3.0"
_transform: true

services:
  - name: ollama-service
    url: http://ollama.ai-platform.svc.cluster.local:11434
    routes:
      - name: ollama-route
        paths:
          - /ollama

  - name: vllm-service
    url: http://vllm.ai-platform.svc.cluster.local:8000
    routes:
      - name: vllm-route
        paths:
          - /vllm

plugins:
  - name: key-auth
    config:
      hide_credentials: false
  - name: rate-limiting
    config:
      minute: 100
      policy: local

Phase 6: Monitoring and Observability

# prometheus.yaml - Monitoring configuration
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'ollama'
    static_configs:
      - targets: ['ollama.ai-platform.svc.cluster.local:11434']
  - job_name: 'vllm'
    static_configs:
      - targets: ['vllm.ai-platform.svc.cluster.local:8000']

alerting:
  rules:
    - alert: HighGPUMemory
      expr: gpu_memory_utilization > 0.9
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "High GPU memory usage"

Phase 7: Security Implementation

# Network policies
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: ai-network-policy
  namespace: ai-platform
spec:
  podSelector:
    matchLabels:
      app: ollama
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          name: ingress-nginx
    ports:
    - protocol: TCP
      port: 11434
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: ai-platform
    ports:
    - protocol: TCP
      port: 53

Phase 8: Backup and Disaster Recovery

# Backup model weights
docker run --rm \
  -v ollama-models:/source \
  -v $(pwd):/backup \
  alpine \
  tar czf /backup/ollama-models-$(date +%Y%m%d).tar.gz -C /source .

# Backup configurations
tar czf ai-infrastructure-config-$(date +%Y%m%d).tar.gz \
  k8s/ \
  docker-compose.yml \
  nginx.conf \
  monitoring/

Expected Output

• Private AI infrastructure deployed
• Multiple models served via API
• GPU cluster operational
• Monitoring and alerting configured
• Security policies enforced
• Backup and DR procedures documented

Verification Procedures

| Check | Command | Expected Result | |-------|---------|-----------------| | API accessible | `curl https://api.ai.internal/v1/models` | Model list returned | | GPU available | `kubectl get nodes --selector=gpu=true` | GPU nodes ready | | Models loaded | `ollama list` | Models available | | Monitoring | `kubectl get pods -n monitoring` | Prometheus/Grafana running |

Troubleshooting Scenarios

| Issue | Diagnosis | Resolution | |-------|-----------|------------| | Model not loading | Insufficient GPU memory | Use smaller model or quantization | | API slow | Network or GPU bottleneck | Check network; add more GPUs | | Pod crashing | OOM or configuration error | Check logs; increase memory | | Authentication failing | Invalid API key | Verify key; check Kong config |

Best Practices

• **Start small:** Begin with development infrastructure, scale to production
• **Use containerization:** Docker/Kubernetes for portability
• **Implement RBAC:** Restrict access to AI resources
• **Monitor costs:** Track GPU hours and storage costs
• **Plan for scale:** Design for multi-GPU, multi-node from start
• **Document everything:** Architecture, procedures, runbooks

Security Considerations

• **Data encryption:** Encrypt model weights at rest
• **Network segmentation:** Isolate AI infrastructure from other networks
• **Access control:** Use API keys and RBAC
• **Audit logging:** Log all API access and model usage
• **Vulnerability scanning:** Scan container images regularly

Conclusion

Building a private AI infrastructure requires careful planning across hardware, software, security, and operations. By following this tutorial, organizations can deploy a production-ready private AI platform that keeps data on-premises, reduces costs, and provides full control over AI workloads.

References

• Kubernetes: https://kubernetes.io/
• Ollama: https://ollama.com/
• vLLM: https://docs.vllm.ai/
• Prometheus: https://prometheus.io/
• Grafana: https://grafana.com/