GPU Passthrough for Virtualized AI Workloads

# GPU Passthrough for Virtualized AI Workloads

Introduction

GPU passthrough allows virtual machines to directly access physical GPU hardware, enabling AI workloads in virtualized environments with near-native performance. This tutorial covers VFIO, SR-IOV, KVM, and Proxmox configuration for GPU passthrough.

Learning Objectives

By the end of this tutorial, you will be able to:

• Understand GPU passthrough concepts and technologies
• Configure VFIO for GPU passthrough
• Set up SR-IOV for GPU sharing
• Configure KVM/QEMU with GPU passthrough
• Implement GPU passthrough in Proxmox
• Troubleshoot common passthrough issues

Technical Background

GPU Passthrough Architecture

flowchart TB
    Host[Host OS] --> Hypervisor[KVM Hypervisor]
    Hypervisor --> VM1[VM 1 - AI Workload]
    Hypervisor --> VM2[VM 2 - AI Workload]
    
    GPU[Physical GPU] --> VFIO[VFIO Driver]
    VFIO --> VM1
    VFIO --> VM2
    
    subgraph Passthrough Methods
    VFIO
    SR-IOV[SR-IOV Virtual Functions]
    end
    
    GPU --> SR-IOV

Passthrough Methods

| Method | Description | Best For | Complexity | |--------|-------------|----------|------------| | VFIO Passthrough | Dedicate entire GPU to one VM | Single AI workload | Medium | | SR-IOV | Split GPU into multiple virtual functions | Multiple AI workloads | High | | Mediated Devices | Share GPU between VMs (NVIDIA vGPU) | Multiple AI workloads | High | | Emulated GPU | Software-emulated graphics | Basic display only | Low |

IOMMU Groups

IOMMU (Input-Output Memory Management Unit) groups determine which devices can be isolated for passthrough. Devices in the same IOMMU group must be passed through together.

Prerequisites

• CPU with VT-d (Intel) or AMD-Vi (AMD) support
• Motherboard with IOMMU support
• Compatible GPU
• Linux host (Ubuntu, Proxmox, etc.)
• 32+ GB RAM

Hardware and Software Requirements

| Component | Requirement | |-----------|-------------| | CPU | Intel VT-d or AMD-Vi capable | | Motherboard | IOMMU support, separate IOMMU groups | | GPU | NVIDIA or AMD with passthrough support | | Host OS | Ubuntu 22.04, Proxmox VE, etc. | | Hypervisor | KVM/QEMU |

Step-by-Step Implementation

Phase 1: Enable IOMMU

#### Intel VT-d

# Edit GRUB configuration
sudo nano /etc/default/grub

# Add to GRUB_CMDLINE_LINUX_DEFAULT:
intel_iommu=on iommu=pt

# Update GRUB
sudo update-grub

# Reboot
sudo reboot

# Verify IOMMU is enabled
dmesg | grep -i iommu

#### AMD-Vi

# Edit GRUB configuration
sudo nano /etc/default/grub

# Add to GRUB_CMDLINE_LINUX_DEFAULT:
amd_iommu=on iommu=pt

# Update GRUB
sudo update-grub

# Reboot
sudo reboot

Phase 2: Check IOMMU Groups

# Script to list IOMMU groups
for d in /sys/kernel/iommu_groups/*/devices/*; do
    n=${d#*/iommu_groups/*}; n=${n%%/*}
    printf 'IOMMU Group %s ' "$n"
    lspci -nns "${d##*/}"
done

Phase 3: Install VFIO Drivers

# Load VFIO modules
sudo modprobe vfio
sudo modprobe vfio_pci
sudo modprobe vfio_iommu_type1
sudo modprobe vfio_virqfd

# Make modules load at boot
sudo tee /etc/modules-load.d/vfio.conf << 'EOF'
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
EOF

# Bind GPU to VFIO (replace with your GPU PCI IDs)
echo "10de 1234" | sudo tee /sys/bus/pci/drivers/vfio-pci/new_id
echo "10de 1235" | sudo tee /sys/bus/pci/drivers/vfio-pci/new_id

Phase 4: KVM/QEMU GPU Passthrough

#### Create VM with GPU Passthrough

# Create VM with virt-install
sudo virt-install \
  --name ai-vm \
  --ram 32768 \
  --vcpus 16 \
  --disk path=/var/lib/libvirt/images/ai-vm.qcow2,size=200 \
  --os-variant ubuntu22.04 \
  --network bridge=virbr0 \
  --graphics none \
  --cpu host \
  --hostdev 0000:01:00.0 \  # GPU VGA
  --hostdev 0000:01:00.1 \  # GPU Audio
  --boot uefi \
  --location /path/to/ubuntu.iso

#### QEMU Command Line Example

qemu-system-x86_64 \
  -enable-kvm \
  -cpu host \
  -smp 16 \
  -m 32G \
  -drive file=/var/lib/libvirt/images/ai-vm.qcow2,format=qcow2 \
  -netdev user,id=net0 \
  -device virtio-net-pci,netdev=net0 \
  -device vfio-pci,host=0000:01:00.0,multifunction=on \
  -device vfio-pci,host=0000:01:00.1 \
  -bios /usr/share/OVMF/OVMF_CODE.fd \
  -vga none

Phase 5: Proxmox VE GPU Passthrough

#### Enable IOMMU in Proxmox

# Edit Proxmox GRUB
nano /etc/default/grub

# Add to GRUB_CMDLINE_LINUX_DEFAULT:
intel_iommu=on iommu=pt

# Update GRUB
update-grub

# Reboot
reboot

#### Add PCI Device to VM

1. In Proxmox web interface, go to your VM 2. Click "Hardware" → "Add" → "PCI Device" 3. Select your GPU (VGA and Audio) 4. Set "PCI Express" and "All Functions" options 5. Start the VM

#### Configure VM for Best Performance

# In /etc/pve/qemu-server/<vmid>.conf
args: -cpu 'host,hv_time,kvm=on,hv=on,+invtsc,+pdpe1gb'
machine: q35
bios: ovmf
machine: q35

Phase 6: SR-IOV Configuration

#### Enable SR-IOV on GPU

# Check SR-IOV support
lspci -s 01:00.0 -v | grep -i sriov

# Enable virtual functions (example for NVIDIA)
echo 4 | sudo tee /sys/bus/pci/devices/0000:01:00.0/sriov_numvfs

# Virtual functions will appear as separate PCI devices
lspci | grep -i nvidia

#### Assign Virtual Function to VM

# Bind VF to VFIO driver
echo "10de 1234" | sudo tee /sys/bus/pci/drivers/vfio-pci/new_id

# Pass VF to VM
sudo virt-install \
  --name ai-vm-sriov \
  --ram 16384 \
  --vcpus 8 \
  --hostdev 0000:02:00.0  # Virtual Function

Expected Output

• GPU passthrough working in VM
• Near-native GPU performance in VM
• Multiple VMs can use GPU (with SR-IOV)
• AI workloads running in virtualized environment

Verification Procedures

| Check | Command | Expected Result | |-------|---------|-----------------| | IOMMU enabled | `dmesg | grep -i iommu` | IOMMU active | | IOMMU groups | List IOMMU groups | GPU in separate group | | VFIO driver | `lspci -nnk | grep -A3 VGA` | vfio-pci driver | | GPU in VM | `nvidia-smi` inside VM | GPU detected |

Troubleshooting Scenarios

| Issue | Diagnosis | Resolution | |-------|-----------|------------| | IOMMU not enabled | BIOS setting disabled | Enable VT-d/AMD-Vi in BIOS | | GPU not in separate group | IOMMU group not isolated | ACS override patch | | Driver not binding to VFIO | Wrong PCI IDs | Verify PCI IDs | | VM won't boot with GPU | UEFI/BIOS mismatch | Use OVMF/UEFI | | Poor performance | Passthrough not working | Verify VFIO binding |

Best Practices

• **Use separate IOMMU groups:** Ensure GPU is isolated
• **Use UEFI/OVMF:** Required for modern GPUs
• **Bind audio function:** Pass through both VGA and audio
• **Disable host GPU drivers:** Prevent host from claiming GPU
• **Test with single VM first:** Verify before scaling to multiple VMs

Security Considerations

• **IOMMU protection:** Prevents DMA attacks from VMs
• **GPU firmware:** Keep GPU firmware updated
• **VM isolation:** Prevent VM-to-VM GPU access
• **Host protection:** Host cannot use GPU while passed through

Conclusion

GPU passthrough enables near-native AI performance in virtualized environments. By configuring VFIO, SR-IOV, and KVM/QEMU properly, organizations can run AI workloads in VMs with minimal performance overhead while maintaining the flexibility of virtualization.

References

• VFIO Documentation: https://docs.kernel.org/vfio/
• KVM GPU Passthrough: https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF
• Proxmox GPU Passthrough: https://pve.proxmox.com/wiki/Pci_passthrough
• SR-IOV: https://www.kernel.org/doc/html/latest/driver-api/pci-sriov.html