Virtualization High Availability and Disaster Recovery Design

# Virtualization High Availability and Disaster Recovery Design

Introduction

High availability (HA) and disaster recovery (DR) are essential for virtualized environments hosting mission-critical workloads. HA protects against host failures with automatic VM restart, while DR protects against site-wide disasters through replication and recovery to a secondary site.

This tutorial covers virtualization HA design, fault tolerance, backup and replication strategies, stretched clusters, site recovery, and disaster recovery planning. You will learn how to design HA/DR architectures, configure HA clusters, implement replication, and validate disaster recovery procedures.

By the end of this guide, you will be able to design and implement high availability and disaster recovery for virtualized environments.

Technical Background

High Availability Concepts

**High Availability**: Automatically restarts VMs on different hosts when a host fails. Minimizes downtime (typically seconds to minutes).

**Fault Tolerance**: Continuous VM replication with zero downtime and zero data loss. VM runs simultaneously on two hosts; if primary fails, secondary takes over immediately.

**Disaster Recovery**: Recovery from site-wide disasters (fire, flood, power loss) through replication to a secondary site. Recovery time ranges from minutes to hours.

Key Metrics

| Metric | Definition | Typical Target | |--------|------------|----------------| | RTO (Recovery Time Objective) | Maximum acceptable downtime | HA: seconds-minutes; DR: hours | | RPO (Recovery Point Objective) | Maximum acceptable data loss | HA: near-zero; DR: minutes-hours | | MTTR (Mean Time To Repair) | Average time to restore service | Depends on procedure |

HA Architectures

| Architecture | Protection | Complexity | Cost | |-------------|------------|------------|------| | Single-host HA | Host reboot restart | Low | Low | | Cluster HA | Host failure VM restart | Medium | Medium | | Fault Tolerance | Zero downtime | High | High | | Stretched Cluster | Site failure | Very High | Very High | | Site-to-Site Replication | Disaster recovery | High | High |

Prerequisites

• Virtualization platform (Proxmox, VMware, Nutanix, XCP-ng, Hyper-V)
• Shared storage for cluster HA
• Network connectivity between hosts
• Secondary site for DR (replication target)
• Backup infrastructure

Architecture

HA Cluster Architecture

graph TB
    subgraph HA_Cluster [HA Cluster]
        H1[Host 1<br/>Running VM 100]
        H2[Host 2<br/>Running VM 200]
        H3[Host 3<br/>Running VM 300]
    end
    
    subgraph Shared_Storage [Shared Storage]
        SR[(Shared Storage<br/>VMFS/NFS/Ceph)]
    end
    
    subgraph Network [Network]
        SW[Cluster Network]
    end
    
    H1 <==> SW <==> H2
    H2 <==> SW <==> H3
    H3 <==> SW <==> H1
    
    H1 --> SR
    H2 --> SR
    H3 --> SR
    
    HA[HA Manager] --> H1
    HA --> H2
    HA --> H3

Disaster Recovery Architecture

Primary Site                          Secondary Site (DR)
┌─────────────────┐                 ┌─────────────────┐
│  Production     │──── Replication │    DR Standby   │
│  Cluster        │  (async/sync)   │    Cluster      │
│                 │                 │                 │
│  VMs Running    │◄── Failover/ ──▶│  VMs Recovered  │
│                 │    Failback     │                 │
└─────────────────┘                 └─────────────────┘

Replication Types

| Type | Frequency | RPO | Use Case | |------|-----------|-----|----------| | Synchronous | Real-time | 0 seconds | Metro clusters, zero data loss | | Asynchronous | Scheduled | minutes-hours | DR, remote sites | | Continuous | Continuous | seconds | Near-zero data loss |

Step-by-Step Implementation

Step 1: Design HA Cluster

**Purpose**: Plan HA cluster for automatic VM restart on host failure.

**Design Decisions**:

| Decision | Recommendation | |----------|---------------| | Host count | 3 nodes minimum (odd number for quorum) | | Shared storage | Required (NFS, iSCSI, Ceph, VMFS) | | Network | Dedicated HA/heartbeat network | | HA policy | Restart priority per VM | | Admission control | Prevent overcommitment |

**Commands**: None (design phase)

**Expected output**: Documented HA cluster design.

Step 2: Configure HA Cluster (Proxmox Example)

**Purpose**: Enable HA on Proxmox VE cluster.

**Commands**:

# Create HA group
ha-manager addgroup ha-group --nodes pve01,pve02,pve03

# Add VM to HA management
ha-manager add vm:100 --group ha-group --state started
ha-manager add vm:101 --group ha-group --state started

# Verify HA resources
ha-manager status

# Test HA - put host in maintenance mode
ha-manager add vm:100 --group ha-group --state started

# Migrate all VMs from host before maintenance
qm migrate 100 pve02 --online
qm migrate 101 pve03 --online

**Explanation**: HA monitors VMs and restarts them on other nodes if a host fails. Requires shared storage so VM disks are accessible from all hosts.

**Expected output**: VMs managed by HA; automatic restart on host failure.

**Verification**:

ha-manager status
# Shows VMs with HA state: started

Step 3: Configure Fault Tolerance (VMware Example)

**Purpose**: Zero-downtime protection for critical VMs.

**Procedure** (vSphere Client):

1. Right-click VM → Turn On Fault Tolerance 2. Select FT network (dedicated 10 GbE) 3. Select secondary host 4. Enable FT 5. Monitor FT status

**Explanation**: FT runs primary and secondary VM in lockstep. If primary host fails, secondary takes over with zero downtime and zero data loss.

**Expected output**: FT enabled; VM protected against host failure with zero downtime.

**Verification**:

# vSphere Client shows FT status: Enabled
# Secondary VM running on different host

Step 4: Implement Backup and Replication

**Purpose**: Protect against data loss and site disasters.

**Commands** (Proxmox Backup Server):

# Create backup job (daily)
# Via Proxmox web interface: Datacenter → Backup

# Replicate to DR site (Proxmox Backup Server sync)
# Via PBS: Datastore → Replication

**Replication schedule example**:

# Daily incremental replication to DR site
Schedule: 0 2 * * *
Retention: keep-daily=7, keep-weekly=4
Target: pbs-dr-site:vm-backups

**Expected output**: VMs backed up and replicated to DR site.

**Verification**:

# Check backup status
pvesm status
proxmox-backup-client snapshot list vm-backups:vm/100

Step 5: Configure Stretched Cluster (Optional)

**Purpose**: Protect against site failure with synchronous replication across sites.

**Procedure** (platform-specific):

• VMware vSphere Metro Storage Cluster (vMSC)
• Proxmox with Ceph across sites
• Nutanix Metro Availability

**Requirements**:

• Low-latency network between sites (< 5 ms round trip)
• Shared storage with synchronous replication
• Both sites active simultaneously
• Quorum witness at third site

**Expected output**: Cluster spans both sites; automatic failover between sites.

Step 6: Disaster Recovery Plan and Testing

**Purpose**: Document and test DR procedures.

**DR Runbook Contents**:

1. Disaster declaration criteria 2. Roles and responsibilities 3. Failover procedures (primary → secondary) 4. Failback procedures (secondary → primary) 5. Communication plan 6. Validation checklist 7. Recovery time targets

**Testing Procedure**:

1. Schedule DR test during maintenance window 2. Simulate primary site failure 3. Failover VMs to secondary site 4. Validate application functionality 5. Document results and issues 6. Failback to primary site

**Commands** (failover example):

# On DR site - start replicated VMs
qm start 100
qm start 101

# Update DNS to point to DR site IP
# Verify application access

**Expected output**: DR test completed; RTO and RPO validated.

**Verification**:

• VMs running on secondary site
• Applications accessible
• Data consistent with last replication
• Documented recovery time

Configuration Examples

HA Configuration (Proxmox)

# /etc/pve/ha/resources.cfg
vm:100
    group ha-group
    state started

vm:101
    group ha-group
    state started

# /etc/pve/ha/groups.cfg
ha-group
    nodes pve01,pve02,pve03
    no-quorum-policy stop

Replication Job Configuration

# Proxmox Backup Server replication
Source: pbs-primary:vm-backups
Target: pbs-dr:vm-backups
Schedule: Daily at 02:00
Retention: keep-daily=7, keep-weekly=4, keep-monthly=12
Encryption: Enabled

VM Priority Configuration

# HA priority (higher = more important)
vm:100    priority=1   # Production database - restart first
vm:101    priority=2   # Application server
vm:102    priority=3   # Development server - restart last

DR Network Configuration

# Primary site: 10.1.10.0/24
# DR site: 10.1.20.0/24
# DNS: Update VM records to DR site IPs during failover
# Load balancer: Update to DR site VIP

Verification

HA Verification

# Test host failure
# 1. Mark host for maintenance or power off
# 2. Verify VMs restart on other hosts
# 3. Check ha-manager status
ha-manager status

# Verify VM is running on new host
qm status 100

DR Verification

# Test failover
# 1. Simulate primary site outage
# 2. Start VMs on DR site
# 3. Verify application functionality
# 4. Document recovery time

# Test failback
# 1. Fix primary site
# 2. Replicate changes back
# 3. Failback to primary

Expected Results

• HA: VMs restart on surviving hosts within configured time
• DR: VMs recovered at secondary site within RTO
• Data loss within RPO
• Applications functional after recovery

Troubleshooting

| Symptom | Possible Cause | Diagnostic Command | Corrective Action | Verification | |---------|---------------|-------------------|-------------------|--------------| | HA not restarting VMs | No shared storage | `pvesm status` | Configure shared storage | HA restarts VMs | | HA quorum lost | Too many hosts down | `pvecm status` | Restore hosts or add quorum device | Quorum regained | | VM starts on wrong host | HA group misconfigured | `ha-manager status` | Adjust HA group nodes | VM starts on correct host | | Replication fails | Network or auth issue | Backup logs | Fix network; verify credentials | Replication succeeds | | DR site VMs stale | Replication lag | PBS snapshot list | Increase replication frequency | Data current | | Split-brain | Both sites active | DR orchestration logs | Failover one site; verify | Single site active | | Failback fails | Data changed at DR | Replication logs | Reverse replication; test | Failback succeeds |

Best Practices

1. **Define RTO and RPO**: Align HA/DR design with business requirements. 2. **Test regularly**: Quarterly HA tests, annual DR tests. 3. **Document procedures**: Written runbooks for failover and failback. 4. **Use shared storage for HA**: Required for VM restart on different hosts. 5. **Monitor HA health**: Alert on host failures and HA events. 6. **Prioritize VMs**: Critical VMs restart first during HA events. 7. **Test network failover**: DNS and load balancer updates during DR. 8. **Document recovery time**: Track actual RTO/RPO vs. targets. 9. **Use encryption**: Protect replicated data in transit and at rest. 10. **Review annually**: Update DR plan as infrastructure changes.

Conclusion

High availability and disaster recovery protect virtualized environments from host failures and site-wide disasters. HA provides automatic VM restart within a cluster, while DR enables recovery at a secondary site. Successful HA/DR requires shared storage, proper network design, documented procedures, and regular testing. Define clear RTO/RPO targets, prioritize critical VMs, and validate recovery procedures to ensure business continuity when failures occur.

References

• Proxmox VE High Availability (https://pve.proxmox.com/wiki/High_Availability)
• VMware vSphere HA Documentation (https://docs.vmware.com/)
• VMware vSphere Fault Tolerance (https://docs.vmware.com/)
• Nutanix HA and DR Guide (https://portal.nutanix.com/)
• NIST SP 800-34 Rev. 1 - Contingency Planning Guide (https://csrc.nist.gov/)
• ISO 22301 - Business Continuity Management (https://www.iso.org/)