Building and Managing a Proxmox Cluster

3 Nov 2026, 18:2124 min readby ITYantra Admin

Complete guide to building and managing a Proxmox VE cluster including corosync, Ceph storage, high availability, and live migration.

# Building and Managing a Proxmox Cluster

Introduction

A Proxmox VE cluster combines multiple Proxmox nodes into a single manageable resource pool, enabling high availability (HA), live migration of virtual machines, shared storage, and centralized management. Clusters provide redundancy, scalability, and simplified operations for virtualized environments.

This tutorial covers building a multi-node Proxmox cluster, configuring corosync cluster communication, setting up Ceph distributed storage, implementing high availability for virtual machines, and performing live migration. You will learn cluster design, deployment, validation, and day-2 operations.

By the end of this guide, you will be able to build and manage a production-ready Proxmox cluster with shared storage and high availability.

Technical Background

Cluster Components

**Corosync**: Cluster communication and membership protocol. Provides:

  • Node membership tracking
  • Quorum calculation
  • Cluster messaging
  • Heartbeat between nodes

**PMXCFS (Proxmox Cluster File System)**: Database-driven filesystem storing cluster configuration. Replicated across all nodes via corosync.

**Ceph**: Distributed storage system providing:

  • Object storage (RADOS)
  • Block devices (RBD)
  • Filesystem (CephFS)
  • Self-healing and replication

**High Availability (HA)**: Automatically restarts VMs and containers on different nodes if a host fails.

**Live Migration**: Moves running VMs between nodes without downtime using shared storage and memory migration.

Quorum

Quorum is the minimum number of nodes required for the cluster to operate. With N nodes, quorum is N/2 + 1 (rounded up). A 3-node cluster requires 2 nodes for quorum. Without quorum, the cluster stops managing resources to prevent split-brain scenarios.

Split-Brain

Split-brain occurs when cluster nodes lose communication and each group believes it is the only active group. This can cause duplicate VM startups and data corruption. Quorum and fencing prevent split-brain.

Prerequisites

  • 3 or more Proxmox VE nodes (odd number recommended for quorum)
  • Identical Proxmox versions on all nodes
  • Static IP addresses on all nodes
  • Passwordless SSH between nodes (for cluster setup)
  • Shared storage (Ceph, NFS, iSCSI) or local storage with replication
  • Dedicated cluster network (recommended)
  • Maintenance window for cluster creation

Architecture

3-Node Cluster with Ceph

graph TB
    subgraph Management Network
        SW1[(Management Switch)]
    end
    subgraph Storage Network
        SW2[(Storage Switch)]
    end
    
    P1[Proxmox Node 1<br/>192.168.1.10<br/>10.0.1.10]
    P2[Proxmox Node 2<br/>192.168.1.11<br/>10.0.1.11]
    P3[Proxmox Node 3<br/>192.168.1.12<br/>10.0.1.12]
    
    P1 --- SW1
    P2 --- SW1
    P3 --- SW1
    
    P1 --- SW2
    P2 --- SW2
    P3 --- SW2
    
    P1 <==> Corosync <==> P2
    P2 <==> Corosync <==> P3
    P3 <==> Corosync <==> P1
    
    P1 <==> Ceph <==> P2
    P2 <==> Ceph <==> P3

Quorum Calculation

| Cluster Size | Quorum Required | Can Tolerate | |-------------|-----------------|---------------| | 1 node | 1 | 0 failures | | 2 nodes | 2 | 0 failures (no redundancy) | | 3 nodes | 2 | 1 failure | | 4 nodes | 3 | 1 failure | | 5 nodes | 3 | 2 failures |

**Recommendation**: Use 3 nodes minimum; avoid 2-node clusters without a quorum device.

Step-by-Step Implementation

Step 1: Prepare Nodes for Clustering

**Purpose**: Ensure all nodes are ready for cluster formation.

**Commands** (on each node):

# Verify Proxmox versions match
pveversion

# Set static hostname
hostnamectl set-hostname pve01.ityantra.lab

# Add all nodes to /etc/hosts
cat >> /etc/hosts << 'EOF'
192.168.1.10 pve01.ityantra.lab pve01
192.168.1.11 pve02.ityantra.lab pve02
192.168.1.12 pve03.ityantra.lab pve03
EOF

# Test SSH connectivity between nodes
ssh root@pve02 hostname
ssh root@pve03 hostname

**Explanation**: All nodes must have identical Proxmox versions. Passwordless SSH is required during cluster creation for key exchange.

**Expected output**: All nodes reachable via SSH; hostnames resolve correctly.

**Verification**:

ping pve02
ping pve03

Step 2: Create Cluster on First Node

**Purpose**: Initialize the cluster on the first Proxmox node.

**Commands** (on pve01 only):

# Create cluster
pvecm create ityantra-cluster --cluster_network 192.168.1.0/24

# Verify cluster creation
pvecm status
pvecm nodes

**Explanation**: `pvecm create` initializes corosync and PMXCFS on the first node. The `--cluster_network` parameter specifies the network subnet for cluster communication.

**Expected output**: Cluster created with one node; `pvecm status` shows quorum OK.

**Verification**:

pvecm status
# Shows: Cluster status: quorate
#        Node count: 1

Step 3: Join Additional Nodes to Cluster

**Purpose**: Add remaining nodes to the existing cluster.

**Commands** (on pve02 and pve03):

# Join cluster
pvecm add 192.168.1.10

# Verify membership
pvecm status
pvecm nodes

**Explanation**: `pvecm add` connects the node to the cluster, exchanges corosync keys, and joins the PMXCFS cluster filesystem. The node will reboot corosync services.

**Expected output**: All 3 nodes visible in cluster; quorum shows 2 of 3.

**Verification**:

pvecm nodes
# Shows all 3 nodes with their IDs and online status

Step 4: Configure Ceph Distributed Storage

**Purpose**: Create shared Ceph storage across all cluster nodes.

**Commands** (on any node):

# Initialize Ceph on all nodes
pveceph init --network 10.0.1.0/24

# Create Ceph pool for VM images
pveceph createpool replicas --size 3 --min_size 2

# Create storage in Proxmox
pvesm add cephrbd ceph-pool --pool replicas --content images,rootdir

**Explanation**: Ceph distributes data across all nodes with 3 copies (size=3), tolerating one node failure. `min_size=2` allows writes with at least 2 copies.

**Expected output**: Ceph pool created; Proxmox storage pool available for VM creation.

**Verification**:

ceph status
pvesm status

Step 5: Configure High Availability (HA)

**Purpose**: Enable automatic VM restart on node failure.

**Commands**:

# Create HA group
ha-manager addgroup ha-group --nodes pve01,pve02,pve03

# Add VM to HA
ha-manager add vm:100 --group ha-group --state started

# Verify HA resources
ha-manager status

**Explanation**: HA monitors VMs and containers. If a node fails, HA restarts the VMs on another node in the group. Requires shared storage (Ceph, NFS, iSCSI).

**Expected output**: VM 100 managed by HA; status shows "started".

**Verification**:

ha-manager status
pvesr status  # Proxmox VE Server status

Step 6: Perform Live Migration

**Purpose**: Move a running VM from one node to another without downtime.

**Commands**:

# Migrate VM 100 from pve01 to pve02
qm migrate 100 pve02 --online

# Verify migration
qm status 100
pvesr status

**Explanation**: Live migration copies VM memory to the destination node while the VM continues running, then switches execution with minimal downtime (typically milliseconds). Requires shared storage.

**Expected output**: VM 100 running on pve02 without interruption.

**Verification**:

pvecm status
# Shows VM on new node
qm migrate 100 pve01 --online  # Move back if needed

Configuration Examples

Corosync Configuration (`/etc/pve/corosync.conf`)

totem {
    version: 2
    cluster_name: ityantra-cluster
    transport: udpu
    link_mode: unicast
    interface {
        ringnumber: 0
        bindnetaddr: 192.168.1.0
        mcastaddr: 239.10.10.10
        mcastport: 5405
        ttl: 1
    }
    interface {
        ringnumber: 1
        bindnetaddr: 10.0.1.0
        mcastaddr: 239.10.10.11
        mcastport: 5406
        ttl: 1
    }
}
quorum {
    provider: corosync_votequorum
    expected_votes: 3
}
nodelist {
    node {
        name: pve01
        nodeid: 1
        ring0_addr: 192.168.1.10
        ring1_addr: 10.0.1.10
    }
    node {
        name: pve02
        nodeid: 2
        ring0_addr: 192.168.1.11
        ring1_addr: 10.0.1.11
    }
    node {
        name: pve03
        nodeid: 3
        ring0_addr: 192.168.1.12
        ring1_addr: 10.0.1.12
    }
}

Ceph Configuration Summary

Pool: replicas
Size: 3 (3 copies of each object)
Min size: 2 (minimum copies for write)
Crush rule: replicated_rule
PG count: 128 (per pool)

HA Configuration (`/etc/pve/ha/resources.cfg`)

vm:100
    group ha-group
    state started

Fencing Configuration Example

ha-manager addgroup ha-group --nodes pve01,pve02,pve03
# Fencing configured via Proxmox Backup Server or IPMI

Verification

Cluster Status Commands

pvecm status              -- Cluster quorum and node count
pvecm nodes               -- List cluster nodes
pvecm nodes --details     -- Detailed node information
ceph status               -- Ceph health and status
ceph osd status           -- Ceph OSD status
ha-manager status         -- HA resources status
pvesm status              -- Storage pool status

Expected Output

Cluster name: ityantra-cluster
Cluster status: quorate
Node count: 3
Node online: 3
Expected votes: 3

Live Migration Verification

# Before migration
qm status 100  # Shows running on pve01

# Migrate
qm migrate 100 pve02 --online

# After migration
qm status 100  # Shows running on pve02

Troubleshooting

| Symptom | Possible Cause | Diagnostic Command | Corrective Action | Verification | |---------|---------------|-------------------|-------------------|--------------| | Node won't join cluster | SSH key mismatch or version mismatch | `pvecm add <ip>` verbose | Regenerate SSH keys; match Proxmox versions | Node joins successfully | | No quorum | Node count below threshold | `pvecm status` | Add more nodes or configure quorum device | Quorum achieved | | Split-brain | Network partition between nodes | `pvecm status`; `corosync-cfgtool -s` | Restore network connectivity; check fencing | Cluster reconciles | | Ceph health degraded | OSD down or PGs unclean | `ceph health detail` | Replace failed disk; rebalance cluster | HEALTH_OK | | VM won't migrate | No shared storage | `pvesm status` | Configure shared storage (Ceph/NFS) | Migration succeeds | | HA not failing over | Shared storage missing | `ha-manager status` | Add shared storage; re-add HA resource | Failover tested | | Slow cluster communication | Wrong network interface | `corosync-cfgtool -s` | Configure correct ring network | Low latency between nodes | | Node fenced incorrectly | Network timeout too short | Corosync logs | Increase timeout values | No false fencing | | PMXCFS out of sync | Corosync ring issue | `pvecm status` | Restart corosync; check ring status | Configuration syncs |

Best Practices

1. **Use odd number of nodes**: 3 or 5 nodes for proper quorum; avoid 2-node clusters without quorum device. 2. **Use dedicated cluster network**: Separate corosync traffic from VM traffic for performance and reliability. 3. **Enable redundant rings**: Configure two corosync rings for network redundancy. 4. **Use shared storage for HA**: Ceph, NFS, or iSCSI required for live migration and HA. 5. **Test failover regularly**: Verify HA works during maintenance windows. 6. **Monitor Ceph health**: Check `ceph status` daily; address warnings immediately. 7. **Use consistent versions**: All cluster nodes must run identical Proxmox versions. 8. **Backup cluster configuration**: PMXCFS is replicated, but maintain off-cluster backups. 9. **Configure fencing**: Use IPMI or PDU fencing to isolate failed nodes. 10. **Document node roles**: Record each node's IP, hostname, and purpose in the cluster.

Conclusion

Building a Proxmox cluster transforms individual virtualization hosts into a resilient, scalable resource pool. With corosync for cluster communication, Ceph for shared storage, HA for automatic failover, and live migration for maintenance without downtime, a Proxmox cluster delivers enterprise-class virtualization on open-source software. Successful cluster deployment requires careful network planning, consistent node configuration, and regular validation of failover and recovery procedures.

References

  • Proxmox VE Cluster Documentation (https://pve.proxmox.com/wiki/Cluster)
  • Proxmox VE High Availability (https://pve.proxmox.com/wiki/High_Availability)
  • Proxmox VE Ceph Integration (https://pve.proxmox.com/wiki/Ceph)
  • Corosync Cluster Engine (https://corosync.github.io/corosync/)
  • Ceph Documentation (https://docs.ceph.com/)
  • Proxmox VE Live Migration (https://pve.proxmox.com/wiki/Live_Migration)
#proxmox#cluster#corosync#ceph#high-availability#live-migration#virtualization#proxmox-ve