Kubernetes High Availability

# Kubernetes High Availability

Introduction

High availability in Kubernetes ensures that applications and cluster components remain operational despite failures. This tutorial covers multi-control-plane clusters, etcd high availability, pod disruption budgets, node redundancy, and disaster recovery strategies.

Learning Objectives

By the end of this tutorial, you will be able to:

• Configure multi-control-plane Kubernetes clusters
• Set up etcd high availability
• Implement Pod Disruption Budgets
• Configure node redundancy and anti-affinity
• Plan disaster recovery and etcd backup/restore
• Design clusters for zero-downtime operations

Technical Background

HA Architecture

flowchart TD
    LB[Load Balancer] --> CP1[Control Plane 1]
    LB --> CP2[Control Plane 2]
    LB --> CP3[Control Plane 3]
    
    CP1 --> ETCD1[(etcd 1)]
    CP2 --> ETCD2[(etcd 2)]
    CP3 --> ETCD3[(etcd 3)]
    
    ETCD1 <-->|Raft| ETCD2
    ETCD2 <-->|Raft| ETCD3
    
    CP1 --> WN1[Worker 1]
    CP2 --> WN2[Worker 2]
    CP3 --> WN3[Worker 3]
    
    subgraph Control Plane
    CP1
    CP2
    CP3
    end
    
    subgraph etcd Cluster
    ETCD1
    ETCD2
    ETCD3
    end

HA Components

| Component | HA Strategy | |-----------|-------------| | Control Plane | 3+ nodes behind load balancer | | etcd | 3+ nodes with Raft consensus | | Worker Nodes | 3+ nodes across availability zones | | Pods | Multiple replicas with PDB | | Ingress | Multiple ingress controllers |

Prerequisites

• 3+ servers for control plane
• 3+ servers for workers
• Load balancer
• Shared storage for etcd (optional)

Hardware and Software Requirements

| Component | Requirement | |-----------|-------------| | Control Plane | 3+ nodes, 4+ GB RAM each | | Worker Nodes | 3+ nodes, 4+ GB RAM each | | Load Balancer | HAProxy, NGINX, or cloud LB | | etcd | 3+ nodes, odd number |

Step-by-Step Implementation

Phase 1: Multi-Control-Plane Setup

#### Initialize First Control Plane

sudo kubeadm init \
  --control-plane-endpoint=10.0.0.100:6443 \
  --upload-certs \
  --pod-network-cidr=10.244.0.0/16

#### Join Additional Control Planes

sudo kubeadm join 10.0.0.100:6443 \
  --token <token> \
  --discovery-token-ca-cert-hash <hash> \
  --control-plane \
  --certificate-key <key>

Phase 2: etcd High Availability

#### etcd Cluster Configuration

# etcd.yaml
apiVersion: "etcd.database.coreos.com/v1beta2"
kind: EtcdCluster
metadata:
  name: k8s-etcd
spec:
  size: 3
  version: 3.5.0
  pod:
    antiAffinity: preferred

#### etcd Backup

# Backup etcd
ETCDCTL_API=3 etcdctl snapshot save /backup/etcd-snapshot.db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

# Restore etcd
ETCDCTL_API=3 etcdctl snapshot restore /backup/etcd-snapshot.db \
  --data-dir=/var/lib/etcd-new

Phase 3: Pod Disruption Budgets

# pdb.yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: web-pdb
  namespace: production
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: web
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: db-pdb
  namespace: production
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app: database

Phase 4: Node Redundancy

#### Pod Anti-Affinity

# anti-affinity.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 5
  template:
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchLabels:
                app: web
            topologyKey: kubernetes.io/hostname

#### Node Affinity (AZ Distribution)

# node-affinity.yaml
spec:
  affinity:
    nodeAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        preference:
          matchExpressions:
          - key: topology.kubernetes.io/zone
            operator: In
            values:
            - us-east-1a
            - us-east-1b
            - us-east-1c

Phase 5: Disaster Recovery

#### Backup Strategy

| Component | Backup Method | Frequency | |-----------|--------------|-----------| | etcd | etcdctl snapshot | Every 4 hours | | Persistent Volumes | Cloud snapshots | Daily | | ConfigMaps/Secrets | Git version control | On change | | Applications | Application-level backup | As needed |

#### Recovery Procedure

1. Restore etcd from snapshot 2. Recreate control plane nodes 3. Restore persistent volumes 4. Redeploy applications 5. Verify data integrity

Expected Output

• Multi-control-plane Kubernetes cluster
• etcd high availability with 3+ nodes
• Pod Disruption Budgets configured
• Pod anti-affinity for node distribution
• Disaster recovery procedures documented

Verification Procedures

| Check | Command | Expected Result | |-------|---------|-----------------| | Control plane nodes | `kubectl get nodes -l node-role.kubernetes.io/control-plane=` | 3+ nodes Ready | | etcd health | `ETCDCTL_API=3 etcdctl endpoint health` | All endpoints healthy | | PDB | `kubectl get pdb -A` | PDBs configured | | Pod distribution | `kubectl get pods -o wide` | Pods across nodes |

Troubleshooting Scenarios

| Issue | Diagnosis | Resolution | |-------|-----------|------------| | Control plane down | Load balancer or etcd issue | Check LB; restore etcd | | etcd quorum lost | Less than majority of etcd nodes | Restore from backup; add nodes | | PDB blocking drain | Not enough healthy replicas | Increase replicas; wait for healthy | | Pod stuck on one node | Anti-affinity not working | Check topology keys; verify labels |

Best Practices

• **3+ control plane nodes:** Odd number for etcd quorum
• **Across availability zones:** Distribute nodes across AZs
• **PDB for critical workloads:** Ensure availability during maintenance
• **Regular etcd backups:** Automated snapshots with offsite storage
• **Load balancer for API server:** Single entry point for control plane
• **Monitor etcd:** Track latency, disk usage, and quorum status

Security Considerations

• **etcd encryption:** Enable encryption at rest
• **API server TLS:** Use certificates for control plane communication
• **Network policies:** Isolate etcd traffic
• **Access control:** Restrict etcd access to control plane only

Conclusion

High availability in Kubernetes requires careful planning across control plane, etcd, worker nodes, and application workloads. By implementing multi-control-plane clusters, etcd clustering, PDBs, and disaster recovery procedures, you can build resilient Kubernetes infrastructure that survives failures with minimal downtime.

References

• Kubernetes High Availability: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/high-availability/
• etcd Clustering: https://etcd.io/docs/v3.5/ops/clustering/
• Pod Disruption Budgets: https://kubernetes.io/docs/tasks/run-application/configure-pdb/
• Kubernetes Disaster Recovery: https://kubernetes.io/docs/tasks/configure-pod-container/disaster-recovery/