Linux Server Backup and Recovery

# Linux Server Backup and Recovery

Introduction

Backup and recovery are essential for protecting Linux server data against hardware failure, accidental deletion, ransomware, and site disasters. A well-designed backup strategy ensures data can be restored within acceptable recovery time and recovery point objectives.

This tutorial covers backup strategies, tools (rsync, tar, rsnapshot, Bacula), full and incremental backups, restoration procedures, bare metal recovery, and disaster recovery planning.

By the end of this guide, you will be able to design and implement a Linux backup and recovery strategy.

Technical Background

Backup Types

| Backup Type | Description | Speed | Storage | Use Case | |-------------|-------------|-------|---------|----------| | Full | All selected data | Slow | High | Weekly/monthly | | Incremental | Changed since last backup | Fast | Low | Daily | | Differential | Changed since last full | Medium | Medium | Between full backups |

RTO and RPO

| Metric | Definition | Typical Target | |--------|------------|----------------| | RTO (Recovery Time Objective) | Maximum acceptable downtime | Hours to days | | RPO (Recovery Point Objective) | Maximum acceptable data loss | Hours to days |

3-2-1 Backup Rule

• **3** copies of data
• **2** different media types
• **1** copy off-site

Prerequisites

• Linux system with root or sudo access
• Backup storage (external disk, network share, backup server)
• Sufficient storage space for retention
• Understanding of backup concepts

Architecture

Backup Architecture

Production Server
├── /etc (configuration)
├── /var/www (web content)
├── /var/lib/mysql (databases)
└── /home (user data)
      │
      └── rsync/rsnapshot → Backup Server
            │
            ├── Daily (7 days)
            ├── Weekly (4 weeks)
            └── Monthly (12 months)

graph TB
    subgraph Production [Production Server]
        ETC[/etc]
        VAR[/var/www]
        MYSQL[/var/lib/mysql]
        HOME[/home]
    end
    
    subgraph Backup [Backup Server]
        DAILY[Daily Backups<br/>7 days]
        WEEKLY[Weekly Backups<br/>4 weeks]
        MONTHLY[Monthly Backups<br/>12 months]
    end
    
    subgraph Offsite [Off-site Storage]
        CLOUD[Cloud Storage]
        TAPE[Tape/LTO]
    end
    
    Production -->|rsync/rsnapshot| Backup
    Backup -->|replication| Offsite

Step-by-Step Implementation

Step 1: Design Backup Strategy

**Purpose**: Define what to back up, how often, and where to store.

**Backup Plan Example**:

| Data | Frequency | Type | Retention | Location | |------|-----------|------|-----------|----------| | /etc | Daily | Incremental | 30 days | Backup server | | /var/www | Daily | Incremental | 30 days | Backup server | | /var/lib/mysql | Daily | Incremental | 30 days | Backup server | | /home | Weekly | Full | 90 days | Backup server + off-site | | Full system | Monthly | Full | 1 year | Off-site |

**Commands**: None (planning phase)

**Expected output**: Documented backup strategy.

Step 2: Install Backup Tools

**Purpose**: Install backup software on production and backup servers.

**Commands** (Ubuntu/Debian):

# Install backup tools
sudo apt update
sudo apt install rsync tar rsnapshot

# Install Bacula (enterprise backup)
sudo apt install bacula-director bacula-storage bacula-client

**Commands** (RHEL/CentOS/Fedora):

sudo dnf install rsync tar rsnapshot

**Expected output**: Backup tools installed.

**Verification**:

which rsync tar rsnapshot
# All tools found

Step 3: Create File-Based Backup with tar

**Purpose**: Create compressed archive of critical directories.

**Commands**:

# Full backup of /etc
sudo tar -czpf /backup/etc-$(date +%Y%m%d).tar.gz /etc

# Full backup of /var/www
sudo tar -czpf /backup/www-$(date +%Y%m%d).tar.gz /var/www

# Backup with exclusion
sudo tar -czpf /backup/home-$(date +%Y%m%d).tar.gz \
  --exclude=/home/*/cache \
  --exclude=/home/*/.thumbnails \
  /home

# List archive contents
sudo tar -tzpf /backup/etc-$(date +%Y%m%d).tar.gz

# Extract archive
sudo tar -xzpf /backup/etc-$(date +%Y%m%d).tar.gz -C /restore/

**Expected output**: Compressed backup archives created.

**Verification**:

tar -tzpf /backup/etc-*.tar.gz | head -20
# Shows archive contents

Step 4: Create Incremental Backup with rsync

**Purpose**: Efficient incremental backups using rsync.

**Commands**:

# Basic rsync backup
sudo rsync -av --delete /etc/ /backup/etc/

# Rsync with compression
sudo rsync -avz --delete /var/www/ /backup/www/

# Rsync to remote server
sudo rsync -avz --delete /etc/ backup-server:/backup/server1/etc/

# Rsync with bandwidth limit
sudo rsync -avz --bwlimit=1000 --delete /home/ /backup/home/

**rsync options**:

• `-a`: Archive mode (preserves permissions, timestamps, symlinks)
• `-v`: Verbose
• `-z`: Compress during transfer
• `--delete`: Delete files in destination not in source
• `--bwlimit`: Limit bandwidth (KB/s)
• `--exclude`: Exclude patterns

**Expected output**: Files synchronized to backup location.

**Verification**:

ls -la /backup/etc/
# Shows backed-up files

Step 5: Configure rsnapshot for Automated Snapshots

**Purpose**: Set up automated rotating snapshots with hard links.

**Commands**:

# Configure rsnapshot
sudo nano /etc/rsnapshot.conf

**Example /etc/rsnapshot.conf**:

# Config file for rsnapshot
version               1.2
snapshot_root         /var/cache/rsnapshot/
no_create_root        1
cmd_cp                /bin/cp
cmd_rm                /bin/rm
cmd_rsync             /usr/bin/rsync
cmd_ssh               /usr/bin/ssh
cmd_logger            /usr/bin/logger
du                    0

# Backup intervals
retain        daily   7
retain        weekly  4
retain        monthly 1

# Backup points
backup        /etc/               localhost/
backup        /var/www/           localhost/
backup        /home/              localhost/
backup        root@server:/etc/   server1/

**Run rsnapshot**:

# Test configuration
sudo rsnapshot configtest

# Run daily snapshot
sudo rsnapshot daily

# Run weekly snapshot
sudo rsnapshot weekly

# Run monthly snapshot
sudo rsnapshot monthly

**Expected output**: Rotating snapshots with hard-linked unchanged files.

**Verification**:

ls /var/cache/rsnapshot/
# Shows daily.0, daily.1, ..., weekly.0, ..., monthly.0

Step 6: Restore Files from Backup

**Purpose**: Recover files from backup.

**Commands**:

# Restore from tar archive
sudo tar -xzpf /backup/etc-20240115.tar.gz -C /restore/

# Restore from rsync backup
sudo rsync -av /backup/etc/ /etc/

# Restore from rsnapshot
sudo rsync -av /var/cache/rsnapshot/daily.0/localhost/etc/ /etc/

# Restore specific file
sudo tar -xzpf /backup/www-20240115.tar.gz -C /restore/ var/www/html/index.html

**Expected output**: Files restored to target location.

**Verification**:

ls /restore/etc/
# Shows restored files

Step 7: Configure Automated Backup with Cron

**Purpose**: Schedule automated backups.

**Commands** (`/etc/cron.d/backups`):

# Daily rsync backup at 2 AM
0 2 * * * root rsync -av --delete /etc/ /backup/etc/ >> /var/log/rsync-backup.log 2>&1

# Daily rsnapshot at 3 AM
0 3 * * * root rsnapshot daily >> /var/log/rsnapshot.log 2>&1

# Weekly rsnapshot on Sunday at 1 AM
0 1 * * 0 root rsnapshot weekly >> /var/log/rsnapshot.log 2>&1

# Monthly rsnapshot on 1st at 1 AM
0 1 1 * * root rsnapshot monthly >> /var/log/rsnapshot.log 2>&1

# Full tar backup on Saturday at 2 AM
0 2 * * 6 root tar -czpf /backup/full-$(date +\%Y\%m\%d).tar.gz /etc /var/www /home >> /var/log/tar-backup.log 2>&1

**Expected output**: Automated backups run on schedule.

**Verification**:

ls /var/log/rsync-backup.log
# Shows backup execution logs

Configuration Examples

rsync Backup Script (`/usr/local/bin/backup.sh`)

#!/bin/bash
# Backup script with logging

BACKUP_DIR="/backup/$(date +%Y%m%d)"
SOURCE_DIRS="/etc /var/www /home"
LOG_FILE="/var/log/backup-$(date +%Y%m%d).log"

mkdir -p "$BACKUP_DIR"

for dir in $SOURCE_DIRS; do
    dirname=$(basename "$dir")
    echo "[$(date)] Backing up $dir" >> "$LOG_FILE"
    rsync -av --delete "$dir" "$BACKUP_DIR/$dirname" >> "$LOG_FILE" 2>&1
done

echo "[$(date)] Backup complete" >> "$LOG_FILE"

rsnapshot Exclude Patterns

# Add to /etc/rsnapshot.conf
exclude         /home/*/.cache
exclude         /home/*/.thumbnails
exclude         /var/cache/*
exclude         /var/tmp/*
exclude         /tmp/*

Bacula Director Configuration (Example)

# /etc/bacula/bacula-dir.conf (simplified)
Job {
  Name = "LinuxBackup"
  Type = Backup
  Client = linux-client-fd
  FileSet = "Linux FileSet"
  Schedule = "WeeklyCycle"
  Storage = File
  Pool = File
  Write Bootstrap = "/var/lib/bacula/%c.bsr"
}

Verification

Backup Verification Commands

tar -tzpf /backup/etc-*.tar.gz    -- List tar archive contents
rsync --dry-run -av /etc/ /backup/etc/  -- Test rsync without changes
rsnapshot configtest              -- Validate rsnapshot config
ls /var/cache/rsnapshot/          -- List snapshots
df -h /backup                     -- Check backup storage space

Restore Testing Checklist

[ ] Select backup file or snapshot
[ ] Choose restore destination
[ ] Restore files
[ ] Verify file contents
[ ] Verify permissions and ownership
[ ] Test application functionality
[ ] Document restore time

Expected Results

• Backup archives contain expected files
• rsync destination matches source
• rsnapshot snapshots contain correct generations
• Restored files are accessible and correct
• Restore completes within RTO

Troubleshooting

| Symptom | Possible Cause | Diagnostic Command | Corrective Action | Verification | |---------|---------------|-------------------|-------------------|--------------| | Backup fails | Insufficient disk space | `df -h /backup` | Free space; add storage | Backup succeeds | | rsync permission denied | Wrong permissions on source | `ls -l /source` | Fix permissions; run as root | rsync succeeds | | rsnapshot config error | Syntax error in config | `rsnapshot configtest` | Fix config syntax | Config validates | | tar archive corrupt | Disk error during backup | `tar -tzpf archive` | Re-create backup; check disk | Archive valid | | Cron job not running | Cron service not running | `sudo systemctl status cron` | Start cron service | Jobs execute | | Restore fails | Wrong path or permissions | `ls -l /restore` | Fix path; check permissions | Restore succeeds | | Backup storage full | Too many backups retained | `df -h /backup` | Reduce retention; add storage | Space available | | Network backup slow | Network bandwidth | `rsync --progress` | Limit bandwidth; schedule off-peak | Backup completes |

Best Practices

1. **Follow 3-2-1 rule**: 3 copies, 2 media, 1 off-site. 2. **Test restores regularly**: Verify backups are actually recoverable. 3. **Automate backups**: Use cron or scheduling tools. 4. **Monitor backup success**: Check logs; set up alerts for failures. 5. **Encrypt sensitive backups**: Protect data at rest. 6. **Document recovery procedures**: Written runbooks for common scenarios. 7. **Include configuration files**: Back up /etc, not just data. 8. **Test backup after changes**: Verify backups work after major changes. 9. **Version backup scripts**: Keep backup scripts in version control. 10. **Review retention regularly**: Adjust retention based on compliance and storage.

Conclusion

Linux backup and recovery requires a multi-layered approach combining file-level backups (tar, rsync), snapshot-based backups (rsnapshot), and enterprise solutions (Bacula). The 3-2-1 rule, regular restore testing, and documented procedures ensure data can be recovered within acceptable RTO and RPO targets. Automation through cron reduces human error and ensures consistent backup execution.

References

• rsync Documentation (https://rsync.samba.org/)
• rsnapshot Documentation (https://rsnapshot.org/)
• tar Documentation (https://www.gnu.org/software/tar/)
• Bacula Documentation (https://www.bacula.org/)
• 3-2-1 Backup Rule (https://www.veeam.com/)
• NIST SP 800-34 Rev. 1 - Contingency Planning Guide (https://csrc.nist.gov/)