Backup advanced ⏱ 60 minutes

Disaster Recovery: Planning and Testing for Homelabs

Complete DR strategy for your homelab. Planning, documentation, and testing procedures.

By homelabbing.io • 2025-12-17
#disaster-recovery#planning#testing#documentation#strategy

Disaster Recovery (DR) is the process of preparing for and recovering from events that disrupt your homelab services. This comprehensive guide covers planning, implementation, testing, and maintenance of a robust DR strategy.

Why Disaster Recovery Matters

Even in a homelab environment, disasters happen:

  • Hardware failures: Disk crashes, PSU failures, server death
  • Data corruption: Filesystem issues, ransomware, accidental deletion
  • Natural disasters: Fire, flood, power surge, lightning
  • Human error: Wrong command, misconfiguration, deleted VMs
  • Software issues: Failed updates, OS corruption, broken services
  • Security incidents: Compromised systems, ransomware attacks

Without DR planning:

  • Lost family photos and documents
  • Weeks rebuilding from scratch
  • No access to critical services
  • Lost configuration knowledge
  • Expensive data recovery services

With proper DR:

  • Quick recovery (hours, not weeks)
  • Minimal data loss
  • Clear recovery procedures
  • Tested and validated backups
  • Peace of mind

Part 1: Understanding DR Fundamentals

Key Concepts

Recovery Time Objective (RTO):

  • How long can services be down?
  • Homelab: Hours to days acceptable
  • Critical services: Minutes to hours

Recovery Point Objective (RPO):

  • How much data loss is acceptable?
  • Documents/photos: Zero loss
  • Metrics/logs: Minutes to hours

Backup vs Archive:

  • Backup: Short-term, quick recovery
  • Archive: Long-term, compliance, historical

3-2-1 Backup Rule:

  • 3 copies of data
  • 2 different media types
  • 1 copy offsite

3-2-1-1-0 Enhanced Rule:

  • 3 copies of data
  • 2 different media types
  • 1 copy offsite
  • 1 copy offline (air-gapped)
  • 0 errors in verification

DR Tiers

Tier 1 - Critical (RTO under 4 hours, RPO under 1 hour):

  • Home automation (security, cameras)
  • Network infrastructure (router, DNS, firewall)
  • Core services (authentication, storage)

Tier 2 - Important (RTO under 24 hours, RPO under 1 day):

  • Media services (Plex, Jellyfin)
  • Home Assistant, dashboards
  • Development environments
  • Docker services

Tier 3 - Nice to have (RTO 1-7 days, RPO 1-7 days):

  • Experimentation VMs
  • Test environments
  • Non-critical services

Part 2: Inventory and Documentation

System Inventory

Create comprehensive inventory:

Hardware inventory spreadsheet:

Server Name | Type      | CPU           | RAM  | Storage    | IP Address    | Purchase Date
-----------------------------------------------------------------------------------------------
pve-01      | Proxmox   | Xeon E-2288G  | 64GB | 2TB NVMe   | 192.168.1.10  | 2023-05-15
nas-01      | TrueNAS   | i3-12100      | 32GB | 4x4TB HDD  | 192.168.1.20  | 2023-06-20
docker-01   | Ubuntu    | Ryzen 5600G   | 32GB | 500GB SSD  | 192.168.1.30  | 2024-01-10

VM/Container inventory:

Name            | Type | Host   | OS          | Services           | Backup Priority
-----------------------------------------------------------------------------------------
nginx-proxy     | LXC  | pve-01 | Ubuntu 22   | Reverse proxy      | Tier 1
pihole          | LXC  | pve-01 | Ubuntu 22   | DNS, Ad blocking   | Tier 1
home-assistant  | VM   | pve-01 | HAOS        | Home automation    | Tier 1
plex            | LXC  | pve-01 | Ubuntu 22   | Media server       | Tier 2
dev-ubuntu      | VM   | pve-01 | Ubuntu 24   | Development        | Tier 3

Network diagram:

Internet
    |
  Router (192.168.1.1)
    |
  -------------------------
  |           |           |
Switch    WiFi AP    Firewall
  |           |           |
Servers    Devices    VLANs

Configuration Documentation

Document critical configurations:

Network configuration /root/dr-docs/network-config.md:

# Network Configuration

## VLANs
- VLAN 10: Management (192.168.10.0/24)
- VLAN 20: Servers (192.168.20.0/24)
- VLAN 30: IoT (192.168.30.0/24)
- VLAN 40: Guest (192.168.40.0/24)

## Static IPs
- Gateway: 192.168.1.1
- DNS Primary: 192.168.1.2 (Pi-hole)
- DNS Secondary: 1.1.1.1
- Proxmox: 192.168.1.10
- NAS: 192.168.1.20
- Docker: 192.168.1.30

## Firewall Rules
[Document critical firewall rules]

Service configuration /root/dr-docs/service-config.md:

# Service Configuration

## Docker Services
Location: /opt/docker-compose/
Backup: Daily to NAS

## Databases
- PostgreSQL: Port 5432, backed up with pg_dump
- MySQL: Port 3306, backed up with mysqldump

## SSL Certificates
Provider: Let's Encrypt via Certbot
Renewal: Automatic via systemd timer
Location: /etc/letsencrypt/

Password Management

Use password manager for credentials:

Critical passwords to document:

  • Root/admin passwords for all systems
  • IPMI/iLO/iDRAC credentials
  • Backup encryption passwords
  • API keys and tokens
  • SSL certificate passwords
  • Database passwords
  • Cloud service credentials

Export encrypted backup:

# Export from password manager
# Store in multiple secure locations:
# 1. USB drive in safe
# 2. Encrypted cloud storage
# 3. Trusted family member

Create DR Runbook

Master runbook /root/dr-docs/disaster-recovery-runbook.md:

# Disaster Recovery Runbook

## Emergency Contacts
- Name: John Doe
- Phone: +1-555-0123
- Email: [email protected]

## Quick Recovery Steps

### Scenario 1: Complete Datacenter Loss

1. Access offsite backup location
2. Restore network infrastructure first
3. Restore Tier 1 services
4. Verify functionality
5. Restore Tier 2 services
6. Document lessons learned

### Scenario 2: Single Server Failure

1. Identify failed server
2. Check backup status
3. Provision replacement hardware
4. Restore from backup
5. Verify and test

### Scenario 3: Data Corruption

1. Stop affected services
2. Identify last good backup
3. Restore from backup
4. Verify data integrity
5. Resume services

## Critical System Recovery

[Detailed steps for each critical system]

Part 3: Backup Strategy Implementation

Local Backups

Primary local backup - NAS:

# ZFS snapshots every hour
zfs snapshot tank/data@hourly-$(date +%Y%m%d-%H%M)

# Retention: 24 hourly, 7 daily, 4 weekly
# Automated with sanoid

# /etc/sanoid/sanoid.conf
[tank/data]
    use_template = production
    recursive = yes

[template_production]
    frequently = 0
    hourly = 24
    daily = 7
    weekly = 4
    monthly = 3
    yearly = 1
    autosnap = yes
    autoprune = yes

Proxmox VM backups to NAS:

# Vzdump to PBS/NAS
vzdump --all     --storage nas-backup     --mode snapshot     --compress zstd     --prune-backups keep-last=7,keep-weekly=4,keep-monthly=3

Docker volumes backup:

#!/bin/bash
# /usr/local/bin/docker-backup.sh

BACKUP_DIR="/mnt/nas/backups/docker"
DATE=$(date +%Y%m%d-%H%M)

# Stop services
docker-compose -f /opt/docker-compose/services.yml down

# Backup volumes
tar -czf "$BACKUP_DIR/volumes-$DATE.tar.gz"     /var/lib/docker/volumes/

# Backup compose files
tar -czf "$BACKUP_DIR/compose-$DATE.tar.gz"     /opt/docker-compose/

# Start services
docker-compose -f /opt/docker-compose/services.yml up -d

# Keep only last 14 days
find "$BACKUP_DIR" -name "*.tar.gz" -mtime +14 -delete

Offsite Backups

Cloud backup with Restic to Backblaze B2:

# /usr/local/bin/offsite-backup.sh
#!/bin/bash

export B2_ACCOUNT_ID="your-account-id"
export B2_ACCOUNT_KEY="your-application-key"
export RESTIC_REPOSITORY="b2:backup-bucket:homelab"
export RESTIC_PASSWORD="strong-encryption-password"

# Backup critical data
restic backup     /mnt/nas/documents     /mnt/nas/photos     /root/dr-docs     /etc     --exclude="*.tmp"     --tag offsite     --tag automated

# Prune old backups
restic forget     --keep-daily 7     --keep-weekly 4     --keep-monthly 12     --keep-yearly 3     --prune

# Check repository
restic check

Sync to remote location:

# Rsync to friend's homelab or remote server
rsync -avz --delete     /mnt/nas/backups/     backup-user@remote-server:/backups/homelab/

Offline/Air-gapped Backups

Monthly backup to external drives:

#!/bin/bash
# /usr/local/bin/offline-backup.sh

# Mount external drive
mount /dev/sdb1 /mnt/offline-backup

# Sync critical data
rsync -avh --delete     /mnt/nas/documents/     /mnt/offline-backup/documents/

rsync -avh --delete     /mnt/nas/photos/     /mnt/offline-backup/photos/

rsync -avh     /root/dr-docs/     /mnt/offline-backup/dr-docs/

# Create encrypted archive of configurations
tar -czf - /etc /opt/docker-compose /root/.ssh |     gpg --encrypt --recipient [email protected]     > /mnt/offline-backup/configs-$(date +%Y%m).tar.gz.gpg

# Unmount and disconnect
umount /mnt/offline-backup

echo "Offline backup complete. Disconnect drive and store securely."

Backup rotation schedule:

  • 2 external drives
  • Alternating monthly
  • One always disconnected and secure

Part 4: Configuration Management

Infrastructure as Code

Store configurations in Git:

# Initialize git repo for configs
cd /opt/homelab-configs
git init

# Add configurations
mkdir -p docker proxmox network scripts docs

# Docker compose files
cp -r /opt/docker-compose/* docker/

# Proxmox configs
cp /etc/pve/vzdump.cron proxmox/

# Network configs
cp /etc/network/interfaces network/

# Documentation
cp -r /root/dr-docs/* docs/

# Commit
git add .
git commit -m "Initial configuration backup"

# Push to GitHub (private repo)
git remote add origin [email protected]:yourusername/homelab-configs.git
git push -u origin main

# Also push to GitLab as backup
git remote add gitlab [email protected]:yourusername/homelab-configs.git
git push -u gitlab main

Automate config backups:

#!/bin/bash
# /usr/local/bin/config-backup.sh

cd /opt/homelab-configs

# Update configurations
rsync -av /opt/docker-compose/ docker/
rsync -av /root/dr-docs/ docs/
rsync -av /etc/network/ network/

# Commit changes
if [[ $(git status --porcelain) ]]; then
    git add .
    git commit -m "Auto backup $(date +%Y-%m-%d)"
    git push origin main
    git push gitlab main
fi

Daily automated push:

# Add to cron
0 2 * * * /usr/local/bin/config-backup.sh >> /var/log/config-backup.log 2>&1

Ansible for Automation

Use Ansible to rebuild systems:

# ansible/playbooks/rebuild-docker-host.yml
---
- name: Rebuild Docker Host
  hosts: docker-hosts
  become: yes

  tasks:
    - name: Update system
      apt:
        update_cache: yes
        upgrade: dist

    - name: Install Docker
      include_role:
        name: geerlingguy.docker

    - name: Create docker-compose directory
      file:
        path: /opt/docker-compose
        state: directory

    - name: Copy docker-compose files
      copy:
        src: "{{ item }}"
        dest: /opt/docker-compose/
      with_fileglob:
        - ../docker-compose/*.yml

    - name: Start services
      community.docker.docker_compose:
        project_src: /opt/docker-compose
        state: present

    - name: Restore volumes from backup
      command: |
        restic restore latest           --target /           --path /var/lib/docker/volumes

Part 5: Recovery Procedures

Network Infrastructure Recovery

Priority 1: Router/Gateway

# 1. Connect to router
# 2. Factory reset if needed
# 3. Restore configuration from backup

# For pfSense/OPNsense:
# Navigate to: Diagnostics → Backup & Restore
# Upload config-backup.xml
# Reboot

# Verify:
ping 8.8.8.8
ping google.com

Priority 2: DNS (Pi-hole)

# Deploy new Pi-hole container/VM
docker run -d     --name pihole     -p 53:53/tcp -p 53:53/udp     -p 80:80     -e TZ="America/New_York"     -e WEBPASSWORD="admin-password"     -v /opt/pihole/config:/etc/pihole     -v /opt/pihole/dnsmasq:/etc/dnsmasq.d     --restart=unless-stopped     pihole/pihole:latest

# Restore configuration
docker exec pihole pihole -a -t /etc/pihole/backup.tar.gz

# Update DNS on router to point to Pi-hole
# Verify: nslookup google.com 192.168.1.2

Priority 3: Reverse Proxy

# Restore Nginx Proxy Manager
docker-compose -f /opt/docker-compose/nginx-proxy.yml up -d

# Restore database
docker exec -i npm-db mysql -u root -pROOT_PASSWORD npm < /backups/npm-db.sql

# Verify SSL certificates
ls -la /opt/npm/letsencrypt/

# Test proxy access
curl -I https://service.yourdomain.com

Server Recovery

Proxmox VE rebuild:

# 1. Install Proxmox from ISO
# 2. Configure network (static IP)
# 3. Update system
apt update && apt dist-upgrade -y

# 4. Add PBS storage
pvesm add pbs pbs-backup     --server YOUR-PBS-IP     --datastore backup-store     --username pve-backup@pbs

# 5. Restore VMs
qmrestore pbs-backup:vm/100/latest 100
qmrestore pbs-backup:vm/101/latest 101

# 6. Start VMs
qm start 100
qm start 101

# 7. Verify functionality

Docker host rebuild:

# 1. Install Ubuntu Server
# 2. Configure network
# 3. Install Docker
curl -fsSL https://get.docker.com | sh

# 4. Restore Docker volumes
restic restore latest --target / --path /var/lib/docker/volumes

# 5. Restore docker-compose files
git clone https://github.com/yourusername/homelab-configs.git /opt/configs
cp -r /opt/configs/docker/* /opt/docker-compose/

# 6. Start all services
cd /opt/docker-compose
docker-compose up -d

# 7. Verify
docker ps
docker logs container-name

Database recovery:

# PostgreSQL
createdb myapp
psql myapp < /backups/myapp-latest.sql

# MySQL
mysql -u root -p
CREATE DATABASE wordpress;
USE wordpress;
SOURCE /backups/wordpress-latest.sql;

# Verify
psql myapp -c "SELECT COUNT(*) FROM users;"

Application-Specific Recovery

Home Assistant:

# 1. Deploy new Home Assistant
docker run -d     --name homeassistant     --restart=unless-stopped     -v /opt/homeassistant:/config     -v /etc/localtime:/etc/localtime:ro     --network=host     ghcr.io/home-assistant/home-assistant:stable

# 2. Stop container
docker stop homeassistant

# 3. Restore configuration
rm -rf /opt/homeassistant/*
tar -xzf /backups/homeassistant-latest.tar.gz -C /opt/homeassistant/

# 4. Start container
docker start homeassistant

# 5. Verify in web UI
# Check automations, integrations, devices

Plex Media Server:

# 1. Install Plex
# 2. Stop service
systemctl stop plexmediaserver

# 3. Restore database
rm -rf /var/lib/plexmediaserver/Library/Application Support/Plex Media Server/*
tar -xzf /backups/plex-latest.tar.gz -C /var/lib/plexmediaserver/Library/Application Support/Plex Media Server/

# 4. Fix permissions
chown -R plex:plex /var/lib/plexmediaserver

# 5. Start service
systemctl start plexmediaserver

# 6. Verify library accessible

Part 6: Testing Your DR Plan

Regular Testing Schedule

Monthly tests:

  • Single file restore from each backup
  • Verify backup completion and logs
  • Test email/notification alerts
  • Review and update documentation

Quarterly tests:

  • Full VM/container restore to test environment
  • Database restore and verification
  • Network equipment configuration restore
  • Update DR runbook

Annual tests:

  • Complete infrastructure rebuild simulation
  • Offsite backup restore test
  • Team disaster drill (if applicable)
  • Review and update entire DR plan

VM Restore Test

Automated test script:

#!/bin/bash
# /usr/local/bin/test-vm-restore.sh

LOGFILE="/var/log/dr-tests.log"
TEST_VM_ID="9999"
BACKUP_STORAGE="pbs-backup"

echo "$(date): Starting VM restore test" >> "$LOGFILE"

# Get latest backup
LATEST_BACKUP=$(pvesm list "$BACKUP_STORAGE" | grep "vm-100" | tail -1)

# Restore to test VM ID
qmrestore "$BACKUP_STORAGE:$LATEST_BACKUP" "$TEST_VM_ID"     --storage local-lvm     >> "$LOGFILE" 2>&1

if [ $? -eq 0 ]; then
    echo "$(date): Restore successful" >> "$LOGFILE"

    # Start VM
    qm start "$TEST_VM_ID"

    # Wait for boot
    sleep 60

    # Test SSH connectivity
    if ssh test-vm "uptime"; then
        echo "$(date): VM is accessible and functioning" >> "$LOGFILE"
        SUCCESS=true
    else
        echo "$(date): VM not accessible" >> "$LOGFILE"
        SUCCESS=false
    fi

    # Cleanup
    qm stop "$TEST_VM_ID"
    qm destroy "$TEST_VM_ID"
else
    echo "$(date): Restore FAILED" >> "$LOGFILE"
    SUCCESS=false
fi

# Send notification
if [ "$SUCCESS" = true ]; then
    echo "DR Test PASSED on $(date)" |         mail -s "✅ DR Test Success" [email protected]
else
    echo "DR Test FAILED on $(date). Check $LOGFILE" |         mail -s "❌ DR Test Failed" [email protected]
fi

Schedule monthly:

# Add to cron
0 3 15 * * /usr/local/bin/test-vm-restore.sh

Backup Verification

Automated verification script:

#!/bin/bash
# /usr/local/bin/verify-backups.sh

ERRORS=0

# Check local backups
if [ ! -f "/mnt/nas/backups/docker/volumes-$(date +%Y%m%d)*.tar.gz" ]; then
    echo "ERROR: Local Docker backup missing"
    ERRORS=$((ERRORS + 1))
fi

# Check offsite backups (Restic)
export RESTIC_REPOSITORY="b2:backup-bucket:homelab"
export RESTIC_PASSWORD_FILE="/root/.restic-password"

if ! restic snapshots --last > /dev/null 2>&1; then
    echo "ERROR: Cannot access offsite backups"
    ERRORS=$((ERRORS + 1))
else
    LAST_BACKUP=$(restic snapshots --last --json | jq -r '.[0].time')
    LAST_BACKUP_AGE=$(( ($(date +%s) - $(date -d "$LAST_BACKUP" +%s)) / 3600 ))

    if [ "$LAST_BACKUP_AGE" -gt 48 ]; then
        echo "ERROR: Offsite backup is $LAST_BACKUP_AGE hours old"
        ERRORS=$((ERRORS + 1))
    fi
fi

# Check Proxmox backups
PBS_BACKUPS=$(pvesm list pbs-backup | wc -l)
if [ "$PBS_BACKUPS" -lt 5 ]; then
    echo "ERROR: Insufficient Proxmox backups"
    ERRORS=$((ERRORS + 1))
fi

# Report results
if [ "$ERRORS" -eq 0 ]; then
    echo "✅ All backup verifications passed" |         mail -s "Backup Verification Success" [email protected]
else
    echo "❌ $ERRORS backup verification errors found" |         mail -s "ALERT: Backup Verification Failed" [email protected]
    exit 1
fi

Disaster Simulation

Complete DR drill:

# Disaster Simulation Checklist

## Preparation
- [ ] Schedule maintenance window
- [ ] Notify users of planned outage
- [ ] Backup current state (just in case)
- [ ] Prepare fresh hardware/VMs

## Simulation Steps
- [ ] Power off all production systems (or use test environment)
- [ ] Access DR documentation and runbook
- [ ] Time: Start recovery timer

## Recovery Process
- [ ] Restore network infrastructure (router, DNS)
- [ ] Restore authentication services
- [ ] Restore Tier 1 services
- [ ] Verify Tier 1 functionality
- [ ] Restore Tier 2 services
- [ ] Verify Tier 2 functionality
- [ ] Time: Record total recovery time

## Validation
- [ ] Test all critical applications
- [ ] Verify data integrity
- [ ] Check service connectivity
- [ ] Review logs for errors

## Documentation
- [ ] Record actual RTO vs planned
- [ ] Document issues encountered
- [ ] Update runbook with improvements
- [ ] Schedule follow-up improvements

## Cleanup
- [ ] Return to production or cleanup test environment
- [ ] Send summary report to stakeholders

Part 7: Monitoring and Alerting

Backup Monitoring Dashboard

Use Grafana + Prometheus:

# docker-compose.yml for monitoring
version: '3.8'

services:
  prometheus:
    image: prom/prometheus:latest
    volumes:
      - ./prometheus:/etc/prometheus
      - prometheus-data:/prometheus
    ports:
      - "9090:9090"

  grafana:
    image: grafana/grafana:latest
    volumes:
      - grafana-data:/var/lib/grafana
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin

  node-exporter:
    image: prom/node-exporter:latest
    ports:
      - "9100:9100"

volumes:
  prometheus-data:
  grafana-data:

Prometheus backup job metrics:

# prometheus/prometheus.yml
scrape_configs:
  - job_name: 'backup-scripts'
    static_configs:
      - targets: ['localhost:9100']
    metric_relabel_configs:
      - source_labels: [__name__]
        regex: 'backup_.*'
        action: keep

Export backup metrics:

#!/bin/bash
# /usr/local/bin/backup-metrics.sh

# Create metrics file
METRICS_FILE="/var/lib/node_exporter/textfile_collector/backups.prom"

# Local backup age
LOCAL_AGE=$(find /mnt/nas/backups -name "*.tar.gz" -mtime -1 | wc -l)
echo "backup_local_recent_count $LOCAL_AGE" > "$METRICS_FILE"

# Offsite backup age
export RESTIC_REPOSITORY="b2:backup-bucket:homelab"
export RESTIC_PASSWORD_FILE="/root/.restic-password"
OFFSITE_AGE=$(( ($(date +%s) - $(date -d "$(restic snapshots --last --json | jq -r '.[0].time')" +%s)) / 3600 ))
echo "backup_offsite_age_hours $OFFSITE_AGE" >> "$METRICS_FILE"

# Disk space
DISK_USAGE=$(df /mnt/nas | tail -1 | awk '{print $5}' | sed 's/%//')
echo "backup_disk_usage_percent $DISK_USAGE" >> "$METRICS_FILE"

Alerting Rules

Prometheus alerting:

# prometheus/alerts.yml
groups:
  - name: backup_alerts
    interval: 5m
    rules:
      - alert: BackupTooOld
        expr: backup_offsite_age_hours > 48
        for: 1h
        labels:
          severity: critical
        annotations:
          summary: "Offsite backup is too old"
          description: "Offsite backup is {{ $value }} hours old"

      - alert: BackupDiskFull
        expr: backup_disk_usage_percent > 85
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "Backup disk almost full"
          description: "Backup disk usage at {{ $value }}%"

      - alert: NoRecentLocalBackup
        expr: backup_local_recent_count < 1
        for: 2h
        labels:
          severity: warning
        annotations:
          summary: "No recent local backups"

Health Checks

Healthchecks.io integration:

# Add to all backup scripts
HEALTHCHECK_URL="https://hc-ping.com/YOUR-UUID"

# Start signal
curl -fsS --retry 3 "$HEALTHCHECK_URL/start"

# Your backup commands
backup_command

# Success/failure
if [ $? -eq 0 ]; then
    curl -fsS --retry 3 "$HEALTHCHECK_URL"
else
    curl -fsS --retry 3 "$HEALTHCHECK_URL/fail"
fi

Monitor multiple checks:

  • Local backups: Daily check
  • Offsite backups: Daily check
  • Configuration backups: Daily check
  • VM backups: Daily check
  • Database backups: Daily check
  • DR test: Monthly check

Part 8: Security Considerations

Backup Encryption

Encrypt all offsite backups:

# Restic (encrypted by default)
export RESTIC_PASSWORD="strong-random-password"

# BorgBackup (encrypted)
borg init --encryption=repokey-blake2

# Manual encryption with GPG
tar -czf - /data | gpg --encrypt --recipient [email protected] > backup.tar.gz.gpg

Key management:

  • Store encryption keys in password manager
  • Export paper backup of keys
  • Keep offline copy in safe
  • Document key locations in DR plan

Access Control

Limit backup access:

# Create dedicated backup user
useradd -r -s /bin/bash backupuser

# Restrict SSH for backups
# In .ssh/authorized_keys:
command="borg serve --restrict-to-path /backups",restrict ssh-ed25519 AAAA...

# Use API tokens instead of passwords
# Rotate credentials quarterly

Ransomware Protection

Implement append-only backups:

# BorgBackup append-only
borg init --append-only /backup/borg-repo

# Restic with REST server append-only
rest-server --append-only --path /backup/restic-repo

# Immutable cloud storage (S3 Object Lock)
aws s3api put-object-lock-configuration     --bucket backup-bucket     --object-lock-configuration     'ObjectLockEnabled=Enabled,Rule={DefaultRetention={Mode=COMPLIANCE,Days=30}}'

Air-gap critical backups:

  • Monthly offline backup to external drive
  • Disconnect and store securely
  • Rotate between two drives
  • Test quarterly

Network Security

Isolate backup infrastructure:

# Firewall rules for backup server
ufw allow from 192.168.1.0/24 to any port 8000  # Restic REST
ufw allow from 192.168.1.0/24 to any port 8007  # PBS
ufw deny from any to any  # Default deny

Use VPN for remote backups:

# Connect through Tailscale/Wireguard
# Never expose backup ports to internet
# Use SSH tunnels for remote access

Part 9: Advanced Topics

Multi-Site Replication

Replicate to second location:

# ZFS replication to remote site
zfs send tank/data@snapshot |     ssh remote-server zfs recv backup/data

# Scheduled replication
syncoid tank/data remote-server:backup/data

PBS sync to remote:

# Configure in PBS web interface
# Remote → Add → Configure remote PBS
# Sync Job → Create sync from local to remote
# Schedule: Daily at 04:00

Disaster Recovery as a Service (DRaaS)

Cloud-based DR options:

  1. AWS EC2 snapshots

    • AMI of critical servers
    • Automated snapshot lifecycle
    • Cross-region replication

  2. Azure Site Recovery

    • Replicate VMs to Azure
    • Automated failover
    • Test failover without impact

  3. Google Cloud backup

    • Cloud Storage for file backups
    • Persistent disk snapshots
    • Global distribution

Automation and Orchestration

Terraform for infrastructure:

# terraform/main.tf
resource "proxmox_vm_qemu" "critical_vm" {
  name        = "nginx-proxy"
  target_node = "pve-01"
  iso         = "local:iso/ubuntu-22.04.iso"

  cores   = 2
  memory  = 4096

  disk {
    size    = "20G"
    type    = "scsi"
    storage = "local-lvm"
  }

  network {
    model  = "virtio"
    bridge = "vmbr0"
  }
}

Ansible for configuration:

# ansible/rebuild-infrastructure.yml
---
- name: Rebuild Complete Infrastructure
  hosts: all

  roles:
    - common
    - docker
    - monitoring
    - backups

  tasks:
    - name: Restore configurations
      include_role:
        name: restore

Part 10: Continuous Improvement

Post-Incident Review

After any recovery event:

# Incident Review Template

## Incident Summary
- Date: YYYY-MM-DD
- Duration: X hours
- Impact: Services affected
- Root cause: What failed

## Timeline
- HH:MM - Issue detected
- HH:MM - Recovery started
- HH:MM - Services restored
- HH:MM - Full functionality

## What Worked Well
- Quick detection
- Clear documentation
- Successful restore

## What Didn't Work
- Slow network recovery
- Missing passwords
- Outdated runbook

## Action Items
- [ ] Update documentation
- [ ] Fix identified gaps
- [ ] Schedule additional testing
- [ ] Implement improvements

Metrics to Track

DR effectiveness metrics:

- Actual RTO vs planned RTO
- Actual RPO vs planned RPO
- Recovery success rate
- Time to first service restored
- Data loss (if any)
- Manual intervention required
- Documentation accuracy
- Total cost of incident

Regular Maintenance

Weekly tasks:

  • Review backup logs
  • Check disk space
  • Verify recent backups

Monthly tasks:

  • Test single file restore
  • Review and update documentation
  • Check for software updates
  • Verify offsite backups

Quarterly tasks:

  • Full VM restore test
  • Review retention policies
  • Update inventory
  • DR plan walkthrough
  • Security audit

Annual tasks:

  • Complete DR drill
  • Hardware refresh planning
  • Budget review
  • Training updates

Conclusion

A comprehensive disaster recovery plan is essential for homelab reliability. This guide provided:

  • Complete DR planning framework
  • Backup strategy implementation
  • Detailed recovery procedures
  • Testing methodologies
  • Monitoring and alerting
  • Security best practices
  • Continuous improvement processes

Final Checklist

  • Inventory all systems and services
  • Document all configurations
  • Implement 3-2-1-1-0 backup strategy
  • Create and store recovery runbooks
  • Test recovery procedures
  • Set up monitoring and alerting
  • Schedule regular testing
  • Review and update quarterly

Critical Reminders

  1. Backups without testing are just hopes

    • Test restores monthly
    • Full DR drill annually

  2. Documentation is your lifeline

    • Keep it updated
    • Store in multiple locations
    • Make it accessible offline

  3. Automate everything possible

    • Reduces human error
    • Ensures consistency
    • Saves time during crisis

  4. Security matters

    • Encrypt backups
    • Protect credentials
    • Air-gap critical data

  5. Plan for the worst

    • Assume total loss
    • Have offsite backups
    • Know your RTO/RPO

Additional Resources

Remember: Hope is not a strategy. Plan, implement, test, repeat.

The difference between a disaster and an inconvenience is preparation. Your future self will thank you for the work you put in today.