Virtualization advanced ⏱ 90 minutes

Proxmox 8 Advanced Features: GPU Passthrough, Clustering, and More

Master Proxmox 8's advanced features including PCIe passthrough, HA clustering, and cloud-init automation.

By homelabbing.io • 2025-12-16
#proxmox#gpu-passthrough#clustering#advanced

You’ve installed Proxmox VE, created your first VMs, and mastered the basics. Now you’re ready to unlock the platform’s true potential. GPU passthrough for machine learning workloads. High availability clustering that keeps critical services running. Cloud-init automation that deploys VMs in seconds. Ceph storage that scales infinitely.

These aren’t just enterprise features anymore—they’re accessible in your homelab.

The challenge? Advanced Proxmox features require deep technical understanding. Documentation assumes you know IOMMU groups, Ceph architecture, and clustering concepts. One wrong configuration can break your entire setup. That’s where this guide comes in.

What You’ll Master

By the end of this guide, you will:

  • Configure PCIe/GPU passthrough for AI workloads and gaming VMs
  • Build a high availability cluster with automatic VM failover
  • Deploy Ceph distributed storage across multiple nodes
  • Automate VM deployment with cloud-init templates
  • Implement advanced networking with VLANs and SDN
  • Design enterprise-grade backup strategies
  • Troubleshoot common advanced configuration issues

This is your roadmap from Proxmox user to Proxmox expert.

Prerequisites

Before diving into advanced features, ensure you have:

Required Knowledge:

  • Proxmox VE basics (installation, VM creation, networking)
  • Linux command line proficiency
  • Understanding of networking concepts (VLANs, subnetting, routing)
  • Basic virtualization concepts

Hardware Requirements:

For GPU Passthrough:

  • CPU with VT-d/AMD-Vi support
  • Motherboard supporting IOMMU
  • Dedicated GPU (separate from host graphics)
  • Sufficient PCIe lanes

For Clustering:

  • 3+ identical or similar servers (minimum for HA quorum)
  • Dedicated network interfaces (1Gbps minimum, 10Gbps recommended)
  • Shared storage or Ceph deployment
  • UPS for all nodes

Software:

  • Proxmox VE 8.0 or later
  • Updated BIOS/UEFI firmware
  • SSH access to all nodes

Section 1: PCIe and GPU Passthrough

Understanding IOMMU and Hardware Isolation

What is IOMMU?

IOMMU (Input-Output Memory Management Unit) provides hardware-level isolation that allows passing physical PCIe devices directly to virtual machines. Think of it as a security guard that ensures VMs can only access their assigned hardware.

Intel VT-d vs. AMD-Vi:

Intel VT-d:
- Available on most modern Intel processors (check ark.intel.com)
- Requires chipset support (Z/Q/X series motherboards)
- Enable in BIOS: VT-d, Virtualization Technology

AMD-Vi:
- Available on Ryzen and EPYC processors
- Better IOMMU grouping on consumer platforms
- Enable in BIOS: IOMMU, AMD-Vi, Virtualization

IOMMU Groups Explained:

IOMMU groups are collections of devices that must be passed through together. Poor grouping means you can’t isolate individual devices.

Example of good IOMMU grouping:

IOMMU Group 1:
  00:02.0 VGA compatible controller [8086:9bc5]

IOMMU Group 15:
  03:00.0 VGA compatible controller [10de:2204] (NVIDIA RTX 3070)
  03:00.1 Audio device [10de:1aef] (NVIDIA Audio)

Example of poor IOMMU grouping:

IOMMU Group 1:
  00:01.0 PCI bridge
  00:02.0 VGA controller
  01:00.0 NVIDIA GPU
  01:00.1 NVIDIA Audio
  02:00.0 Network controller
  03:00.0 SATA controller

Why Poor Grouping Matters: In the poor example, passing through the GPU requires passing through network and SATA controllers too—effectively breaking the host.

Enabling IOMMU in Proxmox

Step 1: BIOS Configuration

Access BIOS/UEFI settings:
1. Reboot server
2. Press DEL/F2/F10 (varies by manufacturer)

Enable Required Features:
Intel Systems:
  - Intel VT-x: Enabled
  - Intel VT-d: Enabled
  - SR-IOV: Enabled (if available)

AMD Systems:
  - AMD-V: Enabled
  - AMD-Vi / IOMMU: Enabled
  - NX Mode: Enabled
  - SVM Mode: Enabled

Additional Settings:
  - Secure Boot: Disabled (can interfere with passthrough)
  - CSM/Legacy Boot: Disabled (use UEFI only)
  - Above 4G Decoding: Enabled (for large GPU memory)
  - Resizable BAR: Enabled (improves GPU performance)

Save and exit BIOS

Step 2: Configure GRUB Bootloader

Edit GRUB configuration:

# SSH into Proxmox host
ssh root@proxmox-server

# Edit GRUB configuration
nano /etc/default/grub

# Find the GRUB_CMDLINE_LINUX_DEFAULT line
# Replace with:

# For Intel systems:
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction video=efifb:off"

# For AMD systems:
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt pcie_acs_override=downstream,multifunction video=efifb:off"

# Save file (Ctrl+O, Enter, Ctrl+X)

# Update GRUB
update-grub

# Reboot to apply changes
reboot

Parameter Explanations:

intel_iommu=on / amd_iommu=on: Enables IOMMU
iommu=pt: Passthrough mode (better performance)
pcie_acs_override=downstream,multifunction: Override ACS for better IOMMU grouping (use cautiously)
video=efifb:off: Prevents host from binding to framebuffer (helps with GPU passthrough)

Step 3: Load VFIO Modules

# Add VFIO modules to load at boot
nano /etc/modules

# Add these lines:
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

# Save and exit

Step 4: Blacklist GPU Drivers

Prevent host from using the GPU you want to pass through:

# Create blacklist configuration
nano /etc/modprobe.d/blacklist.conf

# For NVIDIA GPUs, add:
blacklist nouveau
blacklist nvidia
blacklist nvidiafb
blacklist nvidia_drm

# For AMD GPUs, add:
blacklist amdgpu
blacklist radeon

# Save and exit

# Update initramfs
update-initramfs -u -k all

# Reboot
reboot

Step 5: Verify IOMMU Groups

# Check IOMMU is enabled
dmesg | grep -e DMAR -e IOMMU

# Expected output (Intel):
# DMAR: IOMMU enabled

# List IOMMU groups
#!/bin/bash
shopt -s nullglob
for g in $(find /sys/kernel/iommu_groups/* -maxdepth 0 -type d | sort -V); do
    echo "IOMMU Group ${g##*/}:"
    for d in $g/devices/*; do
        echo -e "\t$(lspci -nns ${d##*/})"
    done;
done;

# Save as check-iommu.sh and run:
chmod +x check-iommu.sh
./check-iommu.sh

Configuring GPU Passthrough for a VM

Step 1: Identify GPU PCI IDs

# List PCI devices
lspci -nn | grep -i nvidia

# Example output:
# 03:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA104 [GeForce RTX 3070] [10de:2204]
# 03:00.1 Audio device [0403]: NVIDIA Corporation GA104 High Definition Audio Controller [10de:1aef]

# Note the device IDs: 10de:2204 and 10de:1aef

Step 2: Bind GPU to VFIO

# Configure VFIO to claim the GPU at boot
nano /etc/modprobe.d/vfio.conf

# Add (replace with your GPU IDs):
options vfio-pci ids=10de:2204,10de:1aef disable_vga=1

# Save and update initramfs
update-initramfs -u -k all
reboot

Step 3: Create VM with GPU Passthrough

Via Web UI:

1. Create new VM:
   - General: VM ID 100, Name: "Gaming-VM"
   - OS: Select Windows 11 ISO (or Linux)
   - System:
     - BIOS: OVMF (UEFI)
     - Add EFI Disk: Yes
     - Machine: q35 (required for PCIe passthrough)
   - Disks: 128GB+ SSD
   - CPU:
     - Type: host
     - Cores: 8-16
   - Memory: 16384-32768 MB
   - Network: VirtIO

2. Do NOT start VM yet

Via Command Line (Alternative):

# Add GPU to VM
qm set 100 -hostpci0 03:00,pcie=1,x-vga=1

# Explanation:
# 03:00 = PCI address (from lspci)
# pcie=1 = Express passthrough
# x-vga=1 = Primary GPU mode (for display output)

# For secondary GPU (non-primary):
qm set 100 -hostpci0 03:00,pcie=1

# If GPU has multiple functions (GPU + Audio):
qm set 100 -hostpci0 03:00,pcie=1,x-vga=1,multifunction=on
qm set 100 -hostpci1 03:00.1,pcie=1

Step 4: Configure VM for GPU

# Edit VM configuration
nano /etc/pve/qemu-server/100.conf

# Ensure these lines exist:
machine: q35
bios: ovmf
cpu: host,hidden=1,flags=+pcid
args: -cpu host,kvm=off,hv_vendor_id=proxmox
hostpci0: 03:00,pcie=1,x-vga=1

# Save and exit

Step 5: Install GPU Drivers in VM

1. Start VM and install OS (Windows 11)
2. Download NVIDIA/AMD drivers from official site
3. Install drivers in VM
4. Reboot VM
5. Verify GPU detected: Device Manager (Windows) or nvidia-smi (Linux)

Troubleshooting GPU Passthrough

Issue 1: VM won’t start - “device or resource busy”

# Check if host is using GPU
lspci -k | grep -A 3 -i "vga|3d|2d"

# If GPU shows kernel driver, blacklist it properly
# Verify vfio-pci claimed device:
lspci -nnk -d 10de:2204

# Should show:
# Kernel driver in use: vfio-pci

Issue 2: VM starts but GPU not detected

# Verify IOMMU enabled in VM config
qm showcmd 100 | grep iommu

# Add CPU flags:
nano /etc/pve/qemu-server/100.conf
cpu: host,hidden=1,flags=+pcid

# Try alternative machine type:
machine: pc-q35-8.0

Issue 3: “Code 43” Error in Windows

# Edit VM config
nano /etc/pve/qemu-server/100.conf

# Add these lines:
args: -cpu host,kvm=off,hv_vendor_id=proxmox
cpu: host,hidden=1

# This hides virtualization from NVIDIA drivers

Issue 4: Poor GPU Performance

# Enable CPU pinning for better performance
qm set 100 -vcpus 8
qm set 100 -cpuunits 10000

# Enable huge pages
echo "vm.nr_hugepages = 8192" >> /etc/sysctl.conf
sysctl -p

# Update VM config
nano /etc/pve/qemu-server/100.conf
hugepages: 1024

Use Cases for GPU Passthrough

1. AI/Machine Learning Workloads

VM Configuration:
- OS: Ubuntu 22.04 LTS
- GPU: NVIDIA RTX 4090 (24GB VRAM)
- CPU: 16 cores
- RAM: 64 GB
- Storage: 500GB NVMe for datasets

Applications:
- PyTorch / TensorFlow training
- Stable Diffusion image generation
- LLM inference (Ollama, LocalAI)
- CUDA development

Performance: Near-native (98-99% of bare metal)

2. Gaming VM (Windows 11)

VM Configuration:
- OS: Windows 11 Pro
- GPU: NVIDIA RTX 3070
- CPU: 8 cores (pinned)
- RAM: 16 GB
- Storage: 256GB NVMe

Features:
- Looking Glass for low-latency display
- USB passthrough for keyboard/mouse
- Audio passthrough via HDMI
- Network bridge for gaming

Performance: Console-equivalent gaming

3. Video Transcoding (Plex/Jellyfin)

VM Configuration:
- OS: Ubuntu 22.04 LTS
- GPU: NVIDIA GTX 1650 (NVENC)
- CPU: 4 cores
- RAM: 8 GB
- Storage: NFS mount to media library

Applications:
- Plex Media Server with hardware transcoding
- Jellyfin with NVENC acceleration
- FFmpeg batch processing

Performance: 10-15 concurrent 4K transcodes

Section 2: High Availability Clustering

Understanding Proxmox Clustering

What is a Proxmox Cluster?

A cluster is a group of Proxmox nodes that work together as a unified system. Benefits include:

  • Centralized management of all nodes
  • Live migration of VMs between hosts
  • High availability (automatic VM restart on failure)
  • Shared configuration across nodes
  • Distributed storage (Ceph)

Quorum and Split-Brain Prevention:

Quorum Requirement: (n/2) + 1 nodes must agree

3-node cluster: 2 nodes required for quorum
5-node cluster: 3 nodes required for quorum
7-node cluster: 4 nodes required for quorum

Why odd numbers? Prevents split-brain scenarios where network partition creates two equal groups.

Cluster Architecture Example:

Homelab 3-Node Cluster:

Node 1 (pve1):
- IP: 192.168.1.101
- CPU: AMD Ryzen 5 5600X
- RAM: 64 GB
- Storage: 1TB NVMe + 4TB HDD

Node 2 (pve2):
- IP: 192.168.1.102
- CPU: AMD Ryzen 5 5600X
- RAM: 64 GB
- Storage: 1TB NVMe + 4TB HDD

Node 3 (pve3):
- IP: 192.168.1.103
- CPU: Intel i5-10400
- RAM: 32 GB
- Storage: 512GB NVMe + 2TB HDD

Cluster Network: Dedicated 10Gbps connections
Shared Storage: Ceph distributed across all nodes

Creating a Proxmox Cluster

Prerequisites:

✓ All nodes on same version of Proxmox
✓ Unique hostnames for each node
✓ NTP synchronized (critical for quorum)
✓ SSH connectivity between all nodes
✓ Dedicated cluster network (recommended)
✓ DNS resolution or /etc/hosts entries

Step 1: Verify Time Synchronization

# On all nodes, ensure NTP is running
systemctl status chronyd

# Check time on all nodes
date

# If time differs by more than a few seconds, sync:
systemctl restart chronyd
hwclock --systohc

Step 2: Configure /etc/hosts

# On ALL nodes, edit /etc/hosts
nano /etc/hosts

# Add entries for all cluster nodes:
192.168.1.101 pve1.homelab.local pve1
192.168.1.102 pve2.homelab.local pve2
192.168.1.103 pve3.homelab.local pve3

# Save on all nodes

Step 3: Create Cluster (on first node only)

# SSH to first node (pve1)
ssh [email protected]

# Create cluster
pvecm create homelab-cluster --link0 192.168.1.101

# Verify cluster created
pvecm status

# Expected output:
# Cluster information
# Name:             homelab-cluster
# Config Version:   1
# Transport:        knet
# Secure auth:      on
# Quorum information
# Date:             Mon Dec 16 10:00:00 2025
# Quorum provider:  corosync_votequorum
# Nodes:            1
# Node ID:          0x00000001
# Ring ID:          1.5
# Quorate:          Yes

Step 4: Join Additional Nodes

# On second node (pve2)
ssh [email protected]

# Join cluster (use IP of first node)
pvecm add 192.168.1.101

# Enter password for root@pve1 when prompted
# Wait for join to complete (30-60 seconds)

# Verify joined
pvecm status

# Repeat on third node (pve3)
ssh [email protected]
pvecm add 192.168.1.101

Step 5: Verify Cluster Health

# On any node, check cluster status
pvecm status

# Output should show all 3 nodes:
# Nodes:            3
# Quorate:          Yes

# Check node list
pvecm nodes

# Expected output:
# Node  Sts   Inc   Joined               Name
#    1   M      1   2025-12-16 10:00:00  pve1
#    2   M      2   2025-12-16 10:05:00  pve2
#    3   M      3   2025-12-16 10:10:00  pve3

# Verify corosync configuration
pvecm expected 3

# Test cluster communication
pvecm updatecerts --force

Configuring High Availability

Step 1: Create HA Groups

HA groups define which nodes should run which VMs:

Via Web UI (Datacenter → HA → Groups):

Group Name: preferred-pve1
Comment: VMs preferring pve1
Nodes: pve1 (priority 100), pve2 (priority 50), pve3 (priority 25)

Group Name: database-servers
Comment: Database VMs on high-memory nodes
Nodes: pve1 (priority 100), pve2 (priority 100)

Group Name: storage-heavy
Comment: VMs needing high I/O
Nodes: pve1 (priority 100), pve3 (priority 50)

Step 2: Configure HA Resources

Via Web UI (Datacenter → HA → Add):

Resource: VM 100 (Production-DB)
Group: database-servers
State: started
Max Restart: 3
Max Relocate: 2

Behavior:
- VM runs on highest priority node (pve1 or pve2)
- If node fails, VM automatically migrates
- Max 3 restart attempts before giving up
- Max 2 relocations per hour

Resource: VM 101 (Web-Server)
Group: preferred-pve1
State: started
Max Restart: 5
Max Relocate: 3

Step 3: Test HA Failover

# On cluster, watch HA status
watch -n 2 'ha-manager status'

# On another terminal, simulate node failure
# SSH to pve1 and force shutdown
ssh root@pve1
poweroff

# Observe HA manager:
# 1. Detects node offline (30-60 seconds)
# 2. Begins VM migration to pve2
# 3. Starts VM on new node (2-5 minutes)

# Once pve1 comes back online:
# VMs remain on pve2 until manually migrated back
# Or configure automatic migration back (not recommended)

Cluster Networking Best Practices

Separate Network Paths:

Production Network (1Gbps):
- VLAN 10: 192.168.10.0/24
- VM traffic
- User access to services

Cluster Network (10Gbps):
- VLAN 20: 10.0.20.0/24
- Corosync cluster communication
- Live migration
- Ceph storage traffic

Management Network (1Gbps):
- VLAN 30: 192.168.30.0/24
- SSH access
- Web UI access
- Backup traffic

Configuring Cluster Network:

# Edit cluster network configuration
pvecm updatecerts
pvecm apiver

# Configure dedicated cluster interface
# On all nodes, edit /etc/network/interfaces
nano /etc/network/interfaces

# Add dedicated cluster interface:
auto vmbr1
iface vmbr1 inet static
    address 10.0.20.101/24  # .102 on pve2, .103 on pve3
    bridge-ports enp2s0
    bridge-stp off
    bridge-fd 0
    # Cluster network - high priority

# Apply network changes
ifreload -a

# Update corosync to use new network
pvecm updatecerts
systemctl restart pve-cluster
systemctl restart corosync

Section 3: Cloud-Init and VM Automation

Understanding Cloud-Init

Cloud-init is an industry-standard method for cloud instance initialization. In Proxmox, it enables:

  • Template-based VM deployment
  • Automated OS configuration
  • Network setup without manual intervention
  • SSH key injection
  • Custom scripts on first boot

Cloud-Init Workflow:

1. Create cloud-init enabled template VM
2. Clone template → new VM created in seconds
3. Cloud-init runs on first boot:
   - Sets hostname
   - Configures networking
   - Creates users
   - Installs SSH keys
   - Runs custom scripts
4. VM ready for use in 60-120 seconds

Creating a Cloud-Init Template

Step 1: Download Cloud Image

# SSH to Proxmox host
ssh root@pve1

# Download Ubuntu cloud image
cd /var/lib/vz/template/iso
wget https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64.img

# Alternative: Debian
wget https://cloud.debian.org/images/cloud/bookworm/latest/debian-12-generic-amd64.qcow2

# Alternative: Rocky Linux
wget https://download.rockylinux.org/pub/rocky/9/images/x86_64/Rocky-9-GenericCloud.latest.x86_64.qcow2

Step 2: Create Template VM

# Create VM (ID 9000 commonly used for templates)
qm create 9000 --name ubuntu-2204-template --memory 2048 --cores 2 --net0 virtio,bridge=vmbr0

# Import downloaded disk
qm importdisk 9000 jammy-server-cloudimg-amd64.img local-lvm

# Attach imported disk
qm set 9000 --scsihw virtio-scsi-pci --scsi0 local-lvm:vm-9000-disk-0

# Add cloud-init drive
qm set 9000 --ide2 local-lvm:cloudinit

# Configure boot order
qm set 9000 --boot c --bootdisk scsi0

# Add serial console (for cloud-init output)
qm set 9000 --serial0 socket --vga serial0

# Enable QEMU guest agent
qm set 9000 --agent enabled=1

# Resize disk (cloud images are small by default)
qm resize 9000 scsi0 +30G

# Convert to template
qm template 9000

Step 3: Configure Cloud-Init Defaults

Via Web UI:

Select VM 9000 → Cloud-Init tab:

User: admin
Password: (leave empty, use SSH keys)
SSH public key: [paste your public key]
IP Config (net0): DHCP (or static)
DNS domain: homelab.local
DNS servers: 192.168.1.1

Via CLI:

# Set cloud-init parameters
qm set 9000 --ciuser admin
qm set 9000 --sshkeys ~/.ssh/authorized_keys
qm set 9000 --ipconfig0 ip=dhcp
qm set 9000 --nameserver 192.168.1.1
qm set 9000 --searchdomain homelab.local

Deploying VMs from Template

Method 1: Web UI Clone

1. Right-click template VM 9000
2. Clone
3. Mode: Full clone
4. Target node: pve1
5. VM ID: 110
6. Name: web-server-01
7. Clone

8. Modify cloud-init settings (optional):
   VM 110 → Cloud-Init:
   - Change IP to static: 192.168.1.110/24
   - Set hostname: web-server-01

9. Start VM

10. Wait 60-120 seconds for cloud-init

11. SSH to VM:
    ssh [email protected]

Method 2: CLI Clone and Customize

# Clone template
qm clone 9000 111 --name app-server-01 --full

# Customize cloud-init
qm set 111 --ipconfig0 ip=192.168.1.111/24,gw=192.168.1.1
qm set 111 --nameserver 192.168.1.1
qm set 111 --searchdomain homelab.local
qm set 111 --ciuser admin
qm set 111 --sshkeys ~/.ssh/id_rsa.pub

# Start VM
qm start 111

# Wait for cloud-init (monitor with)
qm guest exec 111 -- cloud-init status --wait

Method 3: Terraform Automation

# main.tf - Proxmox Terraform configuration

terraform {
  required_providers {
    proxmox = {
      source = "telmate/proxmox"
      version = "2.9.14"
    }
  }
}

provider "proxmox" {
  pm_api_url = "https://pve1.homelab.local:8006/api2/json"
  pm_api_token_id = "terraform@pve!mytoken"
  pm_api_token_secret = "your-token-secret-here"
  pm_tls_insecure = true
}

resource "proxmox_vm_qemu" "web_servers" {
  count = 3  # Deploy 3 web servers

  name = "web-${count.index + 1}"
  target_node = "pve1"
  clone = "ubuntu-2204-template"
  full_clone = true

  cores = 2
  sockets = 1
  memory = 4096

  disk {
    size = "50G"
    type = "scsi"
    storage = "local-lvm"
  }

  network {
    model = "virtio"
    bridge = "vmbr0"
  }

  ipconfig0 = "ip=192.168.1.${120 + count.index}/24,gw=192.168.1.1"

  ciuser = "admin"
  sshkeys = file("~/.ssh/id_rsa.pub")

  lifecycle {
    ignore_changes = [
      network,
    ]
  }
}

output "vm_ips" {
  value = proxmox_vm_qemu.web_servers[*].default_ipv4_address
}

Deploy with Terraform:

# Initialize Terraform
terraform init

# Plan deployment
terraform plan

# Deploy VMs
terraform apply

# Output:
# vm_ips = [
#   "192.168.1.120",
#   "192.168.1.121",
#   "192.168.1.122",
# ]

# Destroy when done
terraform destroy

Advanced Cloud-Init Customization

Custom Cloud-Init User Data:

# Create custom user-data script
# Save as /var/lib/vz/snippets/web-server-init.yaml

#cloud-config
hostname: web-server
fqdn: web-server.homelab.local
manage_etc_hosts: true

users:
  - name: admin
    groups: sudo
    shell: /bin/bash
    sudo: ['ALL=(ALL) NOPASSWD:ALL']
    ssh_authorized_keys:
      - ssh-rsa AAAAB3NzaC1yc2E... user@laptop

packages:
  - nginx
  - docker.io
  - git
  - htop
  - vim

runcmd:
  - systemctl enable nginx
  - systemctl start nginx
  - echo "Hello from cloud-init" > /var/www/html/index.html
  - usermod -aG docker admin
  - systemctl enable docker
  - systemctl start docker

write_files:
  - path: /etc/nginx/sites-available/default
    content: |
      server {
        listen 80 default_server;
        root /var/www/html;
        index index.html;
        server_name _;
        location / {
          try_files $uri $uri/ =404;
        }
      }

  - path: /home/admin/.bashrc
    append: true
    content: |
      export PS1='\u@\h:\w\$ '
      alias ll='ls -alh'

power_state:
  mode: reboot
  condition: True

Apply custom cloud-init:

# Create VM with custom user-data
qm clone 9000 115 --name custom-web-server

# Set custom cloud-init config
qm set 115 --cicustom "user=local:snippets/web-server-init.yaml"

# Start VM
qm start 115

# VM will:
# 1. Set hostname
# 2. Create admin user
# 3. Install nginx, docker, git
# 4. Configure nginx
# 5. Start services
# 6. Reboot

Section 4: Ceph Distributed Storage

Understanding Ceph on Proxmox

What is Ceph?

Ceph is a distributed storage system that provides:

  • Object storage (S3-compatible)
  • Block storage (RBD for VMs)
  • File system storage (CephFS)
  • Self-healing and replication
  • Horizontal scalability

Ceph Architecture:

Components:

Monitors (MON):
- Maintain cluster state
- Track OSD health
- Quorum required (3 minimum recommended)

OSDs (Object Storage Daemons):
- Manage physical disks
- Store data objects
- Handle replication
- Perform recovery

Manager (MGR):
- Monitoring and metrics
- Web dashboard
- REST API

Metadata Servers (MDS):
- Required for CephFS
- Not needed for RBD (VM storage)

Minimum Requirements for Ceph:

3 Nodes: For redundancy and quorum
10Gbps Network: Essential for performance
SSD/NVMe: Recommended for OSDs (HDDs usable but slower)
Dedicated NICs: Separate cluster and public networks
RAM: 2-4GB per OSD daemon

Recommended Homelab Setup:
3 nodes × 4 OSDs = 12 total OSDs
10Gbps cluster network
1Gbps public network
Replication factor: 3

Installing Ceph on Proxmox Cluster

Prerequisites:

✓ Proxmox cluster already configured (3+ nodes)
✓ Dedicated storage drives (separate from OS)
✓ 10Gbps network between nodes (minimum 1Gbps)
✓ No existing data on storage drives
✓ Identical or similar disk configurations

Step 1: Install Ceph Packages

# On ALL cluster nodes, install Ceph
pveceph install --version quincy

# This installs:
# - ceph-mon (monitors)
# - ceph-osd (storage daemons)
# - ceph-mgr (management)
# - ceph-mds (metadata, if using CephFS)

Step 2: Initialize Ceph (First Node Only)

# On pve1
pveceph init --network 10.0.20.0/24

# Network explanation:
# 10.0.20.0/24 = Ceph cluster network (dedicated)
# This network must be accessible from all nodes

Step 3: Create Monitors

# Create monitor on pve1
pveceph mon create

# On pve2
ssh root@pve2
pveceph mon create

# On pve3
ssh root@pve3
pveceph mon create

# Verify monitors
ceph -s

# Expected output:
#   cluster:
#     id:     abc123...
#     health: HEALTH_WARN (no OSDs yet)
#   services:
#     mon: 3 daemons, quorum pve1,pve2,pve3
#     mgr: pve1(active), standbys: pve2, pve3

Step 4: Create OSDs

# List available disks on each node
pveceph lspci

# On pve1, create OSDs (repeat for each disk)
# Assuming /dev/sdb, /dev/sdc, /dev/sdd, /dev/sde
pveceph osd create /dev/sdb
pveceph osd create /dev/sdc
pveceph osd create /dev/sdd
pveceph osd create /dev/sde

# Repeat on pve2 and pve3 for their disks

# Verify OSDs created
ceph osd tree

# Expected output:
# ID  CLASS  WEIGHT   TYPE NAME       STATUS
# -1         12.0000  root default
# -3          4.0000      host pve1
#  0    hdd   1.0000          osd.0       up
#  1    hdd   1.0000          osd.1       up
#  2    hdd   1.0000          osd.2       up
#  3    hdd   1.0000          osd.3       up
# -5          4.0000      host pve2
#  4    hdd   1.0000          osd.4       up
#  5    hdd   1.0000          osd.5       up
#  6    hdd   1.0000          osd.6       up
#  7    hdd   1.0000          osd.7       up
# -7          4.0000      host pve3
#  8    hdd   1.0000          osd.8       up
#  9    hdd   1.0000          osd.9       up
# 10    hdd   1.0000          osd.10      up
# 11    hdd   1.0000          osd.11      up

Step 5: Create Ceph Pools

# Create pool for VM storage
pveceph pool create vm-storage --add_storages

# Parameters (auto-calculated or manual):
# Replicas: 3 (data stored on 3 nodes)
# PG (Placement Groups): Auto-calculated
# Crush Rule: Default (distributes across hosts)

# Create pool for backups
pveceph pool create backups --add_storages

# Verify pools
ceph df

# Expected output:
# GLOBAL:
#     SIZE       AVAIL      RAW USED     %RAW USED
#     12 TiB     11 TiB       500 GiB          4.00
#
# POOLS:
#     NAME         ID     USED       %USED     MAX AVAIL
#     vm-storage    1     100 GiB     2.50       3.5 TiB
#     backups       2      50 GiB     1.25       3.5 TiB

Step 6: Configure Proxmox Storage

Via Web UI:

Datacenter → Storage → Add → RBD

ID: ceph-vm-storage
Pool: vm-storage
Monitor Hosts: 10.0.20.101,10.0.20.102,10.0.20.103
Username: admin
Content: Disk image, Container

Enable

Repeat for backups pool:
ID: ceph-backups
Pool: backups
Content: VZDump backup file

Using Ceph for VM Storage

Create VM on Ceph:

Create VM → Disks:
Storage: ceph-vm-storage
Disk size: 50 GB
Cache: writeback (recommended)
Discard: enabled (for thin provisioning)
SSD emulation: yes (if using SSDs)

VM disk automatically distributed across Ceph cluster
Data replicated 3x for redundancy

Migrate Existing VM to Ceph:

# Move disk from local storage to Ceph
qm move_disk 100 scsi0 ceph-vm-storage --delete

# This:
# 1. Copies disk to Ceph
# 2. Updates VM configuration
# 3. Deletes old disk (if --delete specified)
# 4. VM can now migrate between nodes

Ceph Monitoring and Maintenance

Health Checks:

# Overall cluster status
ceph -s

# Detailed health
ceph health detail

# OSD status
ceph osd status

# Pool statistics
ceph df

# Monitor performance
ceph osd perf

Web Dashboard:

Access Ceph Dashboard:
https://pve1.homelab.local:8443

Login with Ceph credentials

Dashboard shows:
- Cluster health
- OSD status
- Pool utilization
- Performance graphs
- Active I/O operations

Common Maintenance Tasks:

# Scrub OSDs (verify data integrity)
ceph pg scrub 1.0  # Scrub specific placement group
ceph osd deep-scrub all  # Deep scrub all OSDs

# Rebalance cluster after adding OSDs
ceph osd reweight-by-utilization

# Check for inconsistencies
ceph pg dump | grep inconsistent

# Repair inconsistent PGs
ceph pg repair 1.a

Section 5: Advanced Networking

VLANs and Network Segmentation

VLAN Benefits:

  • Network isolation and security
  • Traffic prioritization (QoS)
  • Simplified management
  • Broadcast domain separation

Example VLAN Design:

VLAN 10: Management (Proxmox web UI, SSH)
  Subnet: 192.168.10.0/24
  Gateway: 192.168.10.1

VLAN 20: Production VMs
  Subnet: 192.168.20.0/24
  Gateway: 192.168.20.1

VLAN 30: Guest WiFi/DMZ
  Subnet: 192.168.30.0/24
  Gateway: 192.168.30.1
  Restricted access

VLAN 40: Storage/Backup
  Subnet: 192.168.40.0/24
  No gateway (isolated)

VLAN 50: IoT Devices
  Subnet: 192.168.50.0/24
  Gateway: 192.168.50.1
  No access to other VLANs

Configuring VLANs in Proxmox:

# Edit network configuration
nano /etc/network/interfaces

# Existing bridge
auto vmbr0
iface vmbr0 inet static
    address 192.168.10.101/24
    gateway 192.168.10.1
    bridge-ports enp1s0
    bridge-stp off
    bridge-fd 0
    bridge-vlan-aware yes
    bridge-vids 10 20 30 40 50

# VLAN sub-interfaces (optional, for host access)
auto vmbr0.20
iface vmbr0.20 inet static
    address 192.168.20.101/24
    vlan-raw-device vmbr0

auto vmbr0.40
iface vmbr0.40 inet static
    address 192.168.40.101/24
    vlan-raw-device vmbr0

# Apply configuration
ifreload -a

Assign VLAN to VM:

Via Web UI:
VM → Hardware → Network Device:
Bridge: vmbr0
VLAN Tag: 20

Or via CLI:
qm set 100 --net0 virtio,bridge=vmbr0,tag=20

Software-Defined Networking (SDN)

Proxmox SDN Features:

  • Virtual networks (VNets)
  • Zones (separate broadcast domains)
  • VXLANs for multi-site
  • Subnets and DHCP integration
  • Firewall integration

Creating an SDN Zone:

Via Web UI:
Datacenter → SDN → Zones → Create:

Type: Simple (for basic VLANs)
ID: production-zone
Bridge: vmbr0
VLAN Aware: Yes

Creating VNets:

Datacenter → SDN → VNets → Create:

Zone: production-zone
VNet ID: vnet-web
Tag: 20

Assign Subnet to VNet:

Datacenter → SDN → Subnets → Create:

VNet: vnet-web
Subnet: 192.168.20.0/24
Gateway: 192.168.20.1
SNAT: Enabled (for internet access)
DHCP Range: 192.168.20.100-192.168.20.200

Apply SDN Configuration:

# Apply SDN changes
pvesdn reload

# Verify
pvesdn status

Conclusion

You’ve now mastered Proxmox’s advanced features. From GPU passthrough enabling AI workloads to high availability clusters ensuring uptime, these capabilities transform Proxmox from a simple hypervisor into an enterprise-grade platform.

Key Takeaways:

  • GPU passthrough delivers near-native performance for ML, gaming, and transcoding
  • Clustering provides centralized management and automatic failover
  • Cloud-init enables infrastructure-as-code and rapid VM deployment
  • Ceph offers scalable, self-healing distributed storage
  • Advanced networking isolates and secures your infrastructure

Next Steps:

  • Experiment in a test environment before production
  • Document your configurations thoroughly
  • Join the Proxmox community forums
  • Explore automation with Terraform and Ansible
  • Consider professional Proxmox training for deeper expertise

Resources:

Welcome to advanced Proxmox. Your homelab is now enterprise-ready.