Hardware Transcoding: Intel QSV, NVIDIA NVENC, and AMD VCE

Introduction: Why Hardware Transcoding Matters

Imagine streaming your 4K HDR movie collection to your phone over mobile data. Your media server needs to transcode that 80 Mbps, 50GB file down to 3 Mbps that fits your bandwidth. Using CPU software transcoding, a modern 8-core processor might handle 1-2 simultaneous 4K transcodes while consuming 150W and running at 90% utilization.

Now imagine the same scenario with hardware transcoding: Your integrated Intel GPU handles 6-8 simultaneous 4K transcodes while consuming 35W total system power and leaving your CPU at 15% utilization. The quality is identical. The cost difference? Zero additional hardware for Intel Quick Sync, or a $150 used GPU for NVIDIA.

Hardware transcoding isn’t just an optimization—it’s the difference between a struggling server and a effortless streaming experience. In 2025, every media server should leverage GPU acceleration. This guide shows you exactly how to implement it.

By the end of this guide, you’ll understand:

• The three major hardware transcoding technologies: Intel QSV, NVIDIA NVENC, AMD VCE
• How to choose the right GPU for your needs and budget
• Step-by-step setup for Jellyfin and Plex with Docker
• GPU passthrough for virtual machines (Proxmox, ESXi, Hyper-V)
• Driver installation and troubleshooting for Linux and Windows
• Performance benchmarks and quality comparisons
• Cost-benefit analysis for different hardware options

Whether you’re building a new media server or optimizing an existing one, hardware transcoding is the single most impactful upgrade you can make.

---

Understanding Hardware Transcoding

What is Transcoding?

Transcoding converts video from one format/quality to another in real-time. Your media server transcodes when:

Scenario 1: Bandwidth Limitation

• Original: 4K HDR 50 Mbps
• Client: Mobile phone on LTE (5 Mbps available)
• Transcode: 4K → 1080p, 50 Mbps → 4 Mbps

Scenario 2: Codec Incompatibility

• Original: HEVC/H.265 video
• Client: Old TV doesn’t support HEVC
• Transcode: HEVC → H.264

Scenario 3: Format Conversion

• Original: MKV container
• Client: Requires MP4
• Remux: Change container without re-encoding video

Scenario 4: HDR to SDR

• Original: HDR10 content
• Client: SDR display
• Tone-map: HDR → SDR color space

Software vs Hardware Transcoding

Software Transcoding (CPU):

Uses CPU cores to encode video using software libraries like x264/x265.

Pros:

• Better quality at low bitrates (more encoding options)
• No additional hardware required
• More encoding presets available

Cons:

• Extremely CPU intensive (70-100% usage per stream)
• High power consumption (100-200W for transcoding)
• Limited concurrent streams (1-2 per modern 8-core CPU)
• Server becomes unusable during transcoding
• Heat generation requires robust cooling

Hardware Transcoding (GPU):

Dedicated silicon on GPU specifically designed for video encoding.

Pros:

• 10-20x faster than software encoding
• 5-10x lower power consumption
• Many concurrent streams (4-8+ depending on GPU)
• Minimal CPU usage (5-15%)
• Doesn’t impact other server tasks
• Lower heat output

Cons:

• Requires compatible GPU
• Slightly lower quality at very low bitrates (negligible for most use cases)
• Limited encoding options compared to software
• Plex requires Plex Pass ($120/year) to unlock

Real-World Example:

Transcoding 4K HDR (50 Mbps) → 1080p SDR (8 Mbps):

Verdict: Hardware transcoding is objectively superior for media servers unless you need absolute maximum quality at ultra-low bitrates (rare scenario).

---

Technology Comparison: Intel vs NVIDIA vs AMD

Intel Quick Sync Video (QSV)

What It Is:

Dedicated video encoding/decoding hardware built into Intel integrated GPUs (iGPU) found in most Intel CPUs since 2011.

Supported CPUs:

• 2nd Gen (Sandy Bridge, 2011): H.264 only, poor quality
• 3rd-6th Gen (Ivy Bridge-Skylake, 2012-2015): Improved H.264, basic HEVC
• 7th-10th Gen (Kaby Lake-Comet Lake, 2017-2020): Excellent H.264, good HEVC, HDR passthrough
• 11th Gen+ (Rocket Lake, 2021+): Best quality, AV1 decode, excellent HEVC
• 12th-14th Gen (Alder Lake-Raptor Lake, 2022-2024): Hardware AV1 encode, best performance

Key Features:

• Included free with CPU (no extra hardware)
• Very power efficient (5-10W encoding)
• Excellent quality since 7th gen
• Supports H.264, HEVC, VP9 (decode), AV1 (12th gen+)
• HDR to SDR tone mapping (11th gen+)

Best For:

• Budget builds (no GPU purchase needed)
• Low power servers (Mini PCs, NUCs)
• Most homelab setups (best value)

Performance:

• Concurrent 1080p streams: 10-15
• Concurrent 4K streams: 6-8
• Power consumption: 5-10W encoding

Limitations:

• Requires iGPU enabled in BIOS
• Only one encoder (unlike NVIDIA’s dual encoders)
• Quality slightly behind latest NVIDIA on 7th-10th gen

NVIDIA NVENC

What It Is:

Dedicated video encoding hardware in NVIDIA GPUs, from GT 1030 to RTX 4090.

Supported GPUs:

Consumer (GeForce):

• GTX 1000 Series (2016): 1 encoder, H.264/HEVC, good quality
• GTX 1650/1660 Series (2019): 1 encoder, improved quality
• RTX 2000 Series (2018): 1 encoder, excellent quality
• RTX 3000 Series (2020): 2 encoders, AV1 encode (RTX 3000 mobile only for AV1)
• RTX 4000 Series (2022): 2 dual encoders (4 total), AV1 encode, best quality

Professional (Quadro/Tesla):

• Unlimited concurrent streams (no artificial limit)
• Same encoder hardware as consumer cards
• 2-3x price premium for same performance

Note: GTX 1650 (base model) lacks NVENC. Get GTX 1650 Super or higher.

Key Features:

• Best encoding quality across all bitrates
• Dual encoders on RTX 3000+ (2 simultaneous encodes per encoder = 4 total)
• Excellent driver support (Linux and Windows)
• AV1 encoding (RTX 4000 series)
• HDR to SDR tone mapping

Best For:

• Maximum quality requirements
• High concurrent stream count (Quadro)
• Already have NVIDIA GPU for other tasks
• Future-proofing (AV1 support)

Performance:

• Concurrent 1080p streams: 15-20 (RTX 3000+)
• Concurrent 4K streams: 8-12 (RTX 3000+)
• Power consumption: 20-50W encoding (depends on model)

Limitations:

• Consumer GPUs limited to ~5 concurrent NVENC sessions (driver limit, can be patched)
• Requires GPU purchase ($150-300 used for transcoding)
• Higher power consumption than Intel QSV

AMD Video Coding Engine (VCE) / VCN

What It Is:

AMD’s video encoding hardware, found in Radeon GPUs and some Ryzen APUs.

Supported Hardware:

GPUs (Radeon):

• RX 400/500 Series (2016-2017): VCE 3.0, basic H.264/HEVC
• RX 5000 Series (2019): VCN 2.0, improved HEVC
• RX 6000 Series (2020): VCN 3.0, good quality
• RX 7000 Series (2022): VCN 4.0, AV1 encode, best quality

APUs (Ryzen with Vega/RDNA):

• Ryzen 3000G/5000G Series: VCN 2.0, integrated graphics
• Ryzen 7000 Series (with iGPU): VCN 3.0, competitive with Intel

Key Features:

• Good quality on latest hardware
• Power efficient
• AV1 encode support (RX 7000 series)
• Competitive pricing

Best For:

• AMD-based systems
• Budget builds (older RX cards cheap)
• Already have AMD GPU

Performance:

• Concurrent 1080p streams: 10-15
• Concurrent 4K streams: 5-8
• Power consumption: 15-40W encoding

Limitations:

• Driver support on Linux historically problematic (improving)
• Smaller ecosystem/community compared to Intel/NVIDIA
• Quality behind NVIDIA until RX 7000 series
• Less documentation for media server use

---

Choosing the Right Hardware

Decision Matrix

For New Builds:

For Existing Systems:

CPU Recommendations (Intel Quick Sync)

Best Value (New):

• Intel i3-12100 ($110) - 4 cores, UHD 730, excellent QSV
• Intel i5-12400 ($180) - 6 cores, UHD 730, best value
• Intel i5-13400 ($220) - 10 cores, UHD 730, future-proof

Best Value (Used):

• Intel i5-8500 ($80) - 8th gen, solid QSV
• Intel i5-9400 ($100) - 9th gen, reliable
• Intel i7-10700 ($200) - 8 cores for other tasks + excellent QSV

Low Power / SFF:

• Intel N100 ($100-150 in mini PC) - 6W TDP, 4 cores, surprising performance
• Intel i5-1135G7 (in NUCs) - Excellent efficiency

Important: Avoid Intel F-series CPUs (e.g., i5-12400F) — the “F” means no iGPU.

GPU Recommendations

NVIDIA (Best Quality):

Budget:

• GTX 1650 Super ($120-150 used) - 1 encoder, excellent value
• GTX 1660 Super ($150-180 used) - Same encoder, more VRAM

Mid-Range:

• RTX 3050 ($200-250 used) - 2 encoders, great efficiency
• RTX 3060 12GB ($250-300 used) - 2 encoders, plenty of VRAM

High-End / Unlimited Streams:

• Quadro P2000 ($200-250 used) - No stream limit, low power
• Quadro P4000 ($300-400 used) - No stream limit, more power

AMD (Budget Alternative):

• RX 6400 ($130-160 new) - Low power, VCN 3.0, single slot
• RX 6500 XT ($150-180) - Slightly better performance
• RX 6600 ($180-220 used) - Good all-around

Important: Quadro cards have no stream limits (NVIDIA driver restriction doesn’t apply), making them ideal for high-concurrency servers despite similar hardware to consumer cards.

Power Consumption Comparison

Measured while transcoding 4K → 1080p (2 simultaneous streams):

Efficiency Winner: Intel Quick Sync (integrated) for lowest total system power.

GPU Winner: NVIDIA RTX 3000 series for best performance-per-watt among discrete GPUs.

---

Setup: Intel Quick Sync (QSV)

Hardware Verification

Step 1: Confirm CPU Has Integrated Graphics

Check your CPU model on Intel ARK:

• Look for “Processor Graphics” field
• Should show “Intel UHD Graphics XXX” or “Intel Iris Xe Graphics”
• If “N/A” or “None”, your CPU lacks iGPU (typically F-series)

Step 2: Enable iGPU in BIOS

Even with iGPU, it may be disabled if you have a discrete GPU installed.

1. Reboot into BIOS/UEFI (usually Delete, F2, or F12 during boot)
2. Find integrated graphics settings (varies by motherboard):
   • Common locations:
     • Advanced → System Agent Configuration → Graphics Configuration
     • Chipset → Internal Graphics
     • Advanced → Integrated Peripherals → Integrated Graphics
3. Change setting:
   • “iGPU Multi-Monitor”: Enabled (allows both iGPU and dGPU)
   • “Primary Display”: Auto or PCIe (use dGPU for display, iGPU for transcoding)
   • “IGPU Memory”: 64MB or higher
4. Save and exit

Step 3: Verify iGPU Detection (Linux)

# List video devices
ls -la /dev/dri/

# Should show:
# renderD128 - for GPU rendering (QuickSync)
# card0 - for display output

# Check detailed GPU info
sudo apt install intel-gpu-tools
intel_gpu_top

# Should show Intel GPU activity

Step 4: Verify iGPU Detection (Windows)

1. Device Manager → Display adapters
2. Should show “Intel(R) UHD Graphics XXX”
3. If missing:
   • Install Intel Graphics drivers from intel.com
   • Ensure iGPU enabled in BIOS

Docker Configuration (Linux)

Step 1: Install Intel GPU Drivers

Ubuntu/Debian:

# Update package list
sudo apt update

# Install i965 driver (older CPUs, pre-11th gen)
sudo apt install i965-va-driver

# Install iHD driver (11th gen+, recommended for newer CPUs)
sudo apt install intel-media-va-driver

# Install VA-API verification tool
sudo apt install vainfo

# Test VA-API support
vainfo

# Should show:
# vainfo: VA-API version: 1.x.x
# vainfo: Driver version: Intel iHD driver...
# VAProfile... : VAEntrypointVLD (multiple profiles listed)

Step 2: Configure Docker Access to GPU

# Check permissions on /dev/dri
ls -la /dev/dri/

# Typical output:
# crw-rw---- 1 root video ... renderD128
# crw-rw---- 1 root video ... card0

# Note the "video" group

# Add your user to video group
sudo usermod -aG video $USER

# Add render group (some systems)
sudo usermod -aG render $USER

# Log out and back in for changes to take effect

Step 3: Jellyfin Docker Compose

version: '3.8'

services:
  jellyfin:
    image: jellyfin/jellyfin:latest
    container_name: jellyfin
    user: 1000:1000  # Replace with your UID:GID
    network_mode: host
    volumes:
      - /opt/jellyfin/config:/config
      - /opt/jellyfin/cache:/cache
      - /mnt/media:/media:ro
    devices:
      - /dev/dri/renderD128:/dev/dri/renderD128  # QuickSync
      - /dev/dri/card0:/dev/dri/card0  # Optional, for display
    environment:
      - TZ=America/New_York
    restart: unless-stopped

Key Configuration:

• devices: section passes GPU into container
• user: 1000:1000 must match user in video group
• If container can’t access GPU, check user ID and group membership

Step 4: Deploy and Verify

# Deploy container
docker compose up -d

# Check container has GPU access
docker exec jellyfin ls -la /dev/dri/

# Should show renderD128 and card0 inside container

# Test VA-API inside container
docker exec jellyfin vainfo

# Should show Intel driver information

Jellyfin Configuration

Enable Hardware Transcoding:

1. Access Jellyfin web interface
2. Dashboard → Playback
3. Transcoding Section:
   • Hardware acceleration: Select Intel QuickSync (QSV)
   • Enable hardware decoding for: Check all supported codecs (H264, HEVC, VP9)
   • Hardware encoding options:
     • Enable hardware encoding: Checked
     • Allow encoding in HEVC format: Checked (if 10th gen+)
     • Enable VPP Tone mapping: Checked (11th gen+, for HDR→SDR)
4. Throttle Transcodes: Checked (prevents overload)
5. Transcoding thread count: Leave default (4-8)
6. FFmpeg path: Leave default
7. Save

Test Hardware Transcoding:

1. Create test user with bitrate limit:
   • Dashboard → Users → Add User
   • Name: test-transcode
   • Set password
2. Configure playback:
   • Dashboard → Users → test-transcode → Playback
   • Internet streaming bitrate: 2 Mbps (forces transcode)
3. Login as test-transcode
4. Play a 4K or 1080p movie
5. As admin, check Dashboard → Active Devices
6. Should show:
   • Transcode Reason: “Bitrate limit”
   • Video codec: Shows “qsv” or hardware indication
7. Monitor CPU usage: Should be under 20% (vs 70-90% for software)

Verify in Logs:

docker logs jellyfin | grep -i "qsv|vaapi"

# Should see lines like:
# Stream mapping: 0:0 -> 0:0 (h264_qsv -> h264_qsv)
# Stream #0:0 -> qsv (h264_qsv)

Plex Configuration

Plex requires Plex Pass to use hardware transcoding ($120/year or $120 lifetime).

Enable Hardware Transcoding (Plex Pass):

1. Settings → Transcoder
2. Use hardware acceleration when available: Checked
3. Use hardware-accelerated video encoding: Checked
4. Maximum simultaneous video transcode: Unlimited (or set limit)
5. Save

Test: Similar process to Jellyfin—force transcode and verify low CPU usage.

---

Setup: NVIDIA NVENC

Driver Installation

Linux (Ubuntu/Debian):

# Check if NVIDIA GPU detected
lspci | grep -i nvidia

# Should show your GPU

# Install NVIDIA drivers
sudo apt update
sudo apt install nvidia-driver-535  # Or latest version

# Reboot
sudo reboot

# Verify driver installation
nvidia-smi

# Should show:
# +-----------------------------------------------------------------------------+
# | NVIDIA-SMI 535.xx       Driver Version: 535.xx       CUDA Version: 12.x |
# +-----------------------------------------------------------------------------+
# | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
# |   0  NVIDIA GeForce RTX 3050   Off  | 00000000:01:00.0  On |                  N/A |

# Install nvidia-docker runtime
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list |   sudo tee /etc/apt/sources.list.d/nvidia-docker.list

sudo apt update
sudo apt install nvidia-docker2

# Restart Docker
sudo systemctl restart docker

# Test nvidia-docker
docker run --rm --gpus all nvidia/cuda:11.0-base nvidia-smi

# Should show nvidia-smi output inside container

Windows:

1. Download driver from NVIDIA.com
2. Install (standard process)
3. Verify in Device Manager → Display adapters → NVIDIA GPU listed

Docker Configuration (NVIDIA)

Jellyfin docker-compose.yml:

version: '3.8'

services:
  jellyfin:
    image: jellyfin/jellyfin:latest
    container_name: jellyfin
    runtime: nvidia  # Use NVIDIA runtime
    environment:
      - NVIDIA_VISIBLE_DEVICES=all
      - NVIDIA_DRIVER_CAPABILITIES=compute,video,utility
      - TZ=America/New_York
    volumes:
      - /opt/jellyfin/config:/config
      - /opt/jellyfin/cache:/cache
      - /mnt/media:/media:ro
    ports:
      - "8096:8096"
    restart: unless-stopped

Key Differences from QSV:

• runtime: nvidia instead of devices: section
• NVIDIA_VISIBLE_DEVICES=all exposes GPU to container
• NVIDIA_DRIVER_CAPABILITIES enables video encoding/decoding

Deploy:

docker compose up -d

# Verify GPU access inside container
docker exec jellyfin nvidia-smi

# Should show GPU information

Jellyfin NVENC Configuration

1. Dashboard → Playback
2. Hardware acceleration: Select NVIDIA NVENC
3. Enable hardware decoding for: Check all supported
4. Hardware encoding options:
   • Enable hardware encoding: Checked
   • Enable VPP Tone mapping: Checked (RTX 2000+)
5. Save

Test: Same process as QSV—force transcode and verify low CPU usage.

Removing NVIDIA Stream Limit (Consumer GPUs)

NVIDIA artificially limits consumer GPUs to ~3 concurrent NVENC streams. Quadro/Tesla have no limit.

Keylase Patch (Linux):

# Install dependencies
sudo apt install build-essential

# Clone keylase patch
git clone https://github.com/keylase/nvidia-patch.git
cd nvidia-patch

# Apply patch (modifies NVIDIA driver)
sudo bash ./patch.sh

# Reboot
sudo reboot

# Verify unlimited streams
# Test by transcoding 5+ streams simultaneously

Warning: This modifies official NVIDIA drivers and may violate license terms. Use at own risk. Quadro GPUs don’t require patching.

---

Setup: AMD VCE/VCN

Driver Installation (Linux)

AMD GPU support on Linux requires Mesa drivers with VAAPI support.

Ubuntu 22.04+:

# Install Mesa VAAPI drivers
sudo apt update
sudo apt install mesa-va-drivers vainfo

# For newer AMD GPUs (RX 6000/7000), use latest Mesa
sudo add-apt-repository ppa:kisak/kisak-mesa
sudo apt update
sudo apt upgrade

# Verify VA-API support
vainfo

# Should show:
# vainfo: VA-API version: 1.x.x
# vainfo: Driver version: Mesa Gallium driver...
# Multiple VAProfile entries for H264, HEVC, etc.

# Check GPU detection
ls -la /dev/dri/

# Should show renderD128 and card0

# Add user to video/render groups
sudo usermod -aG video,render $USER

Install AMD Pro drivers (alternative, for RX 5000 series and older):

# Download from AMD.com, then:
sudo apt install ./amdgpu-install_*.deb
sudo amdgpu-install --usecase=workstation --vulkan=pro --opencl=rocr
sudo reboot

Docker Configuration (AMD)

Jellyfin docker-compose.yml:

version: '3.8'

services:
  jellyfin:
    image: jellyfin/jellyfin:latest
    container_name: jellyfin
    user: 1000:1000  # Your UID:GID
    volumes:
      - /opt/jellyfin/config:/config
      - /opt/jellyfin/cache:/cache
      - /mnt/media:/media:ro
    devices:
      - /dev/dri/renderD128:/dev/dri/renderD128
      - /dev/dri/card0:/dev/dri/card0
      - /dev/kfd:/dev/kfd  # For ROCm (optional)
    environment:
      - TZ=America/New_York
    ports:
      - "8096:8096"
    restart: unless-stopped

Deploy:

docker compose up -d

# Verify GPU access
docker exec jellyfin vainfo

# Should show AMD VAAPI driver

Jellyfin AMD Configuration

1. Dashboard → Playback
2. Hardware acceleration: Select Video Acceleration API (VAAPI)
3. VAAPI Device: /dev/dri/renderD128 (default)
4. Enable hardware decoding for: Check H264, HEVC
5. Hardware encoding options:
   • Enable hardware encoding: Checked
   • Enable VPP Tone mapping: Check if supported (RX 6000+)
6. Save

Note: AMD encoding quality improved significantly with RX 6000/7000 series. Older cards (RX 500) may show artifacts at low bitrates.

---

Proxmox GPU Passthrough

For running Jellyfin/Plex in Proxmox VMs with GPU access.

Intel iGPU Passthrough (Proxmox)

Step 1: Enable IOMMU

Edit /etc/default/grub:

sudo nano /etc/default/grub

# Find line starting with GRUB_CMDLINE_LINUX_DEFAULT
# Change to:
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt"

# For AMD CPUs:
# GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt"

# Update grub
sudo update-grub

# Reboot
sudo reboot

Step 2: Load VFIO Modules

Edit /etc/modules:

sudo nano /etc/modules

# Add these lines:
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

Update initramfs:

sudo update-initramfs -u -k all
sudo reboot

Step 3: Identify iGPU PCI Address

lspci | grep VGA

# Example output:
# 00:02.0 VGA compatible controller: Intel Corporation UHD Graphics 730 (rev 0c)

# Note the address: 00:02.0

Step 4: Configure VM

In Proxmox web interface:

1. Select VM → Hardware → Add → PCI Device
2. Device: Select 0000:00:02.0 (Intel Graphics)
3. All Functions: Unchecked
4. Primary GPU: Unchecked (unless using for display)
5. PCI-Express: Checked (if available)
6. Add

Step 5: Start VM and Verify

Inside VM:

# Should see Intel GPU
lspci | grep VGA

# Install drivers and verify VA-API
sudo apt install intel-media-va-driver vainfo
vainfo

NVIDIA GPU Passthrough (Proxmox)

Step 1: Enable IOMMU (same as above)

Step 2: Blacklist NVIDIA Drivers on Host

Proxmox host shouldn’t use GPU—only pass to VM.

sudo nano /etc/modprobe.d/blacklist.conf

# Add:
blacklist nouveau
blacklist nvidia
blacklist nvidiafb
blacklist nvidia_drm

sudo update-initramfs -u -k all

Step 3: Identify GPU PCI Address

lspci | grep NVIDIA

# Example:
# 01:00.0 VGA compatible controller: NVIDIA Corporation GA107 [GeForce RTX 3050]
# 01:00.1 Audio device: NVIDIA Corporation GA107 High Definition Audio

# Note BOTH addresses (GPU + audio)

Step 4: Bind to VFIO

# Get vendor and device IDs
lspci -n -s 01:00

# Example output:
# 01:00.0 0300: 10de:2584 (rev a1)
# 01:00.1 0403: 10de:228b (rev a1)

# Note IDs: 10de:2584 and 10de:228b

sudo nano /etc/modprobe.d/vfio.conf

# Add:
options vfio-pci ids=10de:2584,10de:228b

sudo update-initramfs -u -k all
sudo reboot

Step 5: Pass GPU to VM

In Proxmox:

1. VM → Hardware → Add → PCI Device
2. Select both GPU functions (01:00.0 and 01:00.1)
3. All Functions: Checked
4. Primary GPU: Unchecked
5. ROM-Bar: Checked
6. PCI-Express: Checked
7. Add

Step 6: Configure VM for NVIDIA

Inside Ubuntu VM:

# Install NVIDIA drivers
sudo apt install nvidia-driver-535

# Install nvidia-docker (if using Docker)
# (See earlier NVIDIA Docker section)

sudo reboot

# Verify
nvidia-smi

---

Performance Benchmarks

Test Methodology

All tests performed transcoding same 4K HDR source (50 Mbps HEVC) to 1080p SDR (8 Mbps H.264).

Hardware Tested:

• Intel i5-12400 (UHD 730, 12th gen QSV)
• Intel i7-10700 (UHD 630, 10th gen QSV)
• NVIDIA RTX 3050
• NVIDIA Quadro P2000
• AMD RX 6400

Metrics:

• Transcode speed (FPS)
• CPU usage during transcode
• Power consumption
• Quality (VMAF score)
• Maximum concurrent 4K streams

Results

Single Stream Performance:

VMAF: Higher is better, 90+ is excellent. Differences under 2 points are imperceptible.

Maximum Concurrent 4K Streams:

Findings:

1. Intel 12th gen offers best value: Nearly matches NVIDIA quality at $0 additional cost
2. NVIDIA RTX 3000 series wins quality: Highest VMAF, most streams
3. Quadro advantage: No stream limits, more concurrent sessions
4. AMD competitive on RX 6000+: Earlier generations (RX 500) significantly worse quality
5. Software transcoding obsolete: 10x power consumption, 1/8th the streams

---

Troubleshooting

Intel QSV: “No hardware encoder available”

Symptoms:

• Jellyfin shows “Playback failed” or falls back to software encoding
• Logs show “No VA-API device found”

Solutions:

1. Verify iGPU enabled in BIOS

# On Proxmox host or bare metal
ls -la /dev/dri/

# Should show renderD128 and card0
# If missing, iGPU disabled in BIOS

2. Check Docker device mapping

docker inspect jellyfin | grep -A 5 Devices

# Should show:
# "Devices": [
#   {
#     "PathOnHost": "/dev/dri/renderD128",
#     "PathInContainer": "/dev/dri/renderD128",

3. Verify user in video group

# Inside container
docker exec jellyfin id

# Should show "video" in groups
# If not, fix user mapping in docker-compose.yml

4. Install correct VA-API driver

# Inside container (if installing manually)
apt install intel-media-va-driver  # For 11th gen+
apt install i965-va-driver          # For 7th-10th gen

# Test
vainfo

NVIDIA NVENC: “Cannot load CUDA library”

Symptoms:

• Jellyfin/Plex shows NVENC errors
• Logs mention missing CUDA

Solutions:

1. Verify nvidia-docker runtime

docker info | grep Runtime

# Should show nvidia in runtimes list

# If missing, reinstall nvidia-docker2
sudo apt install nvidia-docker2
sudo systemctl restart docker

2. Check environment variables

# In docker-compose.yml
environment:
  - NVIDIA_VISIBLE_DEVICES=all
  - NVIDIA_DRIVER_CAPABILITIES=compute,video,utility

3. Test GPU inside container

docker exec jellyfin nvidia-smi

# Should show GPU info
# If "command not found", runtime not working

4. Driver version mismatch

# Check host driver
nvidia-smi

# Check container driver support
docker run --rm --gpus all nvidia/cuda:11.0-base nvidia-smi

# If versions differ significantly, update host driver

AMD: Poor quality or artifacts

Symptoms:

• Visible blocking, color banding
• Quality worse than software encode

Solutions:

1. Update Mesa drivers

# Use latest Mesa for RX 6000/7000
sudo add-apt-repository ppa:kisak/kisak-mesa
sudo apt update && sudo apt upgrade

2. Check GPU generation

• RX 400/500: VCE 3.0, quality mediocre (consider software encode)
• RX 5000: VCN 2.0, acceptable quality
• RX 6000/7000: VCN 3.0/4.0, competitive quality

3. Adjust encoding settings In Jellyfin playback settings:

• Lower encoding preset (slower = better quality)
• Increase target bitrate slightly

Proxmox: GPU not visible in VM

Symptoms:

• lspci inside VM doesn’t show GPU
• Passthrough configured but not working

Solutions:

1. Verify IOMMU enabled

# On Proxmox host
dmesg | grep -i iommu

# Should show "IOMMU enabled"

2. Check IOMMU groups

# On Proxmox host
find /sys/kernel/iommu_groups/ -type l

# GPU should be in isolated group
# If GPU shares group with other devices, enable ACS override

3. Enable ACS override (if needed)

# Edit /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction"

sudo update-grub
sudo reboot

4. Check VM machine type

• Set to q35 (not i440fx)
• VM → Hardware → Machine: q35

---

Conclusion

Hardware transcoding transforms your media server from a struggling, power-hungry CPU bottleneck into an efficient, scalable streaming platform. The performance improvements are dramatic: 10x more concurrent streams, 1/5th the power consumption, and near-silent operation.

Key Takeaways:

1. Intel Quick Sync is the best value - Built into most Intel CPUs, free, excellent quality since 7th gen
2. NVIDIA offers maximum quality - RTX 3000+ series delivers best encoding quality and most streams
3. AMD is catching up - RX 6000/7000 series competitive, great for AMD-based systems
4. Quadro removes limits - For serious multi-user servers, Quadro eliminates stream restrictions
5. Software encoding is obsolete - No reason to use CPU encoding if GPU available

Recommended Setup by Use Case:

Personal Server (1-3 users):

• Intel CPU with iGPU (i5-12400 or similar)
• Cost: $0 additional
• Handles 6-8 concurrent 4K streams effortlessly

Family Server (3-6 users):

• Intel i5-12400 or NVIDIA RTX 3050
• Cost: $0-250
• Handles 8-12 streams with headroom

Shared Server (10+ users):

• NVIDIA Quadro P2000/P4000 or RTX 3060+
• Cost: $200-400
• Unlimited streams (Quadro) or 15+ (RTX)

The investment in hardware transcoding—whether $0 (enable iGPU) or $150 (used GPU)—pays for itself immediately in reduced power consumption and improved user experience. In 2025, there’s simply no reason to run a media server without GPU acceleration.

---

Additional Resources

Official Documentation

• Intel Quick Sync
• NVIDIA NVENC
• Jellyfin Hardware Acceleration
• Plex Hardware Transcoding

Community Resources

• r/jellyfin Hardware Guide
• Proxmox GPU Passthrough Guide
• TrueNAS Scale GPU Guide

Tools

• intel-gpu-tools - Intel GPU monitoring
• nvidia-smi - NVIDIA monitoring
• radeontop - AMD GPU monitoring

Document Version: 1.0 Last Updated: December 16, 2025 Word Count: ~6,800 words