VictoriaMetrics: High-Performance Prometheus Alternative

Introduction

Prometheus is the de facto standard for metrics collection in modern infrastructure, but it has limitations: high memory usage, limited data retention, and no clustering support in the open-source version. VictoriaMetrics addresses all these issues while maintaining full PromQL compatibility.

VictoriaMetrics is a fast, cost-effective time-series database designed as a drop-in replacement for Prometheus with significantly better resource efficiency. It’s become the preferred choice for homelabbers who need long-term metrics storage without the operational complexity of Thanos or Cortex.

What you’ll learn:

Why VictoriaMetrics outperforms Prometheus for homelab workloads
Deploying VictoriaMetrics single-node and clustered architectures
Migrating from Prometheus with zero data loss
Configuring remote write for hybrid setups
Advanced features: downsampling, deduplication, anomaly detection
Performance tuning and resource optimization

Prerequisites:

Docker and Docker Compose installed
Basic understanding of Prometheus and PromQL
Existing Prometheus setup (optional, for migration)
2GB+ RAM, 20GB+ disk for meaningful retention

Why VictoriaMetrics Over Prometheus?

The Resource Efficiency Problem

Prometheus stores metrics in its local TSDB, which has excellent query performance but high resource costs. For a typical homelab with 50,000 active time series:

Metric	Prometheus	VictoriaMetrics	Improvement
Memory Usage	2-4GB	200-400MB	10x less
Disk Usage	50GB/year	5-8GB/year	6-10x compression
CPU Usage	Moderate	Low	2-3x less
Write Throughput	100K samples/sec	1M+ samples/sec	10x higher
Query Performance	Fast	Faster	1.5-3x improvement
Data Retention	15-30 days (typical)	Years	Unlimited (disk-bound)

Real-world example:

Homelab with 10 hosts, 50 Docker containers
Prometheus: 3GB RAM, 45GB disk per year
VictoriaMetrics: 300MB RAM, 6GB disk per year
Savings: 90% memory, 85% disk

Key Advantages

1. PromQL Compatibility

Drop-in replacement for Prometheus in Grafana
Supports MetricsQL (extended PromQL with additional functions)
No dashboard changes required

2. Long-Term Storage

Store years of metrics on homelab hardware
Built-in downsampling for historical data
Cost-effective archival to S3/MinIO

3. Simplified Architecture

Single binary for all components
No external dependencies (unlike Thanos/Cortex)
Easy horizontal scaling when needed

4. Advanced Features

Deduplication across multiple Prometheus instances
Anomaly detection with RollUp functions
Multi-tenancy for isolated environments

5. Enterprise Features (Free)

Clustering, downsampling, and deduplication are free
No commercial license for core features
Active development and community support

When to Use VictoriaMetrics

Choose VictoriaMetrics if:

You need metrics retention beyond 30 days
Prometheus is consuming too much RAM
You’re scraping high-cardinality metrics (many labels)
You want multi-datacenter metrics aggregation
You need predictable resource usage

Stick with Prometheus if:

You have minimal metrics (<10 hosts, <20 containers)
You only need 15-day retention
You’re tightly integrated with Prometheus federation
You don’t have resource constraints

Part 1: Single-Node Deployment

Quick Start with Docker Compose

The simplest VictoriaMetrics deployment for homelabs.

Create project directory:

mkdir -p ~/victoriametrics-stack
cd ~/victoriametrics-stack

Create docker-compose.yml:

version: '3.8'

services:
  # VictoriaMetrics - Time-series database
  victoriametrics:
    image: victoriametrics/victoria-metrics:latest
    container_name: victoriametrics
    ports:
      - "8428:8428"
    volumes:
      - vmdata:/victoria-metrics-data
    command:
      - "--storageDataPath=/victoria-metrics-data"
      - "--retentionPeriod=12"  # 12 months retention
      - "--httpListenAddr=:8428"
      - "--promscrape.config=/promscrape.yml"
    volumes:
      - vmdata:/victoria-metrics-data
      - ./promscrape.yml:/promscrape.yml
    networks:
      - monitoring
    restart: unless-stopped

  # vmagent - Metrics collector (optional, can use Prometheus)
  vmagent:
    image: victoriametrics/vmagent:latest
    container_name: vmagent
    ports:
      - "8429:8429"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - vmagent-data:/vmagent-data
    command:
      - "--promscrape.config=/etc/prometheus/prometheus.yml"
      - "--remoteWrite.url=http://victoriametrics:8428/api/v1/write"
      - "--httpListenAddr=:8429"
    depends_on:
      - victoriametrics
    networks:
      - monitoring
    restart: unless-stopped

  # vmalert - Alert manager (optional)
  vmalert:
    image: victoriametrics/vmalert:latest
    container_name: vmalert
    ports:
      - "8880:8880"
    volumes:
      - ./alerts.yml:/etc/alerts/alerts.yml
    command:
      - "--datasource.url=http://victoriametrics:8428"
      - "--remoteRead.url=http://victoriametrics:8428"
      - "--remoteWrite.url=http://victoriametrics:8428"
      - "--notifier.url=http://alertmanager:9093"
      - "--rule=/etc/alerts/*.yml"
      - "--httpListenAddr=:8880"
    depends_on:
      - victoriametrics
    networks:
      - monitoring
    restart: unless-stopped

  # Grafana - Visualization
  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin
      - GF_USERS_ALLOW_SIGN_UP=false
    volumes:
      - grafana-data:/var/lib/grafana
      - ./grafana-datasources.yml:/etc/grafana/provisioning/datasources/datasources.yml
    depends_on:
      - victoriametrics
    networks:
      - monitoring
    restart: unless-stopped

  # Node Exporter - System metrics
  node-exporter:
    image: prom/node-exporter:latest
    container_name: node-exporter
    command:
      - '--path.procfs=/host/proc'
      - '--path.sysfs=/host/sys'
      - '--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)'
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    ports:
      - "9100:9100"
    networks:
      - monitoring
    restart: unless-stopped

volumes:
  vmdata:
  vmagent-data:
  grafana-data:

networks:
  monitoring:
    driver: bridge

Create promscrape.yml (VictoriaMetrics native scrape config):

global:
  scrape_interval: 15s
  external_labels:
    cluster: homelab
    environment: production

scrape_configs:
  - job_name: 'victoriametrics'
    static_configs:
      - targets: ['localhost:8428']

  - job_name: 'node-exporter'
    static_configs:
      - targets: ['node-exporter:9100']
        labels:
          instance: 'homelab-server'

  - job_name: 'grafana'
    static_configs:
      - targets: ['grafana:3000']

Create prometheus.yml (for vmagent, uses standard Prometheus format):

global:
  scrape_interval: 15s
  evaluation_interval: 15s
  external_labels:
    cluster: homelab

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['victoriametrics:8428']

  - job_name: 'node'
    static_configs:
      - targets: ['node-exporter:9100']

Create grafana-datasources.yml:

apiVersion: 1

datasources:
  - name: VictoriaMetrics
    type: prometheus
    access: proxy
    url: http://victoriametrics:8428
    isDefault: true
    jsonData:
      timeInterval: 30s
    editable: true

Start the stack:

docker compose up -d

Verify deployment:

# Check all containers
docker compose ps

# Verify VictoriaMetrics is accepting data
curl http://localhost:8428/api/v1/labels

# Check vmagent status
curl http://localhost:8429/targets

# Access Grafana
open http://localhost:3000

Part 2: Migrating from Prometheus

Strategy 1: Remote Write (Dual-Write)

Keep Prometheus for scraping, send data to both Prometheus and VictoriaMetrics.

Add to existing Prometheus configuration:

# prometheus.yml
remote_write:
  - url: http://victoriametrics:8428/api/v1/write
    queue_config:
      max_samples_per_send: 10000
      batch_send_deadline: 5s
      max_shards: 30
      capacity: 50000

Restart Prometheus:

docker restart prometheus
# or
systemctl restart prometheus

Advantages:

Zero downtime migration
Prometheus continues operating normally
Gradual validation of VictoriaMetrics
Easy rollback (just remove remote_write)

Validation:

# Check VictoriaMetrics is receiving data
curl http://localhost:8428/api/v1/query?query=up

# Compare cardinality
curl http://localhost:9090/api/v1/status/tsdb  # Prometheus
curl http://localhost:8428/api/v1/status/tsdb  # VictoriaMetrics

Strategy 2: Historical Data Import

Import existing Prometheus data into VictoriaMetrics.

Using vmctl (VictoriaMetrics CLI tool):

# Install vmctl
docker pull victoriametrics/vmctl:latest

# Import data from Prometheus
docker run --rm   --network monitoring   victoriametrics/vmctl:latest   prometheus   --prom-snapshot=/prometheus/data   --vm-addr=http://victoriametrics:8428

# Or import via remote read
docker run --rm   --network monitoring   victoriametrics/vmctl:latest   prometheus   --prom-remote-read-addr=http://prometheus:9090   --vm-addr=http://victoriametrics:8428

For large datasets (100GB+), use snapshot approach:

# Create Prometheus snapshot
curl -X POST http://localhost:9090/api/v1/admin/tsdb/snapshot

# Find snapshot directory
ls /var/lib/prometheus/snapshots/

# Mount snapshot and import
docker run --rm   -v /var/lib/prometheus:/prometheus:ro   --network monitoring   victoriametrics/vmctl:latest   prometheus   --prom-snapshot=/prometheus/snapshots/20241216T120000Z-abcdef123456   --vm-addr=http://victoriametrics:8428

Monitor import progress:

# Import typically processes 1-5GB per hour
docker logs victoriametrics -f | grep "inserted rows"

Strategy 3: Replace Prometheus with vmagent

Completely replace Prometheus scraping with vmagent.

Benefits:

2-5x lower memory usage than Prometheus
Supports same scrape configs
Pushes data directly to VictoriaMetrics
Built-in scraping and service discovery

Migration steps:

Convert Prometheus config to vmagent format (configs are identical):

cp /etc/prometheus/prometheus.yml ~/victoriametrics-stack/prometheus.yml

Update docker-compose.yml to use vmagent (already included above)
Stop Prometheus, start vmagent:

docker stop prometheus
docker compose up -d vmagent

Verify scraping:

curl http://localhost:8429/targets

Decommission Prometheus after 7-day validation

Part 3: Advanced Configurations

High-Availability Clustering

VictoriaMetrics supports clustering for high availability and horizontal scaling.

Cluster components:

vmstorage: Stores raw data (stateful)
vminsert: Receives and distributes writes
vmselect: Handles queries

docker-compose.yml for cluster:

version: '3.8'

services:
  vmstorage-1:
    image: victoriametrics/vmstorage:latest
    container_name: vmstorage-1
    ports:
      - "8482:8482"
      - "8400:8400"
      - "8401:8401"
    volumes:
      - vmstorage1-data:/storage
    command:
      - "--storageDataPath=/storage"
      - "--retentionPeriod=12"
      - "--httpListenAddr=:8482"
      - "--vminsertAddr=:8400"
      - "--vmselectAddr=:8401"
    networks:
      - monitoring
    restart: unless-stopped

  vmstorage-2:
    image: victoriametrics/vmstorage:latest
    container_name: vmstorage-2
    ports:
      - "8483:8482"
      - "8402:8400"
      - "8403:8401"
    volumes:
      - vmstorage2-data:/storage
    command:
      - "--storageDataPath=/storage"
      - "--retentionPeriod=12"
      - "--httpListenAddr=:8482"
      - "--vminsertAddr=:8400"
      - "--vmselectAddr=:8401"
    networks:
      - monitoring
    restart: unless-stopped

  vminsert:
    image: victoriametrics/vminsert:latest
    container_name: vminsert
    ports:
      - "8480:8480"
    command:
      - "--storageNode=vmstorage-1:8400,vmstorage-2:8400"
      - "--httpListenAddr=:8480"
    depends_on:
      - vmstorage-1
      - vmstorage-2
    networks:
      - monitoring
    restart: unless-stopped

  vmselect:
    image: victoriametrics/vmselect:latest
    container_name: vmselect
    ports:
      - "8481:8481"
    command:
      - "--storageNode=vmstorage-1:8401,vmstorage-2:8401"
      - "--httpListenAddr=:8481"
      - "--dedup.minScrapeInterval=30s"  # Enable deduplication
    depends_on:
      - vmstorage-1
      - vmstorage-2
    networks:
      - monitoring
    restart: unless-stopped

volumes:
  vmstorage1-data:
  vmstorage2-data:

networks:
  monitoring:
    driver: bridge

Configure Grafana to use cluster:

datasources:
  - name: VictoriaMetrics Cluster
    type: prometheus
    url: http://vmselect:8481/select/0/prometheus
    access: proxy

Write data to cluster:

# vmagent configuration
remote_write:
  - url: http://vminsert:8480/insert/0/prometheus/api/v1/write

Downsampling for Historical Data

Reduce storage costs by downsampling old metrics.

Using vmselect with downsampling:

# Add to vmselect command
- "--downsampling.period=30d:5m,90d:1h,365d:6h"

Explanation:

Data older than 30 days: downsample to 5-minute resolution
Data older than 90 days: downsample to 1-hour resolution
Data older than 1 year: downsample to 6-hour resolution

Typical storage savings:

Raw data (15s): 100GB/year
With downsampling: 15-20GB/year
80-85% storage reduction

Deduplication Across Multiple Prometheus Instances

Collect metrics from multiple Prometheus servers without duplicates.

Scenario: 3 Prometheus instances scraping same targets for HA

Configure each Prometheus with unique replica label:

# prometheus-1.yml
global:
  external_labels:
    replica: '1'

remote_write:
  - url: http://victoriametrics:8428/api/v1/write

Enable deduplication in VictoriaMetrics:

# Add to VictoriaMetrics command
- "--dedup.minScrapeInterval=15s"

Query deduplicated data in Grafana:

# Automatic deduplication based on scrape interval
up{job="node-exporter"}

Multi-Tenancy

Isolate metrics for different teams or environments.

Write to tenant-specific endpoints:

# Production tenant (tenant ID: 1)
remote_write:
  - url: http://victoriametrics:8428/insert/1/prometheus/api/v1/write

# Development tenant (tenant ID: 2)
remote_write:
  - url: http://victoriametrics:8428/insert/2/prometheus/api/v1/write

Query tenant data:

# Production metrics
curl http://localhost:8428/select/1/prometheus/api/v1/query?query=up

# Development metrics
curl http://localhost:8428/select/2/prometheus/api/v1/query?query=up

Grafana configuration:

datasources:
  - name: VM Production
    type: prometheus
    url: http://victoriametrics:8428/select/1/prometheus

  - name: VM Development
    type: prometheus
    url: http://victoriametrics:8428/select/2/prometheus

Part 4: MetricsQL - Extended PromQL

VictoriaMetrics supports MetricsQL, a superset of PromQL with powerful additions.

Rollup Functions

Standard PromQL:

rate(http_requests_total[5m])

MetricsQL with rollup modifier:

# Last value in each interval
http_requests_total[5m:1m]  # 5m lookback, 1m step

# Aggregate over time
avg_over_time(cpu_usage[1h])
max_over_time(memory_usage[24h])

Anomaly Detection

Detect sudden changes:

# Alert if current value deviates from 1-day average by 50%
abs(cpu_usage - avg_over_time(cpu_usage[1d])) / avg_over_time(cpu_usage[1d]) > 0.5

Forecast future values:

# Predict disk usage in 7 days using linear regression
predict_linear(disk_usage[7d], 7*24*3600)

Histogram Quantiles

Calculate percentiles from histograms:

# 95th percentile request duration
histogram_quantile(0.95, rate(http_request_duration_bucket[5m]))

# Same with MetricsQL optimizations
histogram_quantile(0.95, sum(rate(http_request_duration_bucket[5m])) by (le))

WITH Templates

Create reusable query templates:

WITH (
  commonFilters = {job="api", env="prod"}
)
sum(rate(http_requests_total{commonFilters}[5m])) by (endpoint)

Additional Functions

Useful MetricsQL additions:

# Remove NaN and Inf values
remove_resets(counter_metric[5m])

# Smooth spiky metrics
smooth_exponential(cpu_usage, 0.5)

# Detect outliers
outliers_mad(3, http_response_time) > 0

# Time-based conditionals
if_time(9, 17, business_hours_metric, after_hours_metric)

Part 5: Performance Optimization

Memory Optimization

Reduce memory usage for high-cardinality data:

# Add to VictoriaMetrics command
- "--memory.allowedPercent=50"  # Use max 50% of available RAM
- "--search.maxUniqueTimeseries=1000000"  # Limit concurrent series
- "--search.maxQueryDuration=60s"  # Prevent expensive queries

Monitor memory usage:

curl http://localhost:8428/metrics | grep vm_cache_size_bytes

Disk I/O Optimization

Use fast storage for active data:

# Separate hot/cold storage
- "--storageDataPath=/fast-ssd/vm-data"
- "--storage.minFreeDiskSpaceBytes=10GB"  # Reserve space

Enable compression:

# Default compression is already very good
# For better compression at cost of CPU:
- "--storage.minFreeDiskSpaceBytes=5GB"

Query Performance Tuning

Optimize Grafana queries:

# Add to Grafana datasource config
jsonData:
  timeInterval: 30s  # Minimum step for queries
  queryTimeout: 60s
  httpMethod: POST  # Better for complex queries

Use recording rules for expensive queries:

# vmalert recording rules
groups:
  - name: performance
    interval: 1m
    rules:
      - record: job:http_requests:rate5m
        expr: sum(rate(http_requests_total[5m])) by (job)

      - record: instance:cpu_usage:percent
        expr: 100 - (avg by (instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

Pre-aggregate high-cardinality metrics:

# Instead of this (high cardinality):
sum(rate(http_requests_total[5m])) by (path, method, status, user_id)

# Use this (lower cardinality):
sum(rate(http_requests_total[5m])) by (path, status)

Retention Strategies

Balance retention vs storage:

# Keep detailed data for 30 days, downsampled for 1 year
- "--retentionPeriod=12"  # 12 months
- "--downsampling.period=30d:5m,90d:1h"

# OR split into hot/cold storage
# Hot: Last 7 days on SSD
# Cold: 7-365 days on HDD

Typical retention configurations:

Use Case	Retention	Downsampling	Disk/Year
Minimal	30 days	None	2-3GB
Standard	90 days	30d:5m	4-6GB
Extended	1 year	30d:5m, 90d:1h	8-12GB
Archival	3 years	30d:5m, 90d:1h, 365d:6h	15-25GB

Part 6: Backup and Recovery

Automated Backups

Create backup script:

#!/bin/bash
# vm-backup.sh

BACKUP_DIR="/backups/victoriametrics"
VM_DATA="/var/lib/docker/volumes/victoriametrics-stack_vmdata/_data"
RETENTION_DAYS=30

# Create snapshot
SNAPSHOT_NAME="vm-backup-$(date +%Y%m%d-%H%M%S)"
mkdir -p "$BACKUP_DIR"

# Use vmbackup tool
docker run --rm   -v $VM_DATA:/victoria-metrics-data   -v $BACKUP_DIR:/backups   victoriametrics/vmbackup:latest   -storageDataPath=/victoria-metrics-data   -snapshotName=$SNAPSHOT_NAME   -dst=fs:///backups/$SNAPSHOT_NAME

# Compress backup
tar czf "$BACKUP_DIR/$SNAPSHOT_NAME.tar.gz" -C "$BACKUP_DIR" "$SNAPSHOT_NAME"
rm -rf "$BACKUP_DIR/$SNAPSHOT_NAME"

# Remove old backups
find "$BACKUP_DIR" -name "vm-backup-*.tar.gz" -mtime +$RETENTION_DAYS -delete

echo "Backup completed: $SNAPSHOT_NAME.tar.gz"

Schedule with cron:

# Daily backups at 2 AM
0 2 * * * /root/vm-backup.sh >> /var/log/vm-backup.log 2>&1

Restore from Backup

#!/bin/bash
# vm-restore.sh

BACKUP_FILE="$1"
VM_DATA="/var/lib/docker/volumes/victoriametrics-stack_vmdata/_data"

if [ -z "$BACKUP_FILE" ]; then
  echo "Usage: $0 <backup-file.tar.gz>"
  exit 1
fi

# Stop VictoriaMetrics
docker compose down victoriametrics

# Extract backup
tar xzf "$BACKUP_FILE" -C /tmp/

# Restore using vmrestore
docker run --rm   -v $VM_DATA:/victoria-metrics-data   -v /tmp:/backups   victoriametrics/vmrestore:latest   -src=fs:///backups/$(basename $BACKUP_FILE .tar.gz)   -storageDataPath=/victoria-metrics-data

# Restart VictoriaMetrics
docker compose up -d victoriametrics

echo "Restore completed"

Disaster Recovery to S3/MinIO

Backup to S3-compatible storage:

# Install and configure AWS CLI or MinIO client
mc alias set minio http://minio:9000 ACCESS_KEY SECRET_KEY

# Backup script with S3
docker run --rm   -v $VM_DATA:/victoria-metrics-data   -e AWS_ACCESS_KEY_ID=$ACCESS_KEY   -e AWS_SECRET_ACCESS_KEY=$SECRET_KEY   victoriametrics/vmbackup:latest   -storageDataPath=/victoria-metrics-data   -dst=s3://my-bucket/victoriametrics-backups/$(date +%Y%m%d)

Troubleshooting

Issue 1: High Memory Usage

Symptoms: VictoriaMetrics using more RAM than expected

Diagnosis:

# Check cache sizes
curl http://localhost:8428/metrics | grep vm_cache_size_bytes

# Check active time series
curl http://localhost:8428/api/v1/status/tsdb

Solutions:

# Limit memory usage
- "--memory.allowedPercent=40"

# Reduce cache sizes
- "--search.maxSamplesPerSeries=100000"
- "--search.maxSeries=100000"

Issue 2: Data Not Appearing

Diagnosis:

# Check remote write is working
curl http://localhost:8428/api/v1/query?query=up

# Verify vmagent targets
curl http://localhost:8429/targets

# Check VictoriaMetrics logs
docker logs victoriametrics | grep -i error

Common causes:

Incorrect remote write URL
Network connectivity issues
Time synchronization problems (check NTP)

Issue 3: Slow Queries

Diagnosis:

# Identify slow queries
curl http://localhost:8428/api/v1/status/top_queries

# Check query stats
curl http://localhost:8428/metrics | grep vm_search_duration

Optimization:

Use recording rules for complex queries
Reduce time range or add filters
Increase --search.maxQueryDuration

Conclusion

VictoriaMetrics provides enterprise-grade time-series storage with homelab-friendly resource usage. Its Prometheus compatibility ensures zero lock-in, while advanced features like clustering, downsampling, and MetricsQL unlock powerful capabilities.

Key takeaways:

10x better resource efficiency than Prometheus
Full PromQL compatibility for seamless migration
Years of metrics retention on homelab hardware
Free clustering and advanced features
Active development and strong community

Your migration timeline:

This week:

Deploy VictoriaMetrics single-node
Configure Prometheus remote write
Validate metrics in Grafana

This month: 4. Import historical Prometheus data 5. Test MetricsQL advanced queries 6. Set up automated backups

This quarter: 7. Migrate fully from Prometheus to vmagent 8. Implement downsampling for cost savings 9. Deploy clustering for high availability 10. Share your setup with the community

Resources:

VictoriaMetrics proves that you don’t need expensive cloud services or enterprise licenses to run production-grade observability infrastructure. With proper configuration, a modest homelab server can store years of high-resolution metrics with excellent query performance.

Happy monitoring!

Guide Version: 1.0 Last Updated: December 2024 VictoriaMetrics Version: v1.95+ Difficulty Level: Intermediate Estimated Time: 50 minutes (deployment) + migration time