Monitoring intermediate ⏱ 45 minutes

Grafana Monitoring: Complete Homelab Guide

Build a comprehensive monitoring stack with Grafana and Prometheus. Dashboards, alerts, and observability.

By homelabbing.io • 2025-12-16
#grafana#prometheus#monitoring#observability#docker

Introduction

Monitoring is the foundation of a well-maintained homelab. When you’re running multiple services—from media servers to DNS blockers, from Docker containers to custom applications—understanding what’s happening in your infrastructure becomes critical. Grafana provides a powerful, open-source solution for visualizing and analyzing this data.

This guide will walk you through setting up a complete monitoring stack using Grafana, Prometheus, and supporting tools. By the end, you’ll have a production-ready monitoring system with dashboards, alerts, and even AI-powered automation capabilities.

What You’ll Learn:

  • Understanding Grafana’s architecture and why it’s perfect for homelabs
  • Deploying a complete monitoring stack with Docker Compose
  • Creating effective dashboards with real-world queries
  • Setting up intelligent alerting and notifications
  • Implementing AI automation for anomaly detection and predictive analytics
  • Troubleshooting common issues and optimizing performance

Prerequisites:

  • Docker and Docker Compose installed on your system
  • Basic understanding of YAML syntax and command-line tools
  • A homelab server (Raspberry Pi 4+, NAS, or dedicated server)
  • Minimum 2GB RAM, 10GB disk space (4GB+ RAM recommended)

Part 1: Understanding the Monitoring Stack

What is Grafana?

Grafana is an open-source visualization and analytics platform that serves as the “dashboard layer” for your monitoring infrastructure. Think of it as a sophisticated control panel that connects to various data sources and presents the information in meaningful ways.

Key Characteristics:

  • Open-source and freely available (AGPL license)
  • Multi-platform support (Linux, Windows, macOS, ARM devices)
  • Lightweight footprint (~100MB base, 200-500MB runtime)
  • Extensive plugin ecosystem with 10,000+ community dashboards
  • 25+ visualization types for different data presentations

Why Grafana for Homelabs?

Homelabs typically consist of diverse infrastructure: Raspberry Pis, Docker containers, media servers (Jellyfin, Plex), DNS services (Pi-Hole), file servers, network equipment, and IoT devices. Managing these individually is inefficient. Grafana provides:

Centralized Monitoring

  • Single interface for all services
  • Unified view of system health
  • Easy correlation of related metrics

Low Resource Overhead

  • Runs well on Raspberry Pi and other low-power hardware
  • Efficient memory and CPU usage
  • No expensive enterprise licensing

Community Support

  • Active homelab community
  • Pre-built dashboards for common services
  • Extensive documentation and examples

Educational Value

  • Learn observability principles
  • Master time-series databases
  • Understand DevOps practices
  • Gain practical experience with monitoring

The LGTM Stack

Modern observability relies on the LGTM Stack:

  • Loki: Log aggregation and analysis
  • Grafana: Visualization and dashboards
  • Tempo: Distributed tracing (optional for homelabs)
  • Mimir: Scalable metrics storage (optional for homelabs)

For most homelabs, the essential combination is Prometheus + Grafana + Loki. This provides:

  • Prometheus: Metrics collection and storage
  • Grafana: Visualization and alerting
  • Loki: Log aggregation and analysis

Understanding Data Sources

Prometheus: The Metrics Engine

Prometheus is a time-series database designed for collecting and storing metrics. It uses a “pull-based” model where it periodically scrapes metrics from configured endpoints.

How Prometheus Works:

Services expose metrics → Prometheus scrapes (every 15s) → Stores in TSDB → Grafana queries with PromQL → Visualizes

Common Exporters:

  • Node Exporter: OS-level metrics (CPU, memory, disk, network)
  • cAdvisor: Docker container metrics and resource usage
  • Blackbox Exporter: Service availability and latency monitoring
  • Custom exporters: Application-specific metrics

Loki: The Logging Layer

Loki is a log aggregation system inspired by Prometheus. Unlike traditional logging solutions, Loki doesn’t index log content—it only indexes labels, making it extremely cost-effective.

Key Benefits:

  • Label-based indexing (same labels as Prometheus)
  • Compressed storage with configurable retention
  • Natural integration with Prometheus metrics
  • Correlation between logs and metrics using shared labels

When to Use InfluxDB

While Prometheus handles most homelab use cases, InfluxDB might be better if you have:

  • Very high-volume metrics (>1M points/second)
  • High-cardinality data (many unique label combinations)
  • Push-based metric collection requirements

For this guide, we’ll focus on Prometheus as it’s simpler and more suitable for typical homelab scenarios.

Part 2: Deploying Your Monitoring Stack

Quick Start with Docker Compose

We’ll deploy a complete monitoring stack using Docker Compose. This includes Grafana, Prometheus, Node Exporter (for system metrics), and cAdvisor (for container metrics).

Create Project Structure:

mkdir -p ~/grafana-monitoring
cd ~/grafana-monitoring

Create docker-compose.yml:

version: '3.8'

services:
  # Time-Series Database for Metrics
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    restart: unless-stopped
    ports:
      - '9090:9090'
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prometheus-storage:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--storage.tsdb.retention.time=30d'
      - '--web.enable-lifecycle'
    networks:
      - monitoring
    healthcheck:
      test: ["CMD", "wget", "--quiet", "--tries=1", "--spider", "http://localhost:9090/-/healthy"]
      interval: 30s
      timeout: 10s
      retries: 3

  # System Metrics Collector
  node-exporter:
    image: prom/node-exporter:latest
    container_name: node-exporter
    restart: unless-stopped
    ports:
      - '9100:9100'
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    command:
      - '--path.procfs=/host/proc'
      - '--path.sysfs=/host/sys'
      - '--path.rootfs=/rootfs'
      - '--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)'
    networks:
      - monitoring

  # Container Metrics Collector
  cadvisor:
    image: gcr.io/cadvisor/cadvisor:latest
    container_name: cadvisor
    restart: unless-stopped
    ports:
      - '8080:8080'
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
      - /dev/disk/:/dev/disk:ro
    networks:
      - monitoring
    privileged: true
    devices:
      - /dev/kmsg

  # Visualization and Dashboarding
  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    restart: unless-stopped
    ports:
      - '3000:3000'
    environment:
      # Security Configuration
      - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_ADMIN_PASSWORD:-ChangeMeNow123!}
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_COOKIE_SECURE=false
      - GF_SECURITY_DISABLE_BRUTE_FORCE_LOGIN_PROTECTION=false

      # User Management
      - GF_USERS_ALLOW_SIGN_UP=false
      - GF_USERS_DEFAULT_THEME=dark

      # Plugin Installation (optional)
      - GF_INSTALL_PLUGINS=grafana-piechart-panel
    volumes:
      - grafana-storage:/var/lib/grafana
      - ./provisioning:/etc/grafana/provisioning
    networks:
      - monitoring
    depends_on:
      prometheus:
        condition: service_healthy
    healthcheck:
      test: ["CMD", "wget", "--quiet", "--tries=1", "--spider", "http://localhost:3000/api/health"]
      interval: 30s
      timeout: 10s
      retries: 3

volumes:
  prometheus-storage:
    driver: local
  grafana-storage:
    driver: local

networks:
  monitoring:
    driver: bridge
    ipam:
      config:
        - subnet: 172.28.0.0/16

Create prometheus.yml:

global:
  scrape_interval: 15s          # How often to scrape targets
  evaluation_interval: 15s       # How often to evaluate rules
  external_labels:
    monitor: 'homelab'
    cluster: 'main'

# Scrape Configurations
scrape_configs:
  # Monitor Prometheus itself
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']
        labels:
          service: 'prometheus'

  # System metrics from Node Exporter
  - job_name: 'node-exporter'
    static_configs:
      - targets: ['node-exporter:9100']
        labels:
          instance: 'homelab-server'
          environment: 'production'

  # Container metrics from cAdvisor
  - job_name: 'cadvisor'
    static_configs:
      - targets: ['cadvisor:8080']
        labels:
          instance: 'homelab-server'

  # Add your additional services here
  # Example: Pi-Hole DNS monitoring
  # - job_name: 'pihole'
  #   scrape_interval: 30s
  #   static_configs:
  #     - targets: ['192.168.1.5:9617']
  #       labels:
  #         service: 'dns-adblock'

Create .env file (optional, for custom admin password):

GRAFANA_ADMIN_PASSWORD=YourSecurePasswordHere

Deploying the Stack

Start all services:

docker-compose up -d

Verify everything is running:

docker-compose ps

You should see all four services in the “Up” state:

NAME              STATUS    PORTS
grafana           Up        0.0.0.0:3000->3000/tcp
prometheus        Up        0.0.0.0:9090->9090/tcp
node-exporter     Up        0.0.0.0:9100->9100/tcp
cadvisor          Up        0.0.0.0:8080->8080/tcp

Check logs for any errors:

docker-compose logs -f grafana

Access your services:

  • Grafana: http://localhost:3000 (admin / ChangeMeNow123!)
  • Prometheus: http://localhost:9090
  • Node Exporter: http://localhost:9100/metrics
  • cAdvisor: http://localhost:8080

Initial Grafana Configuration

  1. Log in to Grafana at http://localhost:3000
  2. Change the admin password (Settings → Profile → Change Password)
  3. Add Prometheus data source:
    • Navigate to Configuration → Data Sources
    • Click Add data source
    • Select Prometheus
    • Configure:
      • Name: Prometheus
      • URL: http://prometheus:9090
      • Access: Server (default)
    • Click Save & Test
    • You should see “Data source is working”

Part 3: Building Effective Dashboards

Dashboard Design Principles

Before creating dashboards, understand these key principles:

  1. Purpose First: Each dashboard should answer specific questions
  2. Progressive Disclosure: Show summary first, allow drilling down for details
  3. Context Matters: Include time ranges, alert status, and metadata
  4. Consistent Layout: Group related panels logically
  5. Performance: Optimize queries to avoid slow load times

Creating Your First Dashboard

Let’s build a comprehensive “Homelab Overview” dashboard.

Step 1: Create New Dashboard

  1. Click Dashboards in left menu
  2. Click + Create Dashboard
  3. Click + Add visualization

Step 2: CPU Usage Panel

Select Prometheus as data source and enter this query:

100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

Explanation: This calculates CPU usage percentage by:

  • Taking the rate of idle CPU time over 5 minutes
  • Averaging across all CPU cores
  • Converting to percentage
  • Subtracting from 100 to get “used” percentage

Configuration:

  • Panel title: “CPU Usage”
  • Visualization: Time series graph
  • Unit: Percent (0-100)
  • Thresholds:
    • Green: < 50
    • Yellow: 50-80
    • Red: > 80
  • Legend: Show as table
  • Tooltip mode: All series

Step 3: Memory Usage Panel

Add new panel with this query:

(1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100

Explanation: Calculates memory usage by:

  • Dividing available memory by total memory
  • Subtracting from 1 to get “used” ratio
  • Multiplying by 100 for percentage

Configuration:

  • Panel title: “Memory Usage”
  • Visualization: Gauge
  • Unit: Percent (0-100)
  • Thresholds:
    • Green: < 70
    • Yellow: 70-85
    • Red: > 85

Step 4: Disk Space Panel

(node_filesystem_avail_bytes{fstype!~"tmpfs|fuse.*|squashfs|vfat"}
 / node_filesystem_size_bytes{fstype!~"tmpfs|fuse.*|squashfs|vfat"}) * 100

Explanation: Shows available disk space by:

  • Excluding temporary filesystems (tmpfs, fuse, etc.)
  • Calculating ratio of available to total space
  • Converting to percentage

Configuration:

  • Panel title: “Disk Space Available”
  • Visualization: Bar gauge
  • Unit: Percent (0-100)
  • Display labels: Show filesystem mount points
  • Thresholds:
    • Green: > 20
    • Yellow: 10-20
    • Red: < 10

Step 5: Network Traffic Panel

# Receive rate (bytes per second)
rate(node_network_receive_bytes_total{device!~"lo|docker.*|veth.*"}[5m])

# Transmit rate (bytes per second)
rate(node_network_transmit_bytes_total{device!~"lo|docker.*|veth.*"}[5m])

Explanation: Tracks network traffic by:

  • Excluding loopback and virtual interfaces
  • Calculating per-second rate over 5 minutes
  • Showing both receive and transmit separately

Configuration:

  • Panel title: “Network Traffic”
  • Visualization: Time series graph
  • Unit: Bytes/sec
  • Multiple series: Show RX and TX on same graph
  • Legend: Table format with current/max values

Step 6: Container Resource Usage

# Container memory usage in MB
container_memory_usage_bytes{name!=""} / 1024 / 1024

# Container CPU usage percentage
rate(container_cpu_usage_seconds_total{name!=""}[5m]) * 100

Configuration:

  • Panel title: “Container Resources”
  • Visualization: Table
  • Show both CPU and memory columns
  • Sort by memory usage (descending)
  • Format: CPU as percentage, Memory as MB

Step 7: Service Uptime

# Uptime percentage over last 24 hours
avg_over_time(up[24h]) * 100

Configuration:

  • Panel title: “Service Availability (24h)”
  • Visualization: Stat
  • Unit: Percent (0-100)
  • Display: Value and sparkline
  • Thresholds:
    • Green: > 99.5
    • Yellow: 99-99.5
    • Red: < 99

Step 8: Save Dashboard

  1. Click Save dashboard (disk icon)
  2. Name: “Homelab Overview”
  3. Add description: “Primary monitoring dashboard for homelab infrastructure”
  4. Set folder: Create “Homelab” folder
  5. Save

Dashboard Variables and Templates

Variables make dashboards reusable across different hosts or time ranges.

Add Host Variable:

  1. Click Dashboard settings (gear icon)
  2. Select Variables
  3. Click + New variable
  4. Configure:
    • Name: host
    • Type: Query
    • Label: Host
    • Data source: Prometheus
    • Query: label_values(node_uname_info, nodename)
    • Regex: (leave empty)
    • Sort: Alphabetical (asc)
    • Multi-select: Enable
    • Include All option: Enable
  5. Click Apply

Add Interval Variable:

  1. Click + New variable
  2. Configure:
    • Name: interval
    • Type: Interval
    • Label: Refresh Interval
    • Values: 1m,5m,15m,30m,1h,6h,12h,1d
    • Auto option: Enable
  3. Click Apply

Use Variables in Queries:

Modify your CPU query to use the host variable:

100 - (avg by(instance) (rate(node_cpu_seconds_total{instance=~"$host.*", mode="idle"}[$interval])) * 100)

Now you can:

  • Select specific hosts from dropdown
  • Change time interval dynamically
  • Share dashboard configuration across multiple servers

Importing Pre-Built Dashboards

The Grafana community provides thousands of pre-built dashboards. Here are essential ones for homelabs:

Node Exporter Full (ID: 1860)

  1. Dashboards → Import
  2. Enter dashboard ID: 1860
  3. Click Load
  4. Select Prometheus data source
  5. Click Import

This dashboard provides comprehensive Linux system metrics including:

  • CPU, Memory, Disk, Network
  • System load and processes
  • File system statistics
  • Detailed performance metrics

Docker Container & Host Metrics (ID: 11600)

  • Import ID: 11600
  • Provides Docker-specific monitoring
  • Container resource usage
  • Host and container correlations

Customizing Imported Dashboards:

  1. After import, click Edit on any panel
  2. Modify queries to match your setup
  3. Adjust thresholds for your hardware
  4. Remove unnecessary panels
  5. Save As with new name to preserve original

Part 4: Intelligent Alerting and Notifications

Understanding Alert Architecture

Grafana’s alerting system works like this:

Prometheus evaluates metric → Condition met → Alert fires
    ↓
Grafana Alert Manager receives alert
    ↓
Matches notification policy
    ↓
Routes to contact point (Email, Slack, Discord)
    ↓
User receives notification

Setting Up Contact Points

Configure Email Alerts:

  1. Navigate to Alerting → Contact points
  2. Click + Create contact point
  3. Configure:
    • Name: Email Alerts
    • Integration: Email
    • Addresses: [email protected]
    • Include image: Enable (attaches dashboard snapshot)
  4. Click Test to verify
  5. Click Save contact point

Configure Slack Alerts:

  1. Create Slack webhook:

  2. In Grafana:

    • Alerting → Contact points
    • + Create contact point
    • Configure:
      • Name: Slack Alerts
      • Integration: Slack
      • Webhook URL: Paste your webhook URL
      • Username: Grafana Alert Bot
      • Channel: #homelab-alerts
    • Test and save

Configure Discord Alerts:

  1. Create Discord webhook:

    • Server Settings → Integrations → Webhooks
    • Create webhook, copy URL

  2. In Grafana:

    • + Create contact point
    • Integration: Discord
    • Webhook URL: Paste Discord webhook
    • Message title: Homelab Alert
    • Test and save

Creating Essential Alert Rules

Alert 1: High CPU Usage

  1. Alerting → Alert rules
  2. + Create alert rule
  3. Configure:

Rule name: High CPU Usage Query:

100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

Condition: WHEN last() OF query(A) IS ABOVE 80

Evaluation behavior:

  • Folder: Homelab Alerts
  • Evaluation group: System Resources
  • Evaluation interval: 1m
  • Pending period: 5m (wait 5 minutes before firing)

Annotations:

  • Summary: High CPU usage on [[ $labels.instance ]]
  • Description: CPU usage is [[ $value ]]% on [[ $labels.instance ]]
  • Runbook URL: (link to your troubleshooting docs)

Labels:

  • severity: warning
  • component: cpu
  1. Click Save rule and exit

Alert 2: High Memory Usage

(1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100

Condition: IS ABOVE 85 Pending period: 5m Annotations:

  • Summary: High memory usage on [[ $labels.instance ]]
  • Description: Memory usage is [[ $value ]]% on [[ $labels.instance ]] Labels:
  • severity: warning
  • component: memory

Alert 3: Low Disk Space

(node_filesystem_avail_bytes{fstype!~"tmpfs|fuse.*"}
 / node_filesystem_size_bytes{fstype!~"tmpfs|fuse.*"}) * 100

Condition: IS BELOW 15 Pending period: 10m Annotations:

  • Summary: Low disk space on [[ $labels.instance ]]
  • Description: Only [[ $value ]]% disk space remaining on [[ $labels.mountpoint ]] Labels:
  • severity: critical
  • component: disk

Alert 4: Service Down

up{job="prometheus"}

Condition: IS BELOW 1 (0 means down) Pending period: 1m Annotations:

  • Summary: Service [[ $labels.job ]] is down
  • Description: [[ $labels.instance ]] has been unreachable for 1 minute Labels:
  • severity: critical
  • component: availability

Alert 5: Container Restart

increase(container_start_time_seconds{name!=""}[5m]) &gt; 0

Condition: IS ABOVE 0 Pending period: 1m Annotations:

  • Summary: Container [[ $labels.name ]] restarted
  • Description: Container restarted, possible crash or manual restart Labels:
  • severity: info
  • component: containers

Configuring Notification Policies

Notification policies control how alerts are routed to different contact points based on labels.

  1. Alerting → Notification policies
  2. Edit Default policy:
Group by: ['alertname', 'instance', 'severity']
Group wait: 30s          # Wait 30s to batch related alerts
Group interval: 5m       # Send group update every 5m
Repeat interval: 1h      # Repeat unresolved alerts every hour
  1. Add child policies for different severities:

Critical Alerts Policy:

  • Matching labels: severity = critical
  • Contact point: Email Alerts + Slack Alerts
  • Group wait: 10s (faster notification)
  • Repeat interval: 30m (more frequent)

Warning Alerts Policy:

  • Matching labels: severity = warning
  • Contact point: Slack Alerts
  • Group wait: 5m
  • Repeat interval: 4h

Info Alerts Policy:

  • Matching labels: severity = info
  • Contact point: Slack Alerts (separate channel)
  • Group wait: 15m
  • Repeat interval: Never (don’t repeat)

Alert Silencing and Maintenance Windows

Create Silence for Maintenance:

  1. Alerting → Silences
  2. + Create silence
  3. Configure:
    • Matchers: instance = “homelab-server”
    • Duration: 2 hours
    • Comment: “Scheduled maintenance - system upgrade”
    • Creator: your-name
  4. Click Submit

Alerts matching the criteria won’t fire during the silence period.

Part 5: AI Automation and Advanced Features

Anomaly Detection with Machine Learning

Let’s implement intelligent threshold detection using machine learning. This Python script uses historical data from Prometheus to automatically determine appropriate alert thresholds.

Create anomaly_detection.py:

#!/usr/bin/env python3
"""
Grafana Anomaly Detection
Automatically detects unusual patterns in metrics using machine learning
"""

import requests
import numpy as np
from sklearn.ensemble import IsolationForest
from sklearn.preprocessing import StandardScaler
import time
from datetime import datetime

class AnomalyDetector:
    def __init__(self, prometheus_url='http://localhost:9090'):
        self.prom_url = prometheus_url
        self.scaler = StandardScaler()

    def fetch_metrics(self, query, duration_days=7):
        """Fetch historical metrics from Prometheus"""
        url = f'{self.prom_url}/api/v1/query_range'
        end_time = int(time.time())
        start_time = end_time - (duration_days * 86400)

        params = {
            'query': query,
            'start': start_time,
            'end': end_time,
            'step': '1h'  # 1 hour resolution
        }

        response = requests.get(url, params=params, timeout=30)
        response.raise_for_status()
        data = response.json()

        if not data['data']['result']:
            raise ValueError(f"No data returned for query: {query}")

        values = data['data']['result'][0]['values']
        return np.array([float(v[1]) for v in values])

    def detect_anomalies(self, data, contamination=0.05):
        """
        Detect anomalies using Isolation Forest algorithm
        contamination: Expected proportion of outliers (5% default)
        """
        data_scaled = self.scaler.fit_transform(data.reshape(-1, 1))

        clf = IsolationForest(
            contamination=contamination,
            random_state=42,
            n_estimators=100
        )

        predictions = clf.fit_predict(data_scaled)
        # -1 for anomalies, 1 for normal
        return predictions

    def calculate_dynamic_threshold(self, data, percentile=95):
        """
        Calculate recommended threshold based on historical data
        Uses 95th percentile + margin for safety
        """
        base_threshold = np.percentile(data, percentile)
        margin = np.std(data) * 0.5  # Half standard deviation as buffer
        return base_threshold + margin

    def analyze_metric(self, metric_name, query):
        """Complete analysis of a metric"""
        print(f"\n{'='*60}")
        print(f"Analyzing: {metric_name}")
        print(f"{'='*60}")

        try:
            # Fetch data
            data = self.fetch_metrics(query)
            print(f"✓ Fetched {len(data)} data points")

            # Detect anomalies
            anomalies = self.detect_anomalies(data)
            anomaly_count = np.sum(anomalies == -1)
            anomaly_pct = (anomaly_count / len(data)) * 100

            print(f"✓ Found {anomaly_count} anomalies ({anomaly_pct:.2f}%)")

            # Calculate statistics
            stats = {
                'mean': np.mean(data),
                'median': np.median(data),
                'std': np.std(data),
                'min': np.min(data),
                'max': np.max(data),
                'p95': np.percentile(data, 95),
                'p99': np.percentile(data, 99)
            }

            # Calculate recommended threshold
            threshold = self.calculate_dynamic_threshold(data)

            # Display results
            print(f"\nStatistics:")
            print(f"  Mean: {stats['mean']:.2f}")
            print(f"  Median: {stats['median']:.2f}")
            print(f"  Std Dev: {stats['std']:.2f}")
            print(f"  Min/Max: {stats['min']:.2f} / {stats['max']:.2f}")
            print(f"  95th percentile: {stats['p95']:.2f}")
            print(f"  99th percentile: {stats['p99']:.2f}")

            print(f"\n🎯 Recommended Alert Threshold: {threshold:.2f}")

            # Find anomaly times
            if anomaly_count &gt; 0:
                anomaly_indices = np.where(anomalies == -1)[0]
                print(f"\nAnomaly detected at indices: {anomaly_indices[:5].tolist()}...")

            return {
                'metric': metric_name,
                'statistics': stats,
                'recommended_threshold': threshold,
                'anomaly_count': anomaly_count,
                'anomaly_percentage': anomaly_pct
            }

        except Exception as e:
            print(f"✗ Error: {str(e)}")
            return None

# Example usage
if __name__ == '__main__':
    detector = AnomalyDetector()

    # Analyze CPU usage
    cpu_result = detector.analyze_metric(
        'CPU Usage',
        '100 - (avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)'
    )

    # Analyze memory usage
    mem_result = detector.analyze_metric(
        'Memory Usage',
        '(1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100'
    )

    # Analyze disk usage
    disk_result = detector.analyze_metric(
        'Disk Usage',
        '(1 - (node_filesystem_avail_bytes / node_filesystem_size_bytes)) * 100'
    )

    print(f"\n{'='*60}")
    print("Analysis Complete")
    print(f"{'='*60}\n")

Install dependencies:

pip3 install requests numpy scikit-learn

Run the analysis:

python3 anomaly_detection.py

This will output recommended alert thresholds based on your actual usage patterns rather than arbitrary values.

Predictive Analytics for Capacity Planning

This script predicts when resources will run out based on historical trends.

Create capacity_planning.py:

#!/usr/bin/env python3
"""
Predictive Capacity Planning
Forecast when resources will be exhausted
"""

import requests
import numpy as np
from sklearn.linear_model import LinearRegression
from datetime import datetime, timedelta
import time

class CapacityPlanner:
    def __init__(self, prometheus_url='http://localhost:9090'):
        self.prom_url = prometheus_url

    def fetch_metrics(self, query, duration_days=30):
        """Fetch 30 days of historical data"""
        url = f'{self.prom_url}/api/v1/query_range'
        end_time = int(time.time())
        start_time = end_time - (duration_days * 86400)

        params = {
            'query': query,
            'start': start_time,
            'end': end_time,
            'step': '6h'  # 6 hour resolution
        }

        response = requests.get(url, params=params, timeout=30)
        response.raise_for_status()
        data = response.json()

        values = data['data']['result'][0]['values']
        return np.array([float(v[1]) for v in values])

    def predict_exhaustion(self, data, threshold=90, days_ahead=30):
        """
        Predict when resource will exceed threshold
        Returns days until exhaustion or None
        """
        # Prepare data
        X = np.arange(len(data)).reshape(-1, 1)
        y = data

        # Fit linear regression model
        model = LinearRegression()
        model.fit(X, y)

        # Predict future
        hours_ahead = days_ahead * 4  # 6-hour intervals
        future_X = np.arange(len(data), len(data) + hours_ahead).reshape(-1, 1)
        predictions = model.predict(future_X)

        # Find threshold crossing
        crossing_indices = np.where(predictions > threshold)[0]

        if len(crossing_indices) &gt; 0:
            first_crossing = crossing_indices[0]
            hours_until = first_crossing * 6  # 6-hour intervals
            days_until = hours_until / 24
            crossing_date = datetime.now() + timedelta(days=days_until)

            return {
                'will_exhaust': True,
                'days_until': days_until,
                'crossing_date': crossing_date.strftime('%Y-%m-%d %H:%M'),
                'predicted_peak': np.max(predictions),
                'current_trend': model.coef_[0],
                'r_squared': model.score(X, y)
            }

        return {
            'will_exhaust': False,
            'predicted_peak': np.max(predictions),
            'current_trend': model.coef_[0],
            'r_squared': model.score(X, y)
        }

    def analyze_capacity(self, resource_name, query, threshold=90):
        """Analyze capacity for a specific resource"""
        print(f"\n{'='*60}")
        print(f"Capacity Analysis: {resource_name}")
        print(f"{'='*60}")

        try:
            data = self.fetch_metrics(query)
            current_value = data[-1]

            print(f"Current value: {current_value:.2f}%")
            print(f"Alert threshold: {threshold}%")

            result = self.predict_exhaustion(data, threshold)

            print(f"\nTrend: {result['current_trend']:.4f}% per 6 hours")
            print(f"Model accuracy (R²): {result['r_squared']:.4f}")
            print(f"Predicted peak (30 days): {result['predicted_peak']:.2f}%")

            if result['will_exhaust']:
                print(f"\n⚠️  WARNING: {resource_name} will exceed {threshold}%")
                print(f"   Estimated date: {result['crossing_date']}")
                print(f"   Days until: {result['days_until']:.1f}")
                print(f"\n   Recommended actions:")

                if 'disk' in resource_name.lower():
                    print(f"   - Clean up old logs and temporary files")
                    print(f"   - Archive or delete unused data")
                    print(f"   - Consider adding storage capacity")
                elif 'memory' in resource_name.lower():
                    print(f"   - Identify memory-hungry processes")
                    print(f"   - Optimize application memory usage")
                    print(f"   - Consider adding RAM")
            else:
                print(f"\n✓ {resource_name} trending normally")
                print(f"  No capacity issues expected in next 30 days")

            return result

        except Exception as e:
            print(f"✗ Error: {str(e)}")
            return None

# Example usage
if __name__ == '__main__':
    planner = CapacityPlanner()

    # Analyze disk capacity
    disk_result = planner.analyze_capacity(
        'Disk Usage',
        '(1 - (node_filesystem_avail_bytes{fstype!~"tmpfs|fuse.*"} / node_filesystem_size_bytes{fstype!~"tmpfs|fuse.*"})) * 100',
        threshold=90
    )

    # Analyze memory capacity
    mem_result = planner.analyze_capacity(
        'Memory Usage',
        '(1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100',
        threshold=85
    )

    print(f"\n{'='*60}")
    print("Capacity Planning Complete")
    print(f"{'='*60}\n")

Run capacity planning:

python3 capacity_planning.py

Auto-Remediation with Webhooks

Create an automated response system that can fix common issues automatically.

Create auto_remediation.py:

#!/usr/bin/env python3
"""
Auto-Remediation Service
Automatically responds to alerts and fixes common issues
"""

from flask import Flask, request, jsonify
import subprocess
import logging
import json

app = Flask(__name__)
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

def restart_container(container_name):
    """Restart a Docker container"""
    try:
        result = subprocess.run(
            ['docker', 'restart', container_name],
            capture_output=True,
            text=True,
            timeout=30
        )
        if result.returncode == 0:
            logger.info(f"✓ Restarted container: {container_name}")
            return True
        else:
            logger.error(f"✗ Failed to restart {container_name}: {result.stderr}")
            return False
    except Exception as e:
        logger.error(f"✗ Error restarting container: {str(e)}")
        return False

def cleanup_docker():
    """Clean up Docker resources"""
    try:
        # Remove unused images
        subprocess.run(['docker', 'image', 'prune', '-f'], timeout=60)

        # Remove unused volumes
        subprocess.run(['docker', 'volume', 'prune', '-f'], timeout=60)

        # Remove stopped containers
        subprocess.run(['docker', 'container', 'prune', '-f'], timeout=60)

        logger.info("✓ Docker cleanup completed")
        return True
    except Exception as e:
        logger.error(f"✗ Docker cleanup failed: {str(e)}")
        return False

def clear_logs():
    """Clear system logs"""
    try:
        subprocess.run(['journalctl', '--vacuum-time=7d'], timeout=60)
        logger.info("✓ Cleared old system logs")
        return True
    except Exception as e:
        logger.error(f"✗ Log cleanup failed: {str(e)}")
        return False

@app.route('/webhook/alert', methods=['POST'])
def handle_alert():
    """Handle incoming alerts from Grafana"""
    try:
        alert_data = request.json
        logger.info(f"Received alert: {json.dumps(alert_data, indent=2)}")

        if alert_data.get('status') == 'firing':
            alerts = alert_data.get('alerts', [])

            for alert in alerts:
                labels = alert.get('labels', {})
                annotations = alert.get('annotations', {})

                alert_name = labels.get('alertname')
                severity = labels.get('severity')
                component = labels.get('component')

                logger.info(f"Processing: {alert_name} (severity: {severity})")

                # Auto-remediation logic
                actions_taken = []

                # High disk usage - cleanup
                if component == 'disk' and severity in ['critical', 'warning']:
                    logger.info("Taking action: Disk cleanup")
                    if cleanup_docker():
                        actions_taken.append("Docker cleanup")
                    if clear_logs():
                        actions_taken.append("Log cleanup")

                # Service down - restart container
                if component == 'availability':
                    job_name = labels.get('job')
                    if job_name:
                        logger.info(f"Taking action: Restart service {job_name}")
                        if restart_container(job_name):
                            actions_taken.append(f"Restarted {job_name}")

                # Container issues - restart
                if component == 'containers':
                    container_name = labels.get('name')
                    if container_name:
                        logger.info(f"Taking action: Restart container {container_name}")
                        if restart_container(container_name):
                            actions_taken.append(f"Restarted {container_name}")

                # Log actions
                if actions_taken:
                    logger.info(f"✓ Actions completed: {', '.join(actions_taken)}")
                else:
                    logger.info("No automated actions available for this alert")

        return jsonify({'status': 'ok', 'message': 'Alert processed'}), 200

    except Exception as e:
        logger.error(f"Error processing alert: {str(e)}")
        return jsonify({'status': 'error', 'message': str(e)}), 500

@app.route('/health', methods=['GET'])
def health_check():
    """Health check endpoint"""
    return jsonify({'status': 'healthy'}), 200

if __name__ == '__main__':
    logger.info("Starting auto-remediation service on port 5000")
    app.run(host='0.0.0.0', port=5000, debug=False)

Install dependencies:

pip3 install flask

Run the service:

python3 auto_remediation.py

Configure Grafana to use webhook:

  1. Alerting → Contact points

  2. + Create contact point

  3. Configure:

    • Name: Auto Remediation
    • Integration: Webhook
    • URL: http://localhost:5000/webhook/alert
    • HTTP Method: POST

  4. Test and save

  5. Update notification policies to include auto-remediation for critical alerts

Grafana Beyla: Auto-Instrumentation with eBPF

Grafana Beyla automatically instruments applications without code changes using eBPF (Extended Berkeley Packet Filter).

Add to docker-compose.yml:

  beyla:
    image: grafana/beyla:latest
    container_name: beyla
    privileged: true
    pid: host
    restart: unless-stopped
    volumes:
      - /proc:/proc:ro
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
    environment:
      - BEYLA_OPEN_PORT=8080              # Port to instrument
      - BEYLA_METRICS_PORT=9090            # Export to Prometheus
      - BEYLA_SERVICE_NAME=homelab-services
      - BEYLA_PRINT_TRACES=true            # Debug output
    networks:
      - monitoring

Add Beyla to Prometheus configuration:

  - job_name: 'beyla'
    static_configs:
      - targets: ['beyla:9090']
        labels:
          service: 'beyla'

Restart services:

docker-compose up -d

Beyla will now automatically instrument any HTTP/gRPC services and expose RED metrics (Rate, Errors, Duration).

Part 6: Troubleshooting and Optimization

Common Issues and Solutions

Issue: Grafana can’t connect to Prometheus

Symptoms:

  • “Connection refused” error
  • Dashboards show “No data”
  • Red X on data source

Solution:

# 1. Verify Prometheus is running
docker ps | grep prometheus

# 2. Test connectivity from Grafana container
docker exec grafana curl http://prometheus:9090

# 3. Check Prometheus targets
curl http://localhost:9090/api/v1/targets

# 4. Verify data source URL in Grafana
# Should be: http://prometheus:9090 (NOT localhost:9090)

Issue: High CPU or memory usage

Symptoms:

  • Containers using 100% CPU
  • Out of memory errors
  • Slow dashboard loading

Solutions:

# Check resource usage
docker stats grafana prometheus

# View logs for errors
docker logs -f grafana

# Optimize Prometheus retention
# Edit prometheus.yml command section:
- '--storage.tsdb.retention.time=7d'      # Reduce from 30d
- '--storage.tsdb.retention.size=2GB'     # Add size limit

# Restart Prometheus
docker-compose restart prometheus

Issue: Missing metrics

Symptoms:

  • Query returns “no data”
  • Some exporters not reporting

Solutions:

# List all available metrics
curl 'http://localhost:9090/api/v1/label/__name__/values' | jq .

# Check if exporter is accessible
curl http://localhost:9100/metrics

# Verify Prometheus scrape config
docker exec prometheus cat /etc/prometheus/prometheus.yml

# Check Prometheus targets status
curl http://localhost:9090/api/v1/targets | jq .

Issue: Alerts not firing

Symptoms:

  • Rules created but never fire
  • No notifications received

Solutions:

# Check alert rules
curl http://localhost:9090/api/v1/rules | jq .

# View Grafana logs
docker logs grafana | grep -i alert

# Test alert condition manually
# In Grafana: Alerting → Alert rules → [Your Rule] → Test

Query Optimization

Slow queries vs. Fast queries:

# SLOW: No label filters
container_memory_usage_bytes

# BETTER: Filter by job
container_memory_usage_bytes{job="docker"}

# BEST: Specific label matchers
container_memory_usage_bytes{job="docker", instance="homelab-server"}

Recording Rules for Performance:

Create prometheus-rules.yml:

groups:
  - name: performance_rules
    interval: 30s
    rules:
      # Pre-calculate CPU usage
      - record: instance:cpu_usage:rate5m
        expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

      # Pre-calculate memory usage
      - record: instance:memory_usage:percent
        expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100

      # Pre-calculate disk usage
      - record: instance:disk_usage:percent
        expr: (1 - (node_filesystem_avail_bytes / node_filesystem_size_bytes)) * 100

Update docker-compose.yml:

  prometheus:
    volumes:
      - ./prometheus-rules.yml:/etc/prometheus/prometheus-rules.yml:ro
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--web.enable-lifecycle'

Update prometheus.yml:

rule_files:
  - 'prometheus-rules.yml'

Now use pre-calculated metrics in dashboards:

# Instead of complex query, use recording rule
instance:cpu_usage:rate5m

Backup and Recovery

Backup script (backup-grafana.sh):

#!/bin/bash
# Grafana Monitoring Stack Backup

BACKUP_DIR="/backups/grafana"
TIMESTAMP=$(date +%Y%m%d_%H%M%S)

echo "Starting backup at $TIMESTAMP"

# Create backup directory
mkdir -p "$BACKUP_DIR"

# Backup Grafana data
echo "Backing up Grafana..."
docker run --rm   -v grafana-storage:/data   -v "$BACKUP_DIR":/backup   alpine tar czf "/backup/grafana_${TIMESTAMP}.tar.gz" -C /data .

# Backup Prometheus data (optional - can regenerate)
echo "Backing up Prometheus..."
docker run --rm   -v prometheus-storage:/data   -v "$BACKUP_DIR":/backup   alpine tar czf "/backup/prometheus_${TIMESTAMP}.tar.gz" -C /data .

# Backup configurations
echo "Backing up configurations..."
cp docker-compose.yml "$BACKUP_DIR/docker-compose_${TIMESTAMP}.yml"
cp prometheus.yml "$BACKUP_DIR/prometheus_${TIMESTAMP}.yml"
[ -f prometheus-rules.yml ] && cp prometheus-rules.yml "$BACKUP_DIR/prometheus-rules_${TIMESTAMP}.yml"

# Retention: Keep last 7 days
echo "Cleaning old backups..."
find "$BACKUP_DIR" -name "*.tar.gz" -mtime +7 -delete
find "$BACKUP_DIR" -name "*.yml" -mtime +7 -delete

echo "Backup completed: $BACKUP_DIR"
ls -lh "$BACKUP_DIR"/*${TIMESTAMP}*

Make executable and run:

chmod +x backup-grafana.sh
./backup-grafana.sh

Automate with cron:

# Edit crontab
crontab -e

# Add daily backup at 2 AM
0 2 * * * /home/user/backup-grafana.sh >> /var/log/grafana-backup.log 2>&1

Recovery script (restore-grafana.sh):

#!/bin/bash
# Restore Grafana from backup

if [ -z "$1" ]; then
    echo "Usage: $0 <backup-file>"
    echo "Example: $0 grafana_20241216_020000.tar.gz"
    exit 1
fi

BACKUP_FILE="$1"
BACKUP_DIR="/backups/grafana"

echo "Stopping containers..."
docker-compose down

echo "Restoring Grafana data..."
docker run --rm   -v grafana-storage:/data   -v "$BACKUP_DIR":/backup   alpine tar xzf "/backup/$BACKUP_FILE" -C /data

echo "Starting containers..."
docker-compose up -d

echo "Restore complete!"

Conclusion

You now have a complete, production-ready monitoring stack for your homelab. This guide has covered:

Core Capabilities:

  • Full monitoring stack with Grafana, Prometheus, and exporters
  • Comprehensive dashboards for system and container metrics
  • Intelligent alerting with multiple notification channels
  • AI-powered anomaly detection and capacity planning
  • Auto-remediation for common issues
  • Backup and recovery procedures

Your Next Steps:

This Week:

  1. Deploy the Docker Compose stack
  2. Create your first dashboard with CPU, memory, disk, and network panels
  3. Set up 5 essential alerts
  4. Configure at least one notification channel

This Month: 5. Add monitoring for your specific services (Pi-Hole, Jellyfin, etc.) 6. Implement log aggregation with Loki 7. Run the anomaly detection script to optimize alert thresholds 8. Set up automated backups

This Quarter: 9. Deploy Grafana Beyla for application auto-instrumentation 10. Implement predictive capacity planning 11. Create auto-remediation workflows for common issues 12. Share your dashboards with the community

Key Takeaways:

  • Start simple and expand gradually
  • Use community dashboards as templates
  • Optimize alert thresholds based on your actual usage patterns
  • Automate what can be automated
  • Document your setup and procedures
  • Regularly test backups and alerts

Resources:

Support: If you encounter issues or have questions, the homelab and Grafana communities are incredibly helpful. Share your setup, learn from others, and contribute back when you can.

Happy monitoring!

Guide Version: 1.0 Last Updated: December 2024 Target Audience: Homelab enthusiasts, self-hosted infrastructure operators Difficulty Level: Intermediate Estimated Time: 45 minutes + hands-on practice