Monitoring intermediate ⏱ 45 minutes

Alertmanager: Prometheus Alerting Guide

Configure alerts with Alertmanager. Routing, silences, and notification integrations.

By homelabbing.io • 2025-12-17
#alertmanager#prometheus#alerts#notifications#monitoring

Prometheus Alertmanager handles alerts sent by Prometheus, providing intelligent routing, grouping, silencing, and notification management. This guide will teach you how to set up a production-ready alerting system for your homelab.

What You’ll Learn

  • Installing and configuring Alertmanager
  • Creating effective alert rules in Prometheus
  • Setting up notification channels (Discord, Slack, Email, Gotify, PagerDuty)
  • Alert routing and grouping strategies
  • Silencing and inhibition rules
  • Alert templating for informative notifications
  • High availability setup
  • Best practices for homelab alerting

Prerequisites

  • Prometheus installed and running
  • Basic understanding of PromQL
  • Docker and Docker Compose
  • Access to notification services (Discord, Slack, etc.)

Why Alertmanager?

Prometheus evaluates alert rules and sends alerts to Alertmanager, which then:

  • Deduplicates: Removes duplicate alerts
  • Groups: Combines related alerts into single notifications
  • Routes: Sends alerts to appropriate channels based on labels
  • Silences: Temporarily mutes alerts during maintenance
  • Inhibits: Suppresses alerts based on other active alerts
  • Throttles: Rate limits notifications to prevent spam

Architecture Overview

Prometheus → Evaluates Rules → Alertmanager → Routes → Notification Channels
                                    ↓
                            Silences/Inhibitions

Complete Docker Compose Stack

Create a comprehensive monitoring stack with Prometheus, Alertmanager, and exporters.

docker-compose.yml

version: '3.8'

services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    restart: unless-stopped
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus/config:/etc/prometheus
      - ./prometheus/rules:/etc/prometheus/rules
      - prometheus-data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--storage.tsdb.retention.time=30d'
      - '--web.enable-lifecycle'
      - '--web.enable-admin-api'
      - '--web.console.libraries=/usr/share/prometheus/console_libraries'
      - '--web.console.templates=/usr/share/prometheus/consoles'
    networks:
      - monitoring
    depends_on:
      - alertmanager

  alertmanager:
    image: prom/alertmanager:latest
    container_name: alertmanager
    restart: unless-stopped
    ports:
      - "9093:9093"
    volumes:
      - ./alertmanager/config:/etc/alertmanager
      - alertmanager-data:/alertmanager
    command:
      - '--config.file=/etc/alertmanager/alertmanager.yml'
      - '--storage.path=/alertmanager'
      - '--web.external-url=http://YOUR-SERVER-IP:9093'
      - '--cluster.advertise-address=0.0.0.0:9093'
    networks:
      - monitoring

  node-exporter:
    image: prom/node-exporter:latest
    container_name: node-exporter
    restart: unless-stopped
    ports:
      - "9100:9100"
    command:
      - '--path.procfs=/host/proc'
      - '--path.sysfs=/host/sys'
      - '--path.rootfs=/rootfs'
      - '--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)'
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    networks:
      - monitoring

  cadvisor:
    image: gcr.io/cadvisor/cadvisor:latest
    container_name: cadvisor
    restart: unless-stopped
    ports:
      - "8080:8080"
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
      - /dev/disk/:/dev/disk:ro
    privileged: true
    devices:
      - /dev/kmsg
    networks:
      - monitoring

  blackbox-exporter:
    image: prom/blackbox-exporter:latest
    container_name: blackbox-exporter
    restart: unless-stopped
    ports:
      - "9115:9115"
    volumes:
      - ./blackbox/config:/etc/blackbox_exporter
    command:
      - '--config.file=/etc/blackbox_exporter/blackbox.yml'
    networks:
      - monitoring

volumes:
  prometheus-data:
  alertmanager-data:

networks:
  monitoring:
    driver: bridge

Prometheus Configuration

Create ./prometheus/config/prometheus.yml:

global:
  scrape_interval: 15s
  evaluation_interval: 15s
  external_labels:
    cluster: 'homelab'
    environment: 'production'
    alertmanager_cluster: 'homelab-primary'

# Alertmanager configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets: ['alertmanager:9093']
      timeout: 10s
      api_version: v2

# Load alert rules
rule_files:
  - 'rules/*.yml'

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']
        labels:
          instance: 'prometheus-server'

  - job_name: 'alertmanager'
    static_configs:
      - targets: ['alertmanager:9093']

  - job_name: 'node-exporter'
    static_configs:
      - targets: ['node-exporter:9100']
        labels:
          instance: 'homelab-server'
          host_type: 'physical'

  - job_name: 'cadvisor'
    static_configs:
      - targets: ['cadvisor:8080']
        labels:
          instance: 'homelab-server'

  - job_name: 'blackbox-http'
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
          - https://google.com
          - http://YOUR-SERVER-IP:9090
          - http://YOUR-SERVER-IP:3000
        labels:
          environment: 'production'
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox-exporter:9115

Alertmanager Configuration

Create ./alertmanager/config/alertmanager.yml:

global:
  resolve_timeout: 5m
  slack_api_url: 'https://hooks.slack.com/services/YOUR/WEBHOOK/URL'

# Templates for notifications
templates:
  - '/etc/alertmanager/*.tmpl'

# Route tree for alert routing
route:
  # Default receiver for all alerts
  receiver: 'default'

  # Group alerts by these labels
  group_by: ['alertname', 'cluster', 'service']

  # Wait before sending initial notification
  group_wait: 30s

  # Wait before sending notifications about new alerts in group
  group_interval: 5m

  # Wait before resending notification
  repeat_interval: 4h

  # Child routes for specific alert types
  routes:
    # Critical alerts go to Discord immediately
    - match:
        severity: critical
      receiver: 'discord-critical'
      group_wait: 10s
      repeat_interval: 1h
      continue: true

    # Infrastructure alerts
    - match:
        category: infrastructure
      receiver: 'slack-infra'
      group_by: ['alertname', 'instance']
      repeat_interval: 12h

    # Application alerts
    - match:
        category: application
      receiver: 'discord-apps'
      group_wait: 2m
      repeat_interval: 6h

    # Database alerts
    - match_re:
        service: '.*database.*|.*postgres.*|.*mysql.*'
      receiver: 'slack-database'
      repeat_interval: 2h

    # Backup alerts
    - match:
        category: backup
      receiver: 'email-admin'
      group_wait: 5m
      repeat_interval: 24h

    # Low priority alerts
    - match:
        severity: warning
      receiver: 'gotify-warnings'
      group_wait: 5m
      repeat_interval: 24h

# Inhibition rules to suppress redundant alerts
inhibit_rules:
  # Inhibit warning if critical alert is firing
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'instance']

  # Inhibit all alerts if host is down
  - source_match:
      alertname: 'InstanceDown'
    target_match_re:
      alertname: '.*'
    equal: ['instance']

  # Inhibit disk alerts if filesystem alert is firing
  - source_match:
      alertname: 'FilesystemFull'
    target_match:
      alertname: 'DiskSpaceWarning'
    equal: ['instance', 'device']

# Notification receivers
receivers:
  - name: 'default'
    discord_configs:
      - webhook_url: 'https://discord.com/api/webhooks/YOUR-WEBHOOK-ID/YOUR-WEBHOOK-TOKEN'
        title: 'Homelab Alert'
        message: '{{ template "discord.default.message" . }}'

  - name: 'discord-critical'
    discord_configs:
      - webhook_url: 'https://discord.com/api/webhooks/CRITICAL-WEBHOOK-ID/CRITICAL-WEBHOOK-TOKEN'
        title: 'CRITICAL ALERT'
        message: '{{ template "discord.critical.message" . }}'
        send_resolved: true

  - name: 'discord-apps'
    discord_configs:
      - webhook_url: 'https://discord.com/api/webhooks/APPS-WEBHOOK-ID/APPS-WEBHOOK-TOKEN'
        title: 'Application Alert'
        message: '{{ template "discord.default.message" . }}'

  - name: 'slack-infra'
    slack_configs:
      - channel: '#homelab-alerts'
        username: 'Alertmanager'
        icon_emoji: ':warning:'
        title: 'Infrastructure Alert'
        text: '{{ template "slack.default.text" . }}'
        send_resolved: true
        actions:
          - type: button
            text: 'View in Prometheus'
            url: 'http://YOUR-SERVER-IP:9090/alerts'
          - type: button
            text: 'Silence'
            url: 'http://YOUR-SERVER-IP:9093'

  - name: 'slack-database'
    slack_configs:
      - channel: '#database-alerts'
        username: 'Alertmanager'
        title: 'Database Alert'
        text: '{{ template "slack.default.text" . }}'

  - name: 'email-admin'
    email_configs:
      - to: '[email protected]'
        from: '[email protected]'
        smarthost: 'smtp.gmail.com:587'
        auth_username: '[email protected]'
        auth_password: 'your-app-password'
        headers:
          Subject: '[Homelab] {{ .GroupLabels.alertname }}'
        html: '{{ template "email.default.html" . }}'
        send_resolved: true

  - name: 'gotify-warnings'
    webhook_configs:
      - url: 'http://YOUR-SERVER-IP:8081/message?token=YOUR-GOTIFY-TOKEN'
        send_resolved: false
        http_config:
          basic_auth:
            username: 'alertmanager'
            password: 'your-password'

Alert Rules

Create alert rules for common homelab scenarios.

System Alerts

Create ./prometheus/rules/system-alerts.yml:

groups:
  - name: system_alerts
    interval: 30s
    rules:
      # Instance down
      - alert: InstanceDown
        expr: up == 0
        for: 2m
        labels:
          severity: critical
          category: infrastructure
        annotations:
          summary: "Instance {{ $labels.instance }} is down"
          description: "{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 2 minutes."
          runbook_url: "https://homelab.example.com/runbooks/instance-down"

      # High CPU usage
      - alert: HighCPUUsage
        expr: 100 - (avg by (instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
        for: 5m
        labels:
          severity: warning
          category: infrastructure
        annotations:
          summary: "High CPU usage on {{ $labels.instance }}"
          description: "CPU usage is {{ $value | humanize }}% on {{ $labels.instance }}"

      - alert: CriticalCPUUsage
        expr: 100 - (avg by (instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 95
        for: 3m
        labels:
          severity: critical
          category: infrastructure
        annotations:
          summary: "Critical CPU usage on {{ $labels.instance }}"
          description: "CPU usage is {{ $value | humanize }}% on {{ $labels.instance }}"

      # High memory usage
      - alert: HighMemoryUsage
        expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100 > 80
        for: 5m
        labels:
          severity: warning
          category: infrastructure
        annotations:
          summary: "High memory usage on {{ $labels.instance }}"
          description: "Memory usage is {{ $value | humanize }}% on {{ $labels.instance }}"

      - alert: CriticalMemoryUsage
        expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100 > 95
        for: 2m
        labels:
          severity: critical
          category: infrastructure
        annotations:
          summary: "Critical memory usage on {{ $labels.instance }}"
          description: "Memory usage is {{ $value | humanize }}% on {{ $labels.instance }}"

      # Disk space
      - alert: DiskSpaceWarning
        expr: (1 - (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"})) * 100 > 75
        for: 10m
        labels:
          severity: warning
          category: infrastructure
        annotations:
          summary: "Disk space running low on {{ $labels.instance }}"
          description: "Disk usage is {{ $value | humanize }}% on {{ $labels.instance }} mountpoint {{ $labels.mountpoint }}"

      - alert: FilesystemFull
        expr: (1 - (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"})) * 100 > 90
        for: 5m
        labels:
          severity: critical
          category: infrastructure
        annotations:
          summary: "Filesystem almost full on {{ $labels.instance }}"
          description: "Filesystem usage is {{ $value | humanize }}% on {{ $labels.instance }} mountpoint {{ $labels.mountpoint }}"

      # Disk will fill in 4 hours (predictive)
      - alert: DiskWillFillSoon
        expr: predict_linear(node_filesystem_avail_bytes{mountpoint="/"}[1h], 4 * 3600) < 0
        for: 5m
        labels:
          severity: warning
          category: infrastructure
        annotations:
          summary: "Disk will fill in approximately 4 hours"
          description: "Filesystem on {{ $labels.instance }} is predicted to fill within 4 hours based on current growth rate"

      # High I/O wait
      - alert: HighIOWait
        expr: avg by (instance) (irate(node_cpu_seconds_total{mode="iowait"}[5m])) * 100 > 30
        for: 5m
        labels:
          severity: warning
          category: infrastructure
        annotations:
          summary: "High I/O wait on {{ $labels.instance }}"
          description: "I/O wait is {{ $value | humanize }}% on {{ $labels.instance }}"

      # High load average
      - alert: HighLoadAverage
        expr: node_load15 / count(node_cpu_seconds_total{mode="idle"}) without (cpu, mode) > 2
        for: 5m
        labels:
          severity: warning
          category: infrastructure
        annotations:
          summary: "High load average on {{ $labels.instance }}"
          description: "Load average is {{ $value | humanize }} on {{ $labels.instance }}"

      # Network errors
      - alert: NetworkInterfaceErrors
        expr: rate(node_network_receive_errs_total[5m]) > 10
        for: 2m
        labels:
          severity: warning
          category: infrastructure
        annotations:
          summary: "Network interface errors on {{ $labels.instance }}"
          description: "Interface {{ $labels.device }} on {{ $labels.instance }} has {{ $value | humanize }} receive errors/sec"

Container Alerts

Create ./prometheus/rules/container-alerts.yml:

groups:
  - name: container_alerts
    interval: 30s
    rules:
      # Container down
      - alert: ContainerDown
        expr: absent(container_last_seen{name!=""})
        for: 2m
        labels:
          severity: warning
          category: application
        annotations:
          summary: "Container {{ $labels.name }} is down"
          description: "Container {{ $labels.name }} has been down for more than 2 minutes"

      # High container CPU
      - alert: ContainerHighCPU
        expr: sum(rate(container_cpu_usage_seconds_total{name!=""}[5m])) by (name, instance) * 100 > 80
        for: 5m
        labels:
          severity: warning
          category: application
        annotations:
          summary: "High CPU usage in container {{ $labels.name }}"
          description: "Container {{ $labels.name }} on {{ $labels.instance }} is using {{ $value | humanize }}% CPU"

      # High container memory
      - alert: ContainerHighMemory
        expr: (container_memory_usage_bytes{name!=""} / container_spec_memory_limit_bytes{name!=""}) * 100 > 80
        for: 5m
        labels:
          severity: warning
          category: application
        annotations:
          summary: "High memory usage in container {{ $labels.name }}"
          description: "Container {{ $labels.name }} is using {{ $value | humanize }}% of its memory limit"

      # Container restarting
      - alert: ContainerRestarting
        expr: rate(container_start_time_seconds{name!=""}[15m]) > 0.01
        for: 5m
        labels:
          severity: warning
          category: application
        annotations:
          summary: "Container {{ $labels.name }} is restarting frequently"
          description: "Container {{ $labels.name }} has restarted {{ $value | humanize }} times in the last 15 minutes"

      # Container OOM killed
      - alert: ContainerOOMKilled
        expr: sum by (name, instance) (container_oom_events_total) > 0
        for: 1m
        labels:
          severity: critical
          category: application
        annotations:
          summary: "Container {{ $labels.name }} killed by OOM"
          description: "Container {{ $labels.name }} on {{ $labels.instance }} was killed by OOM killer"

Service Availability Alerts

Create ./prometheus/rules/service-alerts.yml:

groups:
  - name: service_alerts
    interval: 30s
    rules:
      # HTTP probe failed
      - alert: ServiceDown
        expr: probe_success == 0
        for: 2m
        labels:
          severity: critical
          category: application
        annotations:
          summary: "Service {{ $labels.instance }} is down"
          description: "HTTP probe failed for {{ $labels.instance }} for more than 2 minutes"

      # SSL certificate expiring
      - alert: SSLCertificateExpiringSoon
        expr: probe_ssl_earliest_cert_expiry - time() < 86400 * 7
        for: 1h
        labels:
          severity: warning
          category: infrastructure
        annotations:
          summary: "SSL certificate expiring soon for {{ $labels.instance }}"
          description: "SSL certificate for {{ $labels.instance }} expires in {{ $value | humanizeDuration }}"

      - alert: SSLCertificateExpiring
        expr: probe_ssl_earliest_cert_expiry - time() < 86400 * 3
        for: 1h
        labels:
          severity: critical
          category: infrastructure
        annotations:
          summary: "SSL certificate expiring very soon for {{ $labels.instance }}"
          description: "SSL certificate for {{ $labels.instance }} expires in {{ $value | humanizeDuration }}"

      # Slow HTTP response
      - alert: SlowHTTPResponse
        expr: probe_duration_seconds > 5
        for: 5m
        labels:
          severity: warning
          category: application
        annotations:
          summary: "Slow HTTP response for {{ $labels.instance }}"
          description: "HTTP probe for {{ $labels.instance }} took {{ $value | humanize }} seconds"

Prometheus Self-Monitoring

Create ./prometheus/rules/prometheus-alerts.yml:

groups:
  - name: prometheus_alerts
    interval: 30s
    rules:
      # Prometheus configuration reload failed
      - alert: PrometheusConfigReloadFailed
        expr: prometheus_config_last_reload_successful == 0
        for: 5m
        labels:
          severity: critical
          category: infrastructure
        annotations:
          summary: "Prometheus configuration reload failed"
          description: "Prometheus configuration reload has been failing for 5 minutes"

      # Too many restarts
      - alert: PrometheusTooManyRestarts
        expr: changes(process_start_time_seconds{job="prometheus"}[15m]) > 2
        for: 5m
        labels:
          severity: warning
          category: infrastructure
        annotations:
          summary: "Prometheus restarting frequently"
          description: "Prometheus has restarted {{ $value }} times in the last 15 minutes"

      # TSDB compaction taking too long
      - alert: PrometheusTSDBCompactionsTakingTooLong
        expr: rate(prometheus_tsdb_compactions_total[5m]) < 0.0001
        for: 1h
        labels:
          severity: warning
          category: infrastructure
        annotations:
          summary: "Prometheus TSDB compactions taking too long"
          description: "TSDB compactions are taking longer than expected"

      # Target scrape failures
      - alert: PrometheusTargetScrapeFailed
        expr: up == 0
        for: 5m
        labels:
          severity: warning
          category: infrastructure
        annotations:
          summary: "Prometheus target scrape failed"
          description: "Target {{ $labels.instance }} scrape has been failing for 5 minutes"

Blackbox Exporter Configuration

Create ./blackbox/config/blackbox.yml:

modules:
  http_2xx:
    prober: http
    timeout: 5s
    http:
      valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
      valid_status_codes: []
      method: GET
      preferred_ip_protocol: "ip4"
      follow_redirects: true
      fail_if_ssl: false
      fail_if_not_ssl: false

  http_post_2xx:
    prober: http
    http:
      method: POST
      headers:
        Content-Type: application/json

  tcp_connect:
    prober: tcp
    timeout: 5s

  icmp:
    prober: icmp
    timeout: 5s
    icmp:
      preferred_ip_protocol: "ip4"

Notification Templates

Create custom templates for better-looking notifications.

Discord Template

Create ./alertmanager/config/discord.tmpl:

{{ define "discord.default.message" }}
{{ range .Alerts }}
**Alert:** {{ .Labels.alertname }}
**Severity:** {{ .Labels.severity }}
**Instance:** {{ .Labels.instance }}
**Description:** {{ .Annotations.description }}
{{ if .Annotations.runbook_url }}**Runbook:** {{ .Annotations.runbook_url }}{{ end }}
**Status:** {{ .Status }}
{{ if eq .Status "firing" }}**Started:** {{ .StartsAt.Format "2006-01-02 15:04:05" }}{{ end }}
{{ if eq .Status "resolved" }}**Resolved:** {{ .EndsAt.Format "2006-01-02 15:04:05" }}{{ end }}
{{ end }}
{{ end }}

{{ define "discord.critical.message" }}
:rotating_light: **CRITICAL ALERT** :rotating_light:
{{ range .Alerts }}
**Alert:** {{ .Labels.alertname }}
**Instance:** {{ .Labels.instance }}
**Description:** {{ .Annotations.description }}
**Started:** {{ .StartsAt.Format "2006-01-02 15:04:05" }}
{{ if .Annotations.runbook_url }}
:book: **Runbook:** {{ .Annotations.runbook_url }}
{{ end }}
{{ end }}
@everyone
{{ end }}

Slack Template

Create ./alertmanager/config/slack.tmpl:

{{ define "slack.default.text" }}
{{ range .Alerts }}
*Alert:* {{ .Labels.alertname }}
*Severity:* {{ .Labels.severity }}
*Instance:* {{ .Labels.instance }}
*Description:* {{ .Annotations.description }}
{{ if .Annotations.runbook_url }}*Runbook:* {{ .Annotations.runbook_url }}{{ end }}
*Status:* {{ .Status }}
{{ if eq .Status "firing" }}*Started:* {{ .StartsAt.Format "2006-01-02 15:04:05" }}{{ end }}
{{ if eq .Status "resolved" }}*Resolved:* {{ .EndsAt.Format "2006-01-02 15:04:05" }}{{ end }}
{{ end }}
{{ end }}

Email Template

Create ./alertmanager/config/email.tmpl:

{{ define "email.default.html" }}
<!DOCTYPE html>
<html>
<head>
  <style>
    body { font-family: Arial, sans-serif; }
    .alert { border: 2px solid #ff6b6b; padding: 15px; margin: 10px 0; border-radius: 5px; }
    .critical { background-color: #ffebee; }
    .warning { border-color: #ffa726; background-color: #fff3e0; }
    .resolved { border-color: #66bb6a; background-color: #e8f5e9; }
    h2 { color: #333; }
    .label { font-weight: bold; }
  </style>
</head>
<body>
  <h2>Homelab Alert Notification</h2>
  {{ range .Alerts }}
  <div class="alert {{ .Labels.severity }}">
    <p><span class="label">Alert:</span> {{ .Labels.alertname }}</p>
    <p><span class="label">Severity:</span> {{ .Labels.severity }}</p>
    <p><span class="label">Instance:</span> {{ .Labels.instance }}</p>
    <p><span class="label">Description:</span> {{ .Annotations.description }}</p>
    {{ if .Annotations.runbook_url }}
    <p><span class="label">Runbook:</span> <a href="{{ .Annotations.runbook_url }}">{{ .Annotations.runbook_url }}</a></p>
    {{ end }}
    <p><span class="label">Status:</span> {{ .Status }}</p>
    {{ if eq .Status "firing" }}
    <p><span class="label">Started:</span> {{ .StartsAt.Format "2006-01-02 15:04:05" }}</p>
    {{ end }}
    {{ if eq .Status "resolved" }}
    <p><span class="label">Resolved:</span> {{ .EndsAt.Format "2006-01-02 15:04:05" }}</p>
    {{ end }}
  </div>
  {{ end }}
</body>
</html>
{{ end }}

Update alertmanager.yml to include templates:

templates:
  - '/etc/alertmanager/*.tmpl'

Mount templates in docker-compose:

volumes:
  - ./alertmanager/config:/etc/alertmanager

Setting Up Notification Channels

Discord Webhook

  1. Go to your Discord server → Server Settings → Integrations
  2. Create Webhook → Choose channel → Copy webhook URL
  3. Add to alertmanager.yml:
receivers:
  - name: 'discord'
    discord_configs:
      - webhook_url: 'https://discord.com/api/webhooks/1234567890/abcdefghijklmnop'
        title: '{{ template "discord.default.title" . }}'
        message: '{{ template "discord.default.message" . }}'

Slack Webhook

  1. Go to https://api.slack.com/apps
  2. Create New App → Choose workspace
  3. Incoming Webhooks → Activate → Add New Webhook
  4. Copy webhook URL

Email with Gmail

  1. Enable 2-factor authentication in Gmail
  2. Create App Password: Google Account → Security → 2-Step Verification → App passwords
  3. Use in alertmanager:
email_configs:
  - to: '[email protected]'
    from: '[email protected]'
    smarthost: 'smtp.gmail.com:587'
    auth_username: '[email protected]'
    auth_password: 'xxxx xxxx xxxx xxxx'

Gotify

Self-hosted notification service:

gotify:
  image: gotify/server:latest
  ports:
    - "8081:80"
  volumes:
    - gotify-data:/app/data

Webhook configuration:

webhook_configs:
  - url: 'http://gotify:80/message?token=YOUR-GOTIFY-TOKEN'
    send_resolved: true

Testing Alerts

Manually Trigger Alert

Create a test alert:

curl -X POST http://YOUR-SERVER-IP:9093/api/v1/alerts -d '[
  {
    "labels": {
      "alertname": "TestAlert",
      "severity": "warning",
      "instance": "test-instance"
    },
    "annotations": {
      "summary": "Test alert from curl",
      "description": "This is a test alert to verify Alertmanager configuration"
    },
    "startsAt": "2025-12-17T00:00:00.000Z"
  }
]'

Verify Alert Rules

Check if Prometheus loaded rules:

curl http://YOUR-SERVER-IP:9090/api/v1/rules

Simulate High CPU

docker run --rm -it progrium/stress --cpu 8 --timeout 300s

This should trigger the HighCPUUsage alert.

Silencing Alerts

Via Web UI

  1. Navigate to http://YOUR-SERVER-IP:9093
  2. Click “Silences” → “New Silence”
  3. Add matchers (e.g., alertname=DiskSpaceWarning)
  4. Set duration
  5. Add comment
  6. Create

Via API

curl -X POST http://YOUR-SERVER-IP:9093/api/v2/silences   -H 'Content-Type: application/json'   -d '{
    "matchers": [
      {
        "name": "alertname",
        "value": "DiskSpaceWarning",
        "isRegex": false
      }
    ],
    "startsAt": "2025-12-17T00:00:00.000Z",
    "endsAt": "2025-12-17T02:00:00.000Z",
    "comment": "Scheduled maintenance",
    "createdBy": "admin"
  }'

Via amtool

Install amtool:

wget https://github.com/prometheus/alertmanager/releases/download/v0.27.0/alertmanager-0.27.0.linux-amd64.tar.gz
tar xvf alertmanager-0.27.0.linux-amd64.tar.gz
sudo cp alertmanager-0.27.0.linux-amd64/amtool /usr/local/bin/

Create silence:

amtool --alertmanager.url=http://YOUR-SERVER-IP:9093 silence add   alertname=DiskSpaceWarning   --duration=2h   --comment="Planned disk cleanup"

List silences:

amtool --alertmanager.url=http://YOUR-SERVER-IP:9093 silence query

High Availability Setup

For production homelabs, run multiple Alertmanager instances.

docker-compose.yml with HA

alertmanager-1:
  image: prom/alertmanager:latest
  container_name: alertmanager-1
  command:
    - '--config.file=/etc/alertmanager/alertmanager.yml'
    - '--storage.path=/alertmanager'
    - '--cluster.listen-address=0.0.0.0:9094'
    - '--cluster.peer=alertmanager-2:9094'
    - '--cluster.peer=alertmanager-3:9094'
  ports:
    - "9093:9093"
  volumes:
    - ./alertmanager/config:/etc/alertmanager
    - alertmanager-1-data:/alertmanager

alertmanager-2:
  image: prom/alertmanager:latest
  container_name: alertmanager-2
  command:
    - '--config.file=/etc/alertmanager/alertmanager.yml'
    - '--storage.path=/alertmanager'
    - '--cluster.listen-address=0.0.0.0:9094'
    - '--cluster.peer=alertmanager-1:9094'
    - '--cluster.peer=alertmanager-3:9094'
  ports:
    - "9095:9093"
  volumes:
    - ./alertmanager/config:/etc/alertmanager
    - alertmanager-2-data:/alertmanager

alertmanager-3:
  image: prom/alertmanager:latest
  container_name: alertmanager-3
  command:
    - '--config.file=/etc/alertmanager/alertmanager.yml'
    - '--storage.path=/alertmanager'
    - '--cluster.listen-address=0.0.0.0:9094'
    - '--cluster.peer=alertmanager-1:9094'
    - '--cluster.peer=alertmanager-2:9094'
  ports:
    - "9096:9093"
  volumes:
    - ./alertmanager/config:/etc/alertmanager
    - alertmanager-3-data:/alertmanager

Update Prometheus to use all instances:

alerting:
  alertmanagers:
    - static_configs:
        - targets:
          - 'alertmanager-1:9093'
          - 'alertmanager-2:9093'
          - 'alertmanager-3:9093'

Best Practices

Alert Design

  1. Clear Names: Use descriptive alert names
  2. Proper Severity: Critical vs Warning appropriately
  3. Actionable: Alerts should require action
  4. Context: Include runbook URLs
  5. Avoid Noise: Don’t alert on expected behavior

Routing Strategy

  1. Route by Severity: Critical alerts to immediate channels
  2. Route by Team: Different services to different teams
  3. Route by Time: Different routing during office hours
  4. Escalation: Increase urgency if not acknowledged

Notification Hygiene

  1. Rate Limiting: Use group_interval to prevent spam
  2. Grouping: Group related alerts
  3. Inhibition: Suppress redundant alerts
  4. Silencing: Use during maintenance windows

Alert Fatigue Prevention

  1. Tune Thresholds: Adjust based on actual usage
  2. Increase Durations: Don’t alert on temporary spikes
  3. Use Inhibition: Prevent alert cascades
  4. Regular Review: Remove or adjust noisy alerts

Troubleshooting

Alerts Not Firing

Check Prometheus rules:

curl http://YOUR-SERVER-IP:9090/api/v1/rules | jq '.data.groups[].rules[] | select(.state=="firing")'

Notifications Not Received

Check Alertmanager logs:

docker logs alertmanager

Test notification manually:

amtool alert add alertname=test severity=warning

Silences Not Working

List active silences:

amtool silence query

Check matcher labels match alert labels exactly.

Monitoring Alertmanager

Add these metrics to your dashboard:

# Active alerts
sum(ALERTS{alertstate="firing"})

# Notifications sent
rate(alertmanager_notifications_total[5m])

# Notification failures
rate(alertmanager_notifications_failed_total[5m])

# Silenced alerts
alertmanager_silences

Next Steps

  • Integrate with PagerDuty for on-call rotation
  • Set up webhook receivers for custom integrations
  • Implement alert runbooks and documentation
  • Create time-based routing for on-call schedules
  • Set up mobile push notifications via Pushover
  • Integrate with ticketing systems (Jira, ServiceNow)

Additional Resources

With a properly configured Alertmanager, you’ll have intelligent, actionable alerts that keep you informed of issues without overwhelming you with notifications. Start with basic rules and refine based on your homelab’s specific needs.