Loki + Grafana: Log Aggregation for Homelabs

Introduction

Logs are the lifeblood of troubleshooting, but traditional log aggregation solutions like Elasticsearch are resource-heavy, complex to operate, and expensive to scale. Grafana Labs created Loki as a “Prometheus for logs”—a lightweight, cost-effective alternative that uses the same label-based approach as Prometheus.

Loki has transformed how homelabbers handle logs: instead of a 20GB Elasticsearch cluster, you can run Loki on 512MB of RAM with better query performance for typical use cases. This guide will teach you everything from basic deployment to advanced multi-tenant configurations.

What you’ll learn:

• Why Loki’s architecture is perfect for homelabs
• Deploying Loki with Docker for multi-container log collection
• Collecting logs with Grafana Alloy (successor to Promtail)
• Mastering LogQL for powerful log queries
• Configuring retention policies and compression
• Building effective log dashboards in Grafana
• Advanced features: multi-tenancy, federation, object storage

Prerequisites:

• Docker and Docker Compose installed
• Basic understanding of containers and logging
• Grafana instance (or willingness to deploy)
• 1GB+ RAM, 10GB+ disk for production use

---

Why Loki?

The Log Aggregation Problem

Traditional solutions index everything:

Elasticsearch/OpenSearch:

• Indexes full text of every log line
• High RAM usage (5-10GB minimum for small deployments)
• Complex cluster management
• Expensive storage costs

Splunk:

• Commercial licensing ($150+/GB/year)
• Feature-rich but cost-prohibitive for homelabs

Loki’s approach: Index labels, not content

How Loki is Different

Loki only indexes metadata (labels):

• Container name, namespace, job, host
• Custom labels from log processing
• NOT the log message content

This means:

• 10x lower storage costs
• 5-10x less RAM usage
• Much simpler architecture
• Slower full-text search (but you rarely need it)

Architecture Comparison

When to use Loki:

• Homelab and small-scale production (under 100GB logs/day)
• Structured logs with known labels
• Cost-sensitive environments
• Kubernetes/Docker infrastructure

When to use Elasticsearch:

• Enterprise full-text search requirements
• Massive scale (TB/day)
• Complex data analytics beyond logs
• Compliance requirements for advanced features

---

Part 1: Basic Deployment

Complete Stack with Docker Compose

We’ll deploy Loki + Grafana Alloy (log collector) + Grafana (visualization).

Create project directory:

mkdir -p ~/loki-stack
cd ~/loki-stack

Create docker-compose.yml:

version: '3.8'

services:
  # Grafana Loki - Log aggregation database
  loki:
    image: grafana/loki:latest
    container_name: loki
    ports:
      - "3100:3100"
    volumes:
      - ./loki-config.yml:/etc/loki/local-config.yaml
      - loki-data:/loki
    command: -config.file=/etc/loki/local-config.yaml
    networks:
      - logging
    restart: unless-stopped

  # Grafana Alloy - Log collector (successor to Promtail)
  alloy:
    image: grafana/alloy:latest
    container_name: alloy
    volumes:
      - ./alloy-config.river:/etc/alloy/config.river
      - /var/log:/var/log:ro
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
    command:
      - run
      - /etc/alloy/config.river
      - --server.http.listen-addr=0.0.0.0:12345
    ports:
      - "12345:12345"
    depends_on:
      - loki
    networks:
      - logging
    restart: unless-stopped

  # Grafana - Visualization and dashboards
  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin
      - GF_USERS_ALLOW_SIGN_UP=false
      - GF_PATHS_PROVISIONING=/etc/grafana/provisioning
    volumes:
      - grafana-data:/var/lib/grafana
      - ./grafana-datasources.yml:/etc/grafana/provisioning/datasources/datasources.yml
    depends_on:
      - loki
    networks:
      - logging
    restart: unless-stopped

volumes:
  loki-data:
  grafana-data:

networks:
  logging:
    driver: bridge

Create loki-config.yml:

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096

common:
  instance_addr: 127.0.0.1
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2020-10-24
      store: tsdb
      object_store: filesystem
      schema: v12
      index:
        prefix: index_
        period: 24h

storage_config:
  tsdb_shipper:
    active_index_directory: /loki/tsdb-index
    cache_location: /loki/tsdb-cache

compactor:
  working_directory: /loki/compactor
  compaction_interval: 10m
  retention_enabled: true
  retention_delete_delay: 2h
  retention_delete_worker_count: 150

limits_config:
  retention_period: 744h  # 31 days
  enforce_metric_name: false
  reject_old_samples: true
  reject_old_samples_max_age: 168h
  ingestion_rate_mb: 16
  ingestion_burst_size_mb: 32
  max_query_series: 10000
  max_query_parallelism: 32

chunk_store_config:
  max_look_back_period: 0s

table_manager:
  retention_deletes_enabled: true
  retention_period: 744h

ruler:
  storage:
    type: local
    local:
      directory: /loki/rules
  rule_path: /loki/rules-temp
  ring:
    kvstore:
      store: inmemory
  enable_api: true

Create alloy-config.river:

// Discover Docker containers
discovery.docker "containers" {
  host = "unix:///var/run/docker.sock"
}

// Collect Docker container logs
loki.source.docker "containers" {
  host       = "unix:///var/run/docker.sock"
  targets    = discovery.docker.containers.targets
  forward_to = [loki.process.add_metadata.receiver]
}

// Process and enrich logs
loki.process "add_metadata" {
  forward_to = [loki.write.loki_endpoint.receiver]

  // Add hostname
  stage.static_labels {
    values = {
      hostname = env("HOSTNAME"),
    }
  }

  // Extract log level
  stage.regex {
    expression = "(?i)level=(?P<level>\w+)"
  }

  stage.labels {
    values = {
      level = "",
    }
  }

  // Drop noisy logs
  stage.match {
    selector = "{container_name="traefik"}"

    stage.drop {
      expression = ".*healthcheck.*"
    }
  }
}

// Send to Loki
loki.write "loki_endpoint" {
  endpoint {
    url = "http://loki:3100/loki/api/v1/push"
  }
}

// System logs from journald
loki.source.journal "system" {
  forward_to = [loki.write.loki_endpoint.receiver]
  labels = {
    job = "systemd-journal",
  }
}

Create grafana-datasources.yml:

apiVersion: 1

datasources:
  - name: Loki
    type: loki
    access: proxy
    url: http://loki:3100
    isDefault: true
    jsonData:
      maxLines: 1000
      derivedFields:
        # Link to traces (if using Tempo)
        - datasourceUid: tempo
          matcherRegex: "trace_id=(\w+)"
          name: TraceID
          url: "$${__value.raw}"
    editable: true

Start the stack:

docker compose up -d

Verify deployment:

# Check containers
docker compose ps

# Test Loki API
curl http://localhost:3100/ready

# Check Alloy status
curl http://localhost:12345/

Access Grafana:

http://localhost:3000
Login: admin / admin

---

Part 2: LogQL Query Language

Basic Queries

View all logs:

{job="varlogs"}

Filter by container:

{container_name="nginx"}

Multiple labels:

{container_name="api", environment="production"}

Label matchers:

• = : Exact match
• != : Not equal
• =~ : Regex match
• !~ : Regex not match

{container_name=~"api.*"}
{environment!="test"}

Line Filters

Contains text:

{container_name="nginx"} |= "error"

Does not contain:

{container_name="nginx"} != "health"

Regex filter:

{container_name="api"} |~ "error|warn|fatal"

Case-insensitive:

{container_name="app"} |~ "(?i)error"

Multiple filters (AND):

{container_name="nginx"} |= "error" != "timeout"

Parsing Logs

JSON parsing:

{container_name="api"} | json

Extract specific JSON fields:

{container_name="api"} | json status="response.status", duration="response.time"

Logfmt parsing:

{container_name="app"} | logfmt

Regex parsing:

{container_name="nginx"}
| regexp `(?P<ip>S+) - - [(?P<timestamp>[^]]+)] "(?P<method>w+) (?P<path>S+)"`

Pattern parsing:

{container_name="app"}
| pattern `<timestamp> <level> <message>`

Filters After Parsing

Filter on parsed fields:

{container_name="api"}
| json
| status >= 400

Multiple conditions:

{container_name="api"}
| json
| status >= 400
| duration > 1000
| method = "POST"

Metric Queries

Count log lines:

count_over_time({container_name="nginx"}[5m])

Rate of logs per second:

rate({container_name="api"}[1m])

Error rate:

sum(rate({container_name="api"} |= "error" [5m]))

Error percentage:

sum(rate({container_name="api"} |= "error" [5m]))
/
sum(rate({container_name="api"}[5m])) * 100

Average response time:

avg_over_time({container_name="api"}
  | json
  | unwrap duration [5m])

95th percentile:

quantile_over_time(0.95, {container_name="api"}
  | json
  | unwrap duration [5m])

Top 10 endpoints by request count:

topk(10, sum by (path) (
  count_over_time({container_name="nginx"}
    | regexp `(?P<path>/S+)` [1h])
))

Aggregations

Group by label:

sum by (container_name) (
  count_over_time({job="docker"}[5m])
)

Count unique values:

count(count by (status) (
  {container_name="api"} | json
))

Aggregate over time:

avg_over_time({container_name="api"}
  | json
  | unwrap response_time [5m]
) by (endpoint)

---

Part 3: Log Collection Strategies

Docker Container Logs

Basic container collection:

discovery.docker "all" {
  host = "unix:///var/run/docker.sock"
}

loki.source.docker "containers" {
  host       = "unix:///var/run/docker.sock"
  targets    = discovery.docker.all.targets
  forward_to = [loki.write.loki.receiver]
}

Selective container collection:

discovery.docker "labeled" {
  host = "unix:///var/run/docker.sock"

  filter {
    name   = "label"
    values = ["logging=loki"]
  }
}

loki.source.docker "selected" {
  host       = "unix:///var/run/docker.sock"
  targets    = discovery.docker.labeled.targets
  forward_to = [loki.write.loki.receiver]
}

Label containers:

services:
  my-app:
    image: my-app:latest
    labels:
      - "logging=loki"

File-Based Log Collection

Tail log files:

local.file_match "logs" {
  path_targets = [{
    __path__ = "/var/log/*.log",
    job      = "varlogs",
  }]
}

loki.source.file "files" {
  targets    = local.file_match.logs.targets
  forward_to = [loki.write.loki.receiver]
}

Multiple file patterns:

local.file_match "app_logs" {
  path_targets = [
    {__path__ = "/var/log/app/*.log", app = "backend"},
    {__path__ = "/var/log/nginx/*.log", app = "nginx"},
    {__path__ = "/var/log/postgres/*.log", app = "database"},
  ]
}

Syslog Collection

Receive syslog:

loki.source.syslog "syslog" {
  listener {
    address  = "0.0.0.0:1514"
    protocol = "tcp"
    labels   = {
      job = "syslog",
    }
  }
  forward_to = [loki.write.loki.receiver]
}

Configure syslog forwarding:

# /etc/rsyslog.d/loki.conf
*.* @@alloy-server:1514

Kubernetes Logs

Pod log collection:

discovery.kubernetes "pods" {
  role = "pod"
}

loki.source.kubernetes "pods" {
  targets    = discovery.kubernetes.pods.targets
  forward_to = [loki.process.k8s.receiver]
}

loki.process "k8s" {
  forward_to = [loki.write.loki.receiver]

  stage.static_labels {
    values = {
      cluster = "homelab",
    }
  }
}

---

Part 4: Grafana Dashboards

Exploring Logs in Grafana

1. Navigate to Explore:

• Click Explore icon (compass) in left sidebar
• Select Loki data source

2. Build queries:

• Use query builder or write LogQL directly
• Add filters, parsers, and metrics
• Visualize as table or time series

3. Live tail:

• Enable Live toggle for real-time streaming
• See logs as they arrive
• Great for troubleshooting active issues

Building Log Dashboard

Create new dashboard:

1. Dashboards → New Dashboard → Add visualization
2. Select Loki data source
3. Enter LogQL query

Panel 1: Log volume over time

sum by (container_name) (
  count_over_time({job="docker"}[1m])
)

Settings:

• Visualization: Time series
• Legend: {{container_name}}
• Y-axis: Logs/min

Panel 2: Recent error logs

{job="docker"} |~ "(?i)error|exception|fatal"

Settings:

• Visualization: Logs
• Show time
• Enable wrapping
• Highlight search terms

Panel 3: Response time percentiles

quantile_over_time(0.50, {container_name="api"}
  | json
  | unwrap response_time [5m]) or
quantile_over_time(0.95, {container_name="api"}
  | json
  | unwrap response_time [5m]) or
quantile_over_time(0.99, {container_name="api"}
  | json
  | unwrap response_time [5m])

Panel 4: Top error messages

topk(10, sum by (message) (
  count_over_time(
    {container_name="api"}
    | json
    | level="error" [1h]
  )
))

Settings:

• Visualization: Bar chart
• Orientation: Horizontal

Panel 5: Log level distribution

sum by (level) (
  count_over_time({job="docker"}
    | json [5m]
  )
)

Settings:

• Visualization: Pie chart
• Legend: Show percentage

Variables for Dynamic Dashboards

Create container variable:

1. Dashboard settings → Variables → New variable
2. Configure:
   • Name: container
   • Type: Query
   • Data source: Loki
   • Query: label_values(container_name)
   • Multi-value: true
   • Include All: true

Use in queries:

{container_name=~"$container"}

Create time range variable:

Name: time_range
Type: Interval
Values: 1m,5m,15m,1h,24h

Use in queries:

rate({container_name="api"}[$time_range])

---

Part 5: Retention and Storage

Configuring Retention

In loki-config.yml:

limits_config:
  retention_period: 744h  # 31 days

compactor:
  retention_enabled: true
  retention_delete_delay: 2h  # Wait 2h before deleting
  retention_delete_worker_count: 150
  compaction_interval: 10m

table_manager:
  retention_deletes_enabled: true
  retention_period: 744h

Per-stream retention overrides:

limits_config:
  retention_period: 168h  # Default: 7 days

  # Override for specific streams
  retention_stream:
    - selector: '{container_name="critical-app"}'
      priority: 1
      retention_period: 8760h  # 1 year

    - selector: '{level="debug"}'
      priority: 2
      retention_period: 24h  # 1 day

Storage Optimization

Enable compression:

storage_config:
  tsdb_shipper:
    active_index_directory: /loki/tsdb-index
    cache_location: /loki/tsdb-cache

  boltdb_shipper:
    compressor: snappy  # or gzip, zstd

Chunk encoding:

chunk_store_config:
  chunk_encoding: snappy

Typical storage usage:

• Raw logs: 1GB/day
• Compressed in Loki: 100-200MB/day
• 5-10x compression ratio

Object Storage Backend

S3-compatible storage (MinIO, AWS S3):

storage_config:
  aws:
    s3: s3://access_key:secret_key@region/bucket_name
    s3forcepathstyle: true  # For MinIO

  tsdb_shipper:
    active_index_directory: /loki/tsdb-index
    cache_location: /loki/tsdb-cache
    shared_store: s3

schema_config:
  configs:
    - from: 2020-10-24
      store: tsdb
      object_store: s3
      schema: v12
      index:
        prefix: loki_index_
        period: 24h

MinIO docker-compose.yml addition:

services:
  minio:
    image: minio/minio:latest
    container_name: minio
    ports:
      - "9000:9000"
      - "9001:9001"
    environment:
      - MINIO_ROOT_USER=minio
      - MINIO_ROOT_PASSWORD=minio123
    command: server /data --console-address ":9001"
    volumes:
      - minio-data:/data
    networks:
      - logging

volumes:
  minio-data:

Create bucket:

docker exec minio mc alias set minio http://localhost:9000 minio minio123
docker exec minio mc mb minio/loki-data

---

Part 6: Multi-Tenancy

Why Multi-Tenancy?

Use cases:

• Separate logs by environment (dev/staging/prod)
• Isolate teams or projects
• Implement access control
• Track usage per tenant

Enable Multi-Tenancy

Update loki-config.yml:

auth_enabled: true  # Enable multi-tenancy

server:
  http_listen_port: 3100
  grpc_listen_port: 9096

# Per-tenant limits
limits_config:
  # Global defaults
  ingestion_rate_mb: 4
  ingestion_burst_size_mb: 6

  # Per-tenant overrides
  per_tenant_override_config: /etc/loki/overrides.yaml

Create overrides.yaml:

overrides:
  # Production tenant - higher limits
  tenant-prod:
    ingestion_rate_mb: 16
    ingestion_burst_size_mb: 32
    retention_period: 2160h  # 90 days

  # Development tenant - lower limits, short retention
  tenant-dev:
    ingestion_rate_mb: 2
    ingestion_burst_size_mb: 4
    retention_period: 168h  # 7 days

Send logs to specific tenant:

loki.write "prod" {
  endpoint {
    url = "http://loki:3100/loki/api/v1/push"

    headers = {
      "X-Scope-OrgID" = "tenant-prod",
    }
  }
}

loki.write "dev" {
  endpoint {
    url = "http://loki:3100/loki/api/v1/push"

    headers = {
      "X-Scope-OrgID" = "tenant-dev",
    }
  }
}

Query tenant-specific logs in Grafana:

Data source configuration:

datasources:
  - name: Loki Production
    type: loki
    url: http://loki:3100
    jsonData:
      httpHeaderName1: "X-Scope-OrgID"
    secureJsonData:
      httpHeaderValue1: "tenant-prod"

  - name: Loki Development
    type: loki
    url: http://loki:3100
    jsonData:
      httpHeaderName1: "X-Scope-OrgID"
    secureJsonData:
      httpHeaderValue1: "tenant-dev"

---

Part 7: Alerting with Loki Ruler

Configure Ruler

Update loki-config.yml:

ruler:
  storage:
    type: local
    local:
      directory: /loki/rules
  rule_path: /loki/rules-temp
  alertmanager_url: http://alertmanager:9093
  ring:
    kvstore:
      store: inmemory
  enable_api: true
  enable_alertmanager_v2: true

Create Alert Rules

Create /loki/rules/alerts.yml:

groups:
  - name: application_errors
    interval: 1m
    rules:
      # Alert on high error rate
      - alert: HighErrorRate
        expr: |
          sum(rate({container_name="api"} |= "error" [5m])) > 10
        for: 5m
        labels:
          severity: warning
          component: api
        annotations:
          summary: "High error rate in API"
          description: "API is logging more than 10 errors per second"

      # Alert on specific error pattern
      - alert: DatabaseConnectionFailed
        expr: |
          count_over_time({container_name="api"}
            |= "database connection failed" [5m]) > 0
        for: 1m
        labels:
          severity: critical
          component: database
        annotations:
          summary: "Database connection failures detected"
          description: "API cannot connect to database"

      # Alert on missing logs (dead man's switch)
      - alert: NoLogsReceived
        expr: |
          absent_over_time({job="critical-service"}[10m])
        for: 10m
        labels:
          severity: critical
        annotations:
          summary: "No logs from critical service"
          description: "Critical service has not sent logs in 10 minutes"

      # Alert on high memory usage (from parsed logs)
      - alert: HighMemoryUsage
        expr: |
          avg_over_time({container_name="app"}
            | json
            | unwrap memory_mb [5m]) > 1024
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Application memory usage high"
          description: "App using more than 1GB memory"

Reload rules:

curl -X POST http://localhost:3100/loki/api/v1/rules

---

Part 8: Advanced Features

Log Deduplication

Prevent duplicate logs:

loki.write "loki" {
  endpoint {
    url = "http://loki:3100/loki/api/v1/push"

    // Deduplicate based on timestamp and content
    batch_wait = "1s"
  }
}

Recording Rules

Pre-aggregate expensive queries:

groups:
  - name: recording_rules
    interval: 1m
    rules:
      - record: job:log_lines:rate1m
        expr: sum by (job) (rate({job=~".+"}[1m]))

      - record: container:errors:rate5m
        expr: |
          sum by (container_name) (
            rate({job="docker"} |= "error" [5m])
          )

Query recorded metrics:

job:log_lines:rate1m{job="api"}

Distributed Tracing Integration

Link logs to traces:

In application, emit trace_id in logs:

{
  "timestamp": "2024-12-16T10:23:45Z",
  "level": "info",
  "message": "Request processed",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736"
}

Configure derived fields in Grafana:

# grafana-datasources.yml
datasources:
  - name: Loki
    type: loki
    url: http://loki:3100
    jsonData:
      derivedFields:
        - datasourceUid: tempo_uid
          matcherRegex: "trace_id=(\w+)"
          name: TraceID
          url: "$${__value.raw}"

Click trace_id in logs to jump to trace in Tempo.

---

Troubleshooting

Issue 1: Logs Not Appearing

Check Alloy is sending:

docker logs alloy | grep -i error
curl http://localhost:12345/targets

Check Loki is receiving:

curl http://localhost:3100/ready
curl 'http://localhost:3100/loki/api/v1/label/__name__/values'

Verify labels:

curl 'http://localhost:3100/loki/api/v1/labels'

Issue 2: High Memory Usage

Reduce cache sizes:

# loki-config.yml
chunk_store_config:
  max_look_back_period: 168h  # Don't query beyond 7 days

limits_config:
  max_query_parallelism: 16  # Reduce from 32
  max_streams_per_user: 10000
  max_entries_limit_per_query: 5000

Issue 3: Slow Queries

Add filters to queries:

# Slow (scans all logs)
{job="docker"} |= "error"

# Fast (filters by label first)
{job="docker", container_name="api"} |= "error"

Use smaller time ranges:

# Query last 1 hour instead of 24 hours
{container_name="api"}[1h]

---

Conclusion

Loki brings Prometheus-like efficiency to log aggregation, making it perfect for homelabs and production environments alike. Its label-based indexing, minimal resource requirements, and powerful LogQL queries enable sophisticated log analysis without the operational burden of traditional solutions.

Key takeaways:

• Label-based indexing provides 10x storage savings vs full-text indexing
• LogQL offers powerful filtering, parsing, and aggregation
• Integration with Grafana provides familiar, beautiful dashboards
• Multi-tenancy enables environment separation and access control
• Object storage backends enable limitless retention

Your deployment timeline:

This week:

1. Deploy Loki + Alloy + Grafana stack
2. Collect Docker container logs
3. Build first dashboard with log volume and errors

This month: 4. Master LogQL queries for advanced filtering 5. Configure retention policies 6. Set up alerting with Loki Ruler

This quarter: 7. Implement multi-tenancy for different environments 8. Migrate to object storage for long-term retention 9. Integrate with distributed tracing (Tempo)

Resources:

• Loki Documentation
• LogQL Reference
• Grafana Alloy Migration Guide
• Community Dashboards

Loki proves that effective log aggregation doesn’t require expensive infrastructure or complex operations. With proper configuration, a homelab server can handle millions of log lines with ease.

Happy logging!

---

Guide Version: 1.0 Last Updated: December 2024 Loki Version: v2.9+ Difficulty Level: Intermediate Estimated Time: 60 minutes