Database intermediate ⏱ 30 minutes

Redis: In-Memory Caching for Homelab Apps

High-performance caching and message broker. Speed up your apps with Redis.

By homelabbing.io • 2025-12-17
#redis#caching#database#docker#performance

Introduction

Redis (Remote Dictionary Server) is an open-source, in-memory data structure store used as a database, cache, message broker, and streaming engine. With microsecond response times and support for rich data structures, Redis has become the de facto standard for caching and real-time applications.

For homelabbers, Redis can dramatically improve application performance, enable session management across services, and provide pub/sub messaging for home automation and monitoring systems.

What You’ll Learn:

  • Deploy Redis with Docker in single and clustered configurations
  • Install and configure Redis Insight for visual management
  • Implement persistence strategies (RDB snapshots and AOF)
  • Configure Redis as a cache, database, and message broker
  • Set up Redis Sentinel for high availability
  • Secure Redis with authentication and encryption
  • Monitor performance with metrics and slow query logs
  • Integrate Redis with common homelab applications

Prerequisites:

  • Docker and Docker Compose installed
  • Basic understanding of key-value stores
  • Command-line proficiency
  • Homelab server with minimum 1GB RAM, 10GB storage
  • Recommended: 4GB+ RAM for production caching workloads

Part 1: Understanding Redis

What is Redis?

Redis is an in-memory data structure store that persists data to disk. Unlike traditional databases that store data on disk and cache in memory, Redis keeps the entire dataset in RAM for ultra-fast access while optionally persisting to disk for durability.

Key Characteristics:

  • In-Memory Storage: All data stored in RAM for microsecond latency
  • Rich Data Structures: Strings, hashes, lists, sets, sorted sets, bitmaps, streams
  • Persistence Options: RDB snapshots and/or AOF (Append-Only File)
  • Atomic Operations: All operations are atomic, even complex ones
  • Pub/Sub Messaging: Built-in publish/subscribe capabilities
  • Lua Scripting: Server-side scripting for complex operations
  • Replication: Master-replica with automatic failover via Sentinel
  • Clustering: Horizontal scaling with automatic sharding

Why Redis for Homelabs?

Technical Advantages:

  1. Blazing Fast Performance

    • Sub-millisecond response times (typically under 1ms)
    • 100,000+ operations per second on modest hardware
    • Perfect for real-time applications

  2. Versatile Data Structures

    • String: Simple key-value pairs
    • Hash: Store objects with multiple fields
    • List: Ordered collections, perfect for queues
    • Set: Unique values, great for tags
    • Sorted Set: Ranked data like leaderboards
    • Streams: Log-like data structures for event streaming

  3. Flexible Use Cases

    • Session storage for web applications
    • API rate limiting
    • Real-time analytics
    • Job queues (Celery, Bull, Sidekiq)
    • Cache layer for databases
    • Pub/Sub for microservices communication

  4. Easy to Deploy

    • Single binary, minimal configuration
    • Runs well on Raspberry Pi and low-power devices
    • Docker-friendly

Common Homelab Use Cases:

  • Application Caching: Speed up Nextcloud, Immich, Authentik, Gitea
  • Session Management: Centralized sessions for multiple web apps
  • Rate Limiting: Protect APIs from abuse
  • Home Assistant: State storage and automation triggers
  • Monitoring: Temporary metrics storage (before writing to InfluxDB)
  • Job Queues: Background task processing
  • Real-Time Analytics: Track website visitors, API calls
  • Message Broker: Inter-service communication

Redis vs. Alternatives

FeatureRedisMemcachedKeyDBValkey
Data StructuresManyStrings onlyManyMany
PersistenceYesNoYesYes
ReplicationYesNoYesYes
ClusteringYesClient-sideYesYes
Pub/SubYesNoYesYes
LicenseBSDBSDBSDBSD
MultithreadingLimitedYesYesYes
PerformanceExcellentExcellentFasterFaster
MaturityVery HighHighMediumNew
Best ForFull-featuredSimple cacheHigh perfRedis fork

Note: KeyDB and Valkey are Redis forks with multi-threading and other enhancements. They’re drop-in replacements with better performance on multi-core systems.

Redis Architecture

Single Instance:

Redis Server Process
├── Event Loop (single-threaded)
│   ├── Network I/O
│   ├── Command Processing
│   └── Client Response
├── Memory (Data Structures)
│   ├── Key Space
│   ├── Expires (TTL tracking)
│   └── Pub/Sub Channels
└── Persistence
    ├── RDB Writer (background)
    └── AOF Writer (append-only)

How Redis Handles Commands:

Client Request → TCP Connection → Parse Command
    ↓
Execute Atomically → Modify In-Memory Data
    ↓
Optional: Write to AOF → Reply to Client
    ↓
Background: RDB Snapshot (if configured)

Part 2: Installing Redis with Docker

Quick Start Deployment

Create project directory:

mkdir -p ~/redis-homelab
cd ~/redis-homelab

Create docker-compose.yml:

version: '3.8'

services:
  redis:
    image: redis:7-alpine
    container_name: redis
    hostname: redis
    restart: unless-stopped

    command: >
      redis-server
      --requirepass ${REDIS_PASSWORD:-ChangeMe123!}
      --appendonly yes
      --appendfsync everysec
      --maxmemory 512mb
      --maxmemory-policy allkeys-lru
      --save 900 1
      --save 300 10
      --save 60 10000

    ports:
      - "6379:6379"

    volumes:
      # Data persistence
      - redis-data:/data

      # Custom configuration (optional)
      # - ./redis.conf:/usr/local/etc/redis/redis.conf:ro

    networks:
      - redis-network

    healthcheck:
      test: ["CMD", "redis-cli", "--raw", "incr", "ping"]
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 20s

    # Resource limits
    deploy:
      resources:
        limits:
          cpus: '1'
          memory: 1G
        reservations:
          cpus: '0.25'
          memory: 256M

volumes:
  redis-data:
    driver: local
    driver_opts:
      type: none
      o: bind
      device: /mnt/storage/redis-data

networks:
  redis-network:
    driver: bridge

Create secure password (.env):

# Generate strong password
REDIS_PASSWORD=$(openssl rand -base64 32)

# Save to .env
cat > .env <<EOF
REDIS_PASSWORD=$REDIS_PASSWORD
EOF

chmod 600 .env

echo "Redis password: $REDIS_PASSWORD"
echo "Save this password securely!"

Deploy Redis:

# Start Redis
docker-compose up -d

# View logs
docker-compose logs -f redis

# Verify status
docker-compose ps
docker exec redis redis-cli -a "${REDIS_PASSWORD}" ping
# Should return: PONG

Connect to Redis CLI:

# Using redis-cli in container
docker exec -it redis redis-cli -a "${REDIS_PASSWORD}"

# Once connected, try these commands:
PING                    # Test connection
SET mykey "Hello"       # Set a value
GET mykey               # Get the value
KEYS *                  # List all keys (avoid in production!)
INFO                    # Server info
EXIT                    # Quit

Redis with Redis Insight

Extended docker-compose.yml:

version: '3.8'

services:
  redis:
    # ... (previous redis config)

  redis-insight:
    image: redislabs/redisinsight:latest
    container_name: redis-insight
    restart: unless-stopped

    ports:
      - "8001:8001"

    volumes:
      - redis-insight-data:/db

    networks:
      - redis-network

    depends_on:
      redis:
        condition: service_healthy

volumes:
  redis-data:
    driver: local
  redis-insight-data:
    driver: local

networks:
  redis-network:
    driver: bridge

Start both services:

docker-compose up -d

Access Redis Insight at http://your-server-ip:8001

Configure connection in Redis Insight:

  1. Click “Add Redis Database”
  2. Host: redis
  3. Port: 6379
  4. Password: Your Redis password
  5. Click “Add Redis Database”

Part 3: Configuration and Persistence

Custom Configuration File

Create redis.conf:

# Network
bind 0.0.0.0
port 6379
protected-mode yes
tcp-backlog 511
timeout 0
tcp-keepalive 300

# General
daemonize no
supervised no
pidfile /var/run/redis_6379.pid
loglevel notice
logfile ""
databases 16

# Security
requirepass your_redis_password_here
# rename-command FLUSHDB ""
# rename-command FLUSHALL ""
# rename-command CONFIG ""

# Memory Management
maxmemory 512mb
maxmemory-policy allkeys-lru
# maxmemory-policy options:
#   noeviction       - Return errors when memory limit is reached
#   allkeys-lru      - Evict any key using LRU (Least Recently Used)
#   volatile-lru     - Evict keys with TTL using LRU
#   allkeys-random   - Evict random keys
#   volatile-random  - Evict random keys with TTL
#   volatile-ttl     - Evict keys with shortest TTL

# Persistence - RDB (Snapshots)
save 900 1          # Save after 900 sec (15 min) if at least 1 key changed
save 300 10         # Save after 300 sec (5 min) if at least 10 keys changed
save 60 10000       # Save after 60 sec if at least 10000 keys changed
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename dump.rdb
dir /data

# Persistence - AOF (Append Only File)
appendonly yes
appendfilename "appendonly.aof"
appendfsync everysec
# appendfsync options:
#   always    - Fsync every write (slow, safest)
#   everysec  - Fsync every second (good balance)
#   no        - Let OS decide when to fsync (fastest, least safe)

no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
aof-load-truncated yes
aof-use-rdb-preamble yes

# Replication (if using master-replica)
# replicaof YOUR-SERVER-IP 6379
# masterauth your_redis_password_here
replica-serve-stale-data yes
replica-read-only yes
repl-diskless-sync no
repl-diskless-sync-delay 5
repl-disable-tcp-nodelay no
replica-priority 100

# Security - Command Renaming (optional hardening)
# rename-command FLUSHDB ""
# rename-command FLUSHALL ""
# rename-command KEYS ""
# rename-command CONFIG b840fc02d524045429941cc15f59e41cb7be6c52

# Slow Log
slowlog-log-slower-than 10000
slowlog-max-len 128

# Latency Monitor
latency-monitor-threshold 100

# Event Notification
notify-keyspace-events ""

# Advanced Config
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-size -2
list-compress-depth 0
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
hll-sparse-max-bytes 3000
stream-node-max-bytes 4096
stream-node-max-entries 100

# Active Rehashing
activerehashing yes

# Client Output Buffers
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit replica 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60

# Frequency
hz 10
dynamic-hz yes

# AOF Rewrite
aof-rewrite-incremental-fsync yes
rdb-save-incremental-fsync yes

Use custom config:

# In docker-compose.yml
services:
  redis:
    volumes:
      - ./redis.conf:/usr/local/etc/redis/redis.conf:ro
    command: redis-server /usr/local/etc/redis/redis.conf

Understanding Persistence Strategies

RDB (Redis Database Backup):

Pros:

  • Point-in-time snapshots
  • Compact single-file backups
  • Faster restarts (binary format)
  • Good for disaster recovery

Cons:

  • Can lose data between snapshots
  • Fork() can pause Redis on large datasets
  • Not ideal for write-heavy workloads

AOF (Append-Only File):

Pros:

  • More durable (minimal data loss)
  • Append-only, less corruption risk
  • Automatic log rewriting
  • Human-readable (can be edited)

Cons:

  • Larger file sizes
  • Slower restarts
  • Slightly slower writes with fsync

Recommended Strategy:

# Use both for maximum durability
appendonly yes
appendfsync everysec
save 900 1
save 300 10
save 60 10000

Memory Eviction Policies

Choose based on your use case:

1. Cache Only (LRU):

maxmemory 512mb
maxmemory-policy allkeys-lru

Best for: Caching database queries, API responses

2. Cache with TTLs:

maxmemory 512mb
maxmemory-policy volatile-lru

Best for: Session storage, temporary data

3. Never Evict (Database Mode):

maxmemory 512mb
maxmemory-policy noeviction

Best for: Job queues, critical data Warning: Redis will return errors when memory is full

4. Random Eviction:

maxmemory 512mb
maxmemory-policy allkeys-random

Best for: When all keys have equal importance

Part 4: Common Use Cases and Patterns

1. Application Caching

Cache database queries:

# Set cache with 1-hour expiration
SET user:1000:profile '{"name":"John","email":"[email protected]"}' EX 3600

# Get cached data
GET user:1000:profile

# Check if key exists
EXISTS user:1000:profile

# Delete cache (invalidation)
DEL user:1000:profile

# Pattern: Cache-aside
# 1. Check cache first
# 2. If miss, query database
# 3. Store result in cache
# 4. Return data

Python example (cache-aside pattern):

import redis
import json

# Connect to Redis
r = redis.Redis(host='localhost', port=6379, password='your_password', decode_responses=True)

def get_user_profile(user_id):
    # Check cache
    cache_key = f"user:{user_id}:profile"
    cached = r.get(cache_key)

    if cached:
        return json.loads(cached)

    # Cache miss - query database
    user_data = database.query(f"SELECT * FROM users WHERE id = {user_id}")

    # Store in cache for 1 hour
    r.setex(cache_key, 3600, json.dumps(user_data))

    return user_data

2. Session Storage

Store user sessions:

# Create session (24-hour expiration)
HSET session:abc123 user_id 1000
HSET session:abc123 username "john"
HSET session:abc123 last_seen "2024-12-16T10:30:00"
EXPIRE session:abc123 86400

# Get session data
HGETALL session:abc123

# Update last seen
HSET session:abc123 last_seen "2024-12-16T11:00:00"

# Delete session (logout)
DEL session:abc123

3. Rate Limiting

Limit API requests:

# Simple rate limiting (10 requests per minute)
# Redis key: rate_limit:IP_ADDRESS

# Increment counter
INCR rate_limit:192.168.1.100

# Set expiration on first request
EXPIRE rate_limit:192.168.1.100 60

# Check current count
GET rate_limit:192.168.1.100

Python rate limiter:

def check_rate_limit(ip_address, max_requests=10, window=60):
    key = f"rate_limit:{ip_address}"

    # Get current count
    count = r.incr(key)

    # Set expiration on first request
    if count == 1:
        r.expire(key, window)

    # Check if over limit
    if count > max_requests:
        return False  # Rate limited

    return True  # Allowed

4. Job Queue

Simple job queue with lists:

# Producer: Add job to queue
LPUSH jobs:email '{"to":"[email protected]","subject":"Welcome"}'

# Consumer: Get job from queue (blocking)
BRPOP jobs:email 0

# Get queue length
LLEN jobs:email

# Peek at next job without removing
LINDEX jobs:email -1

5. Pub/Sub Messaging

Publish/Subscribe pattern:

# Subscriber (in one terminal)
SUBSCRIBE homeassistant:events

# Publisher (in another terminal)
PUBLISH homeassistant:events '{"type":"motion_detected","sensor":"living_room"}'

# Pattern subscription
PSUBSCRIBE homeassistant:*

Python pub/sub example:

# Publisher
def publish_event(channel, message):
    r.publish(channel, json.dumps(message))

# Subscriber
def event_listener():
    pubsub = r.pubsub()
    pubsub.subscribe('homeassistant:events')

    for message in pubsub.listen():
        if message['type'] == 'message':
            data = json.loads(message['data'])
            print(f"Event received: {data}")

6. Leaderboard (Sorted Sets)

Store ranked data:

# Add scores
ZADD leaderboard 100 "player1"
ZADD leaderboard 250 "player2"
ZADD leaderboard 180 "player3"

# Get top 10
ZREVRANGE leaderboard 0 9 WITHSCORES

# Get player rank
ZREVRANK leaderboard "player2"

# Increment score
ZINCRBY leaderboard 50 "player1"

# Get score range
ZRANGEBYSCORE leaderboard 100 200 WITHSCORES

Part 5: High Availability with Sentinel

Understanding Redis Sentinel

Redis Sentinel provides:

  • Monitoring: Checks if master and replicas are working
  • Notification: Alerts when something goes wrong
  • Automatic Failover: Promotes replica to master if master fails
  • Configuration Provider: Clients discover current master address

Sentinel Architecture:

Sentinel 1 ←→ Sentinel 2 ←→ Sentinel 3
     ↓              ↓              ↓
  Redis Master ← Redis Replica 1
                ← Redis Replica 2

Deploying Redis with Sentinel

Create sentinel-compose.yml:

version: '3.8'

services:
  redis-master:
    image: redis:7-alpine
    container_name: redis-master
    command: redis-server --requirepass ${REDIS_PASSWORD} --appendonly yes
    ports:
      - "6379:6379"
    volumes:
      - redis-master-data:/data
    networks:
      redis-ha-network:
        ipv4_address: 172.30.0.10

  redis-replica-1:
    image: redis:7-alpine
    container_name: redis-replica-1
    command: >
      redis-server
      --replicaof redis-master 6379
      --requirepass ${REDIS_PASSWORD}
      --masterauth ${REDIS_PASSWORD}
      --appendonly yes
    ports:
      - "6380:6379"
    volumes:
      - redis-replica-1-data:/data
    networks:
      redis-ha-network:
        ipv4_address: 172.30.0.11
    depends_on:
      - redis-master

  redis-replica-2:
    image: redis:7-alpine
    container_name: redis-replica-2
    command: >
      redis-server
      --replicaof redis-master 6379
      --requirepass ${REDIS_PASSWORD}
      --masterauth ${REDIS_PASSWORD}
      --appendonly yes
    ports:
      - "6381:6379"
    volumes:
      - redis-replica-2-data:/data
    networks:
      redis-ha-network:
        ipv4_address: 172.30.0.12
    depends_on:
      - redis-master

  sentinel-1:
    image: redis:7-alpine
    container_name: sentinel-1
    command: >
      redis-sentinel /etc/redis/sentinel.conf
      --sentinel
    ports:
      - "26379:26379"
    volumes:
      - ./sentinel-1.conf:/etc/redis/sentinel.conf
    networks:
      redis-ha-network:
        ipv4_address: 172.30.0.20
    depends_on:
      - redis-master

  sentinel-2:
    image: redis:7-alpine
    container_name: sentinel-2
    command: redis-sentinel /etc/redis/sentinel.conf --sentinel
    ports:
      - "26380:26379"
    volumes:
      - ./sentinel-2.conf:/etc/redis/sentinel.conf
    networks:
      redis-ha-network:
        ipv4_address: 172.30.0.21
    depends_on:
      - redis-master

  sentinel-3:
    image: redis:7-alpine
    container_name: sentinel-3
    command: redis-sentinel /etc/redis/sentinel.conf --sentinel
    ports:
      - "26381:26379"
    volumes:
      - ./sentinel-3.conf:/etc/redis/sentinel.conf
    networks:
      redis-ha-network:
        ipv4_address: 172.30.0.22
    depends_on:
      - redis-master

volumes:
  redis-master-data:
  redis-replica-1-data:
  redis-replica-2-data:

networks:
  redis-ha-network:
    driver: bridge
    ipam:
      config:
        - subnet: 172.30.0.0/16

Create sentinel.conf:

port 26379
dir /tmp
sentinel monitor mymaster redis-master 6379 2
sentinel auth-pass mymaster your_redis_password
sentinel down-after-milliseconds mymaster 5000
sentinel parallel-syncs mymaster 1
sentinel failover-timeout mymaster 10000

Test failover:

# Stop master
docker stop redis-master

# Check sentinel logs
docker logs -f sentinel-1

# Sentinels will promote a replica to master
# Verify new master
docker exec redis-replica-1 redis-cli -a "${REDIS_PASSWORD}" ROLE

Part 6: Monitoring and Performance

Essential Redis Commands

Server info:

# Comprehensive server information
INFO

# Specific sections
INFO server
INFO clients
INFO memory
INFO persistence
INFO stats
INFO replication
INFO cpu
INFO keyspace

# Memory usage
MEMORY STATS
MEMORY DOCTOR

# Client list
CLIENT LIST

# Slow log
SLOWLOG GET 10
SLOWLOG LEN
SLOWLOG RESET

Performance monitoring:

# Monitor commands in real-time
MONITOR

# Latency diagnostics
LATENCY DOCTOR
LATENCY GRAPH
LATENCY HISTORY

# Stats
INFO stats

Prometheus Redis Exporter

Add to docker-compose.yml:

services:
  redis-exporter:
    image: oliver006/redis_exporter:latest
    container_name: redis-exporter
    environment:
      REDIS_ADDR: redis:6379
      REDIS_PASSWORD: ${REDIS_PASSWORD}
    ports:
      - "9121:9121"
    networks:
      - redis-network
    depends_on:
      - redis

Prometheus config:

# prometheus.yml
scrape_configs:
  - job_name: 'redis'
    static_configs:
      - targets: ['redis-exporter:9121']

Grafana Dashboard:

Import Dashboard ID: 11835 (Redis Dashboard for Prometheus)

Part 7: Security Best Practices

1. Enable Password Authentication

requirepass your_strong_password_here

2. Bind to Specific Interface

# Only local connections
bind 127.0.0.1

# Specific network
bind 192.168.1.100

3. Disable Dangerous Commands

rename-command FLUSHDB ""
rename-command FLUSHALL ""
rename-command KEYS ""
rename-command CONFIG "CONFIG_b840fc02d524045429"

4. Enable TLS/SSL

port 0
tls-port 6379
tls-cert-file /path/to/redis.crt
tls-key-file /path/to/redis.key
tls-ca-cert-file /path/to/ca.crt

5. Network Isolation

Use Docker networks to isolate Redis from untrusted containers.

Conclusion

You now have a comprehensive understanding of Redis deployment and management in your homelab.

Key Takeaways:

  1. Performance: In-memory storage provides microsecond latency
  2. Versatility: Multiple data structures for different use cases
  3. Persistence: RDB and AOF options for durability
  4. High Availability: Sentinel for automatic failover
  5. Easy Integration: Works with virtually all programming languages

Next Steps:

  • Integrate Redis caching into your applications
  • Set up monitoring and alerting
  • Implement Redis Cluster for horizontal scaling
  • Explore Redis Streams for event processing
  • Practice disaster recovery procedures

Resources:

Happy caching!

Guide Version: 1.0 Last Updated: December 2024 Target Audience: Homelab administrators, developers Difficulty Level: Intermediate Estimated Time: 45 minutes