K3s Kubernetes: The Complete Homelab Guide

K3s is a lightweight, certified Kubernetes distribution designed specifically for edge computing, IoT, and resource-constrained environments. This comprehensive guide will walk you through everything from basic installation to advanced automation, helping you build production-grade container orchestration in your homelab.

Table of Contents

1. Understanding K3s
2. Prerequisites & Planning
3. Installation & Cluster Setup
4. Storage Management
5. Networking, Ingress & Load Balancing
6. Application Deployment with Helm
7. Monitoring & Observability
8. AI Automation & Intelligent Management
9. Troubleshooting & Optimization
10. Next Steps

---

Understanding K3s

What is K3s?

K3s is a lightweight Kubernetes distribution developed by Rancher Labs (now SUSE) that packages everything you need for a production-ready Kubernetes cluster into a single binary under 100MB. It’s CNCF certified and maintains full Kubernetes API compatibility while removing legacy, alpha, and non-default features to minimize its footprint.

Key Characteristics:

• Ultra-lightweight: Single binary under 100MB with no external dependencies
• Minimal memory: Runs comfortably with 512MB of RAM (compared to 3GB+ for full Kubernetes)
• CNCF certified: Fully conformant Kubernetes distribution
• Built-in components: Includes CoreDNS, Traefik ingress controller, and ServiceLB load balancer
• Simple datastore: Uses SQLite by default, with support for external databases

Why K3s for Homelabs?

K3s strikes the perfect balance between having full Kubernetes capabilities and maintaining minimal resource overhead, making it ideal for homelab environments:

Learning Value:

• Provides production-like Kubernetes experience without enterprise complexity
• Skills directly transferable to professional Kubernetes environments
• Experiment with microservices architecture and cloud-native patterns

Resource Efficiency:

• Runs on commodity hardware, Raspberry Pi, or repurposed systems
• Minimal CPU, memory, and disk requirements
• Efficient resource usage allows consolidation of multiple services

Cost Effectiveness:

• Completely free and open source
• No licensing fees or vendor lock-in
• Low hardware requirements reduce power consumption

Operational Simplicity:

• Single command installation
• No complex configuration required
• Built-in features reduce dependency management

K3s vs Kubernetes vs MicroK8s

Decision Guide:

• Choose Kubernetes for: Large teams, enterprise production, complex requirements
• Choose K3s for: Homelabs, learning, edge computing, resource-constrained environments
• Choose MicroK8s for: Ubuntu workstations, snap-friendly systems, Canonical ecosystem

K3s Architecture

K3s consolidates Kubernetes components into a streamlined architecture:

┌─────────────────────────────────────────┐
│         K3s Server Node                 │
├─────────────────────────────────────────┤
│  API Server                             │
│  Scheduler                              │
│  Controller Manager                     │
│  Kubelet                                │
│  Data Store (SQLite/etcd)               │
│  ├── CoreDNS                            │
│  ├── Traefik Ingress Controller         │
│  ├── ServiceLB                          │
│  └── Local Path Provisioner             │
└─────────────────────────────────────────┘
           │
           ├── Worker Node(s)
           │   ├── Kubelet
           │   ├── containerd
           │   └── Pods
           │
           └── Container Runtime (containerd)

Core Components:

• API Server: Central control plane accepting API requests
• Kubelet: Node agent managing pod lifecycle
• CoreDNS: Service discovery via DNS
• Traefik: HTTP/HTTPS ingress routing
• ServiceLB: Load balancer implementation
• Local Path Provisioner: Dynamic volume provisioning

Common Homelab Use Cases

1. Personal Application Hosting: Run multiple web apps, APIs, databases on unified platform
2. Development Environment: Test applications in Kubernetes before production deployment
3. Self-Hosted Services: Deploy Nextcloud, Gitea, Jellyfin, Home Assistant
4. IoT & Edge Computing: Orchestrate containerized services on Raspberry Pi clusters
5. Data Processing: Run batch jobs, data pipelines, scheduled tasks
6. Smart Home Automation: Centralize home automation services
7. Learning Platform: Build professional Kubernetes skills without enterprise costs

---

Prerequisites and Planning

Knowledge Requirements

Before diving into K3s, you should be comfortable with:

• Linux fundamentals: File system navigation, systemd services, package management
• Container basics: Docker concepts, images vs containers, container networking
• Networking fundamentals: IP addressing, DNS, ports, basic firewalling
• YAML syntax: Kubernetes manifests are written in YAML

If you’re new to containers, consider learning Docker basics first. Many concepts translate directly to Kubernetes.

Hardware Requirements

Minimum (Single Node):

• 2GB RAM
• 2 CPU cores (virtual or physical)
• 10GB storage
• Linux operating system

Recommended (Learning/Development):

• 4GB+ RAM per node
• 4 CPU cores
• 20GB+ storage
• Multiple nodes for HA testing

Production Homelab:

• 8GB+ RAM per node
• 4-8 CPU cores
• 50GB+ storage
• 3+ nodes for high availability

Supported Platforms:

• Linux x86_64 (most common)
• ARM64 (Raspberry Pi 4, Apple Silicon)
• ARM32 (older Raspberry Pi models)

Network Planning

Consider your network architecture before installation:

IP Addressing:

• Reserve static IPs for server nodes
• Plan IP ranges for services (ClusterIP ranges)
• Consider separate VLAN for cluster traffic (optional)

DNS Strategy:

• Local DNS entries for ingress endpoints
• Split-horizon DNS for internal/external access
• Consider using CoreDNS for service discovery

Firewall Rules:

# Required ports for K3s
# Server node:
# - 6443: Kubernetes API server
# - 10250: Kubelet metrics
# - 2379-2380: etcd (HA deployments only)

# Worker nodes:
# - 10250: Kubelet metrics

# Optional:
# - 80/443: HTTP/HTTPS ingress (Traefik)

Storage Planning

Choose storage strategy based on your needs:

Local Path (Default):

• Best for: Single-node, development, learning
• Pros: Zero configuration, good performance
• Cons: No HA, data tied to node

Longhorn (Distributed):

• Best for: Multi-node HA clusters
• Pros: Data replication, backup features, snapshots
• Cons: Requires multiple nodes, more complex setup

NFS:

• Best for: Shared storage across nodes
• Pros: Simple shared access
• Cons: Single point of failure, network dependency

Pre-Installation Checklist

Before starting installation:

• Linux system with root/sudo access
• Stable internet connection
• Sufficient disk space (df -h /)
• Network connectivity between nodes (if multi-node)
• Firewall rules configured or disabled
• Static IPs assigned (recommended for servers)
• System time synchronized (NTP)

Quick system check:

# Verify resources
free -h          # Check available memory
nproc            # Check CPU cores
df -h /          # Check disk space

# Test internet connectivity
ping -c 3 8.8.8.8

# Check kernel version (should be 3.10+)
uname -r

---

Installation and Cluster Setup

Single-Node Installation

The simplest way to get started with K3s is a single-node installation, perfect for learning and development.

Basic Installation

Install K3s with a single command:

# Download and install K3s
curl -sfL https://get.k3s.io | sh -

# The installer will:
# - Download the K3s binary
# - Install it as a systemd service
# - Start the service automatically
# - Create kubeconfig at /etc/rancher/k3s/k3s.yaml

Verify installation:

# Check K3s service status
sudo systemctl status k3s

# Verify node is ready (may take 30-60 seconds)
sudo kubectl get nodes

# Expected output:
# NAME       STATUS   ROLES                  AGE   VERSION
# hostname   Ready    control-plane,master   1m    v1.28.5+k3s1

# Check system pods
sudo kubectl get pods -A

# Should see pods in kube-system namespace:
# - coredns
# - local-path-provisioner
# - metrics-server
# - traefik

Accessing kubectl

By default, K3s requires root access for kubectl. To use kubectl as a regular user:

# Create .kube directory
mkdir -p ~/.kube

# Copy kubeconfig
sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config

# Fix ownership
sudo chown $USER:$USER ~/.kube/config

# Set restrictive permissions
chmod 600 ~/.kube/config

# Test access
kubectl get nodes

Custom Installation Options

For more control over the installation, use environment variables and flags:

# Install specific version
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.28.5+k3s1 sh -

# Install without default components
curl -sfL https://get.k3s.io | sh -s - server   --disable traefik   --disable servicelb   --disable local-storage   --node-name my-k3s-node

# Common options:
# --disable traefik: Skip Traefik ingress controller
# --disable servicelb: Skip ServiceLB load balancer
# --disable local-storage: Skip local path provisioner
# --node-name: Custom node identifier
# --write-kubeconfig-mode: Set kubeconfig permissions (e.g., "644")

First Deployment Test

Verify your cluster works by deploying a simple application:

# Create a test deployment
kubectl create deployment nginx --image=nginx

# Expose it as a LoadBalancer service
kubectl expose deployment nginx --port=80 --type=LoadBalancer

# Check the service
kubectl get svc nginx

# Get the external IP and test
EXTERNAL_IP=$(kubectl get svc nginx -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
curl http://$EXTERNAL_IP

# Should return nginx welcome page

# Cleanup
kubectl delete deployment nginx
kubectl delete service nginx

Multi-Node Cluster

Expand your K3s deployment to multiple nodes for increased capacity and resilience.

Architecture

A multi-node K3s cluster consists of:

• Server nodes: Run control plane and workloads
• Agent nodes: Run workloads only

┌──────────────┐
│ Server Node  │  ← Control plane + workloads
└──────────────┘
      │
      ├── Agent Node 1  ← Workloads only
      │
      └── Agent Node 2  ← Workloads only

Adding Worker Nodes

Step 1: Retrieve the node token from the server:

# On the server node
sudo cat /var/lib/rancher/k3s/server/node-token

# Output looks like:
# K10abc123def456ghi789jkl012mno345pqr678stu901vwx234yz::server:567890abcdef123456

Step 2: Install K3s agent on worker nodes:

# On each worker node
curl -sfL https://get.k3s.io | K3S_URL=https://SERVER_IP:6443   K3S_TOKEN=YOUR_NODE_TOKEN sh -

# Replace SERVER_IP with your server's IP address
# Replace YOUR_NODE_TOKEN with the token from step 1

Step 3: Verify cluster:

# On the server node
kubectl get nodes

# Expected output:
# NAME          STATUS   ROLES                  AGE
# k3s-server    Ready    control-plane,master   10m
# k3s-worker-1  Ready    <none&gt; 2m
# k3s-worker-2  Ready    <none&gt; 1m

Testing Multi-Node Deployment

Deploy an application that spreads across nodes:

# multi-node-test.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: multi-node-app
spec:
  replicas: 6  # More replicas than nodes
  selector:
    matchLabels:
      app: test
  template:
    metadata:
      labels:
        app: test
    spec:
      # Spread pods across nodes
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - test
              topologyKey: kubernetes.io/hostname
      containers:
      - name: nginx
        image: nginx:alpine
        ports:
        - containerPort: 80
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 200m
            memory: 256Mi

Deploy and verify:

# Apply the deployment
kubectl apply -f multi-node-test.yaml

# Watch pods distribute across nodes
kubectl get pods -o wide

# Should see pods on different nodes

High Availability (HA) Setup

For production-grade reliability, deploy K3s in high availability mode with 3+ server nodes.

HA with Embedded etcd

The recommended HA approach uses K3s’s embedded etcd (requires K3s v1.19+):

Architecture:

┌────────────┐   ┌────────────┐   ┌────────────┐
│  Server 1  │   │  Server 2  │   │  Server 3  │
│ (etcd member)   │ (etcd member)   │ (etcd member) │
└────────────┘   └────────────┘   └────────────┘
       └──────────────┬──────────────┘
                      │
         ┌────────────┴────────────┐
         │                         │
    ┌────────┐               ┌────────┐
    │Worker 1│               │Worker 2│
    └────────┘               └────────┘

Installation steps:

# Server 1: Initialize the cluster
curl -sfL https://get.k3s.io | sh -s - server   --cluster-init   --node-name=k3s-server-1   --tls-san=k3s-cluster.local

# Get the token
TOKEN=$(sudo cat /var/lib/rancher/k3s/server/node-token)

# Server 2: Join the cluster
curl -sfL https://get.k3s.io | K3S_TOKEN=$TOKEN sh -s - server   --server https://SERVER1_IP:6443   --node-name=k3s-server-2

# Server 3: Join the cluster
curl -sfL https://get.k3s.io | K3S_TOKEN=$TOKEN sh -s - server   --server https://SERVER1_IP:6443   --node-name=k3s-server-3

# Verify HA cluster
kubectl get nodes
kubectl get endpoints kubernetes
# Should show all 3 server IPs

Important HA considerations:

• Use odd number of server nodes (3, 5, 7)
• Minimum 3 servers for quorum
• Place servers on different physical machines
• Consider using a load balancer in front of API servers

Testing HA Failover

Verify your HA setup can handle node failures:

# Deploy a test application
kubectl create deployment ha-test --image=nginx --replicas=3

# Watch the deployment
watch kubectl get pods -o wide

# On one server node, stop K3s
sudo systemctl stop k3s

# Observe:
# - Cluster continues operating
# - Pods on failed node reschedule to other nodes
# - kubectl commands still work via other servers

# Restart the node
sudo systemctl start k3s

# Node rejoins automatically

HA with External Database

For larger deployments or if you prefer separate database management:

Supported databases:

• MySQL/MariaDB
• PostgreSQL

Example with MySQL:

# Prepare MySQL database
mysql -u root -p
CREATE DATABASE k3s;
CREATE USER 'k3s'@'%' IDENTIFIED BY 'secure_password';
GRANT ALL ON k3s.* TO 'k3s'@'%';
FLUSH PRIVILEGES;

# On each server node
curl -sfL https://get.k3s.io | sh -s - server   --datastore-endpoint="mysql://k3s:secure_password@tcp(MYSQL_HOST:3306)/k3s"   --node-name=k3s-server-1

# Repeat for other server nodes with unique names

Cluster Upgrades

Keep your K3s cluster updated with the latest security patches and features.

Manual Upgrade

# Check current version
kubectl version

# Upgrade to latest version
curl -sfL https://get.k3s.io | sh -

# Or upgrade to specific version
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.28.5+k3s1 sh -

# Verify upgrade
kubectl version
kubectl get nodes

Zero-Downtime Upgrade (Multi-Node)

For clusters with workloads, use a rolling upgrade strategy:

# For each node (one at a time):

# 1. Cordon node (prevent new pods)
kubectl cordon NODE_NAME

# 2. Drain workloads
kubectl drain NODE_NAME --ignore-daemonsets --delete-emptydir-data

# 3. Upgrade K3s
curl -sfL https://get.k3s.io | sh -

# 4. Wait for node to rejoin
kubectl get nodes

# 5. Uncordon node
kubectl uncordon NODE_NAME

# Move to next node

Uninstalling K3s

If you need to remove K3s:

# On server nodes
/usr/local/bin/k3s-uninstall.sh

# On worker nodes
/usr/local/bin/k3s-agent-uninstall.sh

# This removes:
# - K3s binaries
# - Systemd services
# - Data directories
# - Network configurations

---

Storage Management

Kubernetes uses a flexible storage system with three key concepts: PersistentVolumes (PV), PersistentVolumeClaims (PVC), and StorageClasses. Understanding these is crucial for managing stateful applications.

Kubernetes Storage Fundamentals

Storage Architecture:

Application Pod
    ↓ (mounts)
PersistentVolumeClaim (PVC) ← Request for storage
    ↓ (binds to)
StorageClass ← Defines provisioning rules
    ↓ (creates)
PersistentVolume (PV) ← Actual storage
    ↓ (backed by)
Physical Storage ← Host disk, NFS, cloud storage, etc.

Key Concepts:

• PersistentVolume (PV): Cluster-level storage resource
• PersistentVolumeClaim (PVC): Namespace-level storage request
• StorageClass: Defines how storage is dynamically provisioned
• Access Modes:
  • ReadWriteOnce (RWO): Single node read-write
  • ReadOnlyMany (ROX): Multiple nodes read-only
  • ReadWriteMany (RWX): Multiple nodes read-write

Local Path Storage (Built-in)

K3s includes Rancher’s Local Path Provisioner by default, providing zero-configuration storage for development and single-node deployments.

How It Works

• Creates volumes on the node’s filesystem at /var/lib/rancher/k3s/storage
• Dynamically provisions storage when PVCs are created
• Binds volumes to the node where the pod is scheduled
• Perfect for development, testing, and single-node clusters

Basic Usage

# pvc-local-path.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: app-storage
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: local-path
  resources:
    requests:
      storage: 5Gi

Example: MySQL with Persistent Storage

# mysql-with-storage.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysql-pvc
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: local-path
  resources:
    requests:
      storage: 10Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mysql
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mysql
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
      - name: mysql
        image: mysql:8.0
        env:
        - name: MYSQL_ROOT_PASSWORD
          value: "changeme"  # Use secrets in production!
        - name: MYSQL_DATABASE
          value: "myapp"
        ports:
        - containerPort: 3306
          name: mysql
        volumeMounts:
        - name: mysql-data
          mountPath: /var/lib/mysql
        resources:
          requests:
            cpu: 500m
            memory: 512Mi
          limits:
            cpu: 1000m
            memory: 1Gi
      volumes:
      - name: mysql-data
        persistentVolumeClaim:
          claimName: mysql-pvc
---
apiVersion: v1
kind: Service
metadata:
  name: mysql
spec:
  ports:
  - port: 3306
    targetPort: 3306
  selector:
    app: mysql

Deploy and test:

# Apply the manifests
kubectl apply -f mysql-with-storage.yaml

# Wait for pod to be ready
kubectl wait --for=condition=ready pod -l app=mysql --timeout=120s

# Verify storage
kubectl get pvc mysql-pvc
kubectl get pv

# Connect to MySQL
kubectl exec -it deployment/mysql -- mysql -u root -pchangeme

# Create test data
mysql> CREATE TABLE test (id INT, data VARCHAR(255));
mysql> INSERT INTO test VALUES (1, 'persistent data');
mysql> SELECT * FROM test;
mysql> exit

# Delete the pod (simulating failure)
kubectl delete pod -l app=mysql

# Wait for new pod
kubectl wait --for=condition=ready pod -l app=mysql --timeout=120s

# Verify data persisted
kubectl exec -it deployment/mysql -- mysql -u root -pchangeme -e "SELECT * FROM test.test;"

# Should show: 1 | persistent data

Limitations of Local Path Storage

• No HA: Data is tied to a specific node
• Pod migration: If a node fails, pods can’t access their data on another node
• No replication: Data loss if the node’s disk fails
• Single node write: Can’t share storage between nodes

Use local path for:

• Development environments
• Single-node clusters
• Non-critical data
• Learning and testing

Don’t use local path for:

• Production databases
• Multi-node HA clusters
• Critical data requiring backup
• Applications requiring shared storage

Longhorn - Distributed Block Storage

For production-grade storage with high availability, Longhorn provides cloud-native distributed block storage.

What is Longhorn?

Longhorn is a CNCF project that provides:

• Distributed storage: Data replicated across multiple nodes
• High availability: Automatic failover and recovery
• Snapshots: Point-in-time backup capability
• Backup & restore: Integration with NFS, S3, etc.
• Web UI: Graphical management interface
• Monitoring: Built-in metrics and alerting

Architecture

┌─────────────┐   ┌─────────────┐   ┌─────────────┐
│   Node 1    │   │   Node 2    │   │   Node 3    │
│  Replica 1  │   │  Replica 2  │   │  Replica 3  │
└─────────────┘   └─────────────┘   └─────────────┘
       │                  │                  │
       └──────────────────┴──────────────────┘
                          │
                   ┌──────┴──────┐
                   │  Application │
                   │     Pod      │
                   └──────────────┘

Installation Prerequisites

# Install iSCSI initiator on ALL nodes
# Ubuntu/Debian:
sudo apt-get update
sudo apt-get install -y open-iscsi
sudo systemctl enable iscsid
sudo systemctl start iscsid

# CentOS/RHEL:
sudo yum install -y iscsi-initiator-utils
sudo systemctl enable iscsid
sudo systemctl start iscsid

# Verify iSCSI is running
sudo systemctl status iscsid

Installing Longhorn via Helm

# Add Longhorn repository
helm repo add longhorn https://charts.longhorn.io
helm repo update

# Create namespace
kubectl create namespace longhorn-system

# Install Longhorn
helm install longhorn longhorn/longhorn   --namespace longhorn-system   --set persistence.defaultClass=true   --set persistence.defaultClassReplicaCount=3

# This process takes 2-5 minutes

# Verify installation
kubectl get pods -n longhorn-system

# Should see multiple pods:
# - longhorn-manager
# - longhorn-driver-deployer
# - longhorn-ui
# - engine-image pods
# - instance-manager pods

# Check storage class
kubectl get storageclass

# Should show:
# NAME                 PROVISIONER          RECLAIMPOLICY
# longhorn (default)   driver.longhorn.io   Delete

Accessing Longhorn UI

# Port forward to Longhorn UI
kubectl port-forward -n longhorn-system svc/longhorn-frontend 8080:80

# Visit http://localhost:8080

# Or create an ingress
kubectl apply -f - <<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: longhorn-ingress
  namespace: longhorn-system
spec:
  rules:
  - host: longhorn.local
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: longhorn-frontend
            port:
              number: 80
EOF

Using Longhorn Storage

# postgresql-ha.yaml - Database with HA storage
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgresql
spec:
  serviceName: postgresql
  replicas: 1
  selector:
    matchLabels:
      app: postgresql
  template:
    metadata:
      labels:
        app: postgresql
    spec:
      containers:
      - name: postgresql
        image: postgres:15
        env:
        - name: POSTGRES_PASSWORD
          value: "postgres123"
        - name: POSTGRES_DB
          value: "myapp"
        ports:
        - containerPort: 5432
          name: postgres
        volumeMounts:
        - name: data
          mountPath: /var/lib/postgresql/data
        resources:
          requests:
            cpu: 250m
            memory: 256Mi
          limits:
            cpu: 1000m
            memory: 1Gi
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: longhorn
      resources:
        requests:
          storage: 20Gi
---
apiVersion: v1
kind: Service
metadata:
  name: postgresql
spec:
  clusterIP: None
  ports:
  - port: 5432
    targetPort: 5432
    name: postgres
  selector:
    app: postgresql

Deploy and test HA:

# Deploy PostgreSQL
kubectl apply -f postgresql-ha.yaml

# Wait for ready
kubectl wait --for=condition=ready pod postgresql-0 --timeout=120s

# Connect and create test data
kubectl exec -it postgresql-0 -- psql -U postgres -d myapp

# In psql:
CREATE TABLE test (id SERIAL PRIMARY KEY, data TEXT);
INSERT INTO test (data) VALUES ('HA test data'), ('This will survive node failure');
SELECT * FROM test;
q

# Identify which node the pod is on
kubectl get pod postgresql-0 -o wide

# Simulate node failure by cordoning and draining
NODE=$(kubectl get pod postgresql-0 -o jsonpath='{.spec.nodeName}')
kubectl cordon $NODE
kubectl drain $NODE --ignore-daemonsets --delete-emptydir-data --force

# Pod reschedules to another node
kubectl get pod postgresql-0 -o wide

# Verify data persisted
kubectl exec -it postgresql-0 -- psql -U postgres -d myapp -c "SELECT * FROM test;"

# Should show both rows!

# Uncordon the node
kubectl uncordon $NODE

Longhorn Backup Configuration

Configure automated backups to external storage:

# In Longhorn UI:
# 1. Settings → General → Backup Target
# 2. Set backup target:
#    - NFS: nfs://nfs-server:/backup-path
#    - S3: s3://bucket-name@region/path/
# 3. Optional: Set backup target credential secret

# Or via kubectl
kubectl apply -f - <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: longhorn-storageclass
  namespace: longhorn-system
data:
  backupTarget: "nfs://192.168.1.100:/mnt/k3s-backups"
EOF

Storage Comparison

Selection guide:

Do you need high availability?
  ├── NO → Use Local Path
  │
  └── YES → Do you have 3+ nodes?
      ├── NO → Use NFS or external storage
      └── YES → Use Longhorn

---

Networking, Ingress and Load Balancing

K3s includes built-in networking components that make it simple to expose services and route traffic. Understanding these components is essential for deploying accessible applications.

K3s Networking Architecture

K3s uses several components for networking:

• Flannel (CNI): Pod-to-pod networking across nodes
• CoreDNS: Service discovery via DNS
• Traefik: HTTP/HTTPS ingress controller
• ServiceLB: Load balancer for external access

Traffic flow:

Internet/External Client
        ↓
    Traefik (Ingress)
    Port 80/443
        ↓
    Service (ClusterIP/LoadBalancer)
        ↓
    Pod Network (Flannel CNI)
        ↓
    Application Container

Service Types

Kubernetes provides different service types for different access patterns:

ClusterIP (default):

• Internal cluster access only
• Service gets internal IP
• Pods can reach via DNS name

NodePort:

• Exposes service on each node’s IP at a static port
• External access via NODE_IP:NODE_PORT
• Port range: 30000-32767

LoadBalancer:

• Creates external load balancer (cloud provider or ServiceLB in K3s)
• Gets external IP for access
• Ideal for production services

Traefik Ingress Controller

Traefik is a modern reverse proxy and load balancer pre-installed in K3s. It automatically discovers services and routes HTTP/HTTPS traffic.

Verify Traefik Installation

# Check Traefik deployment
kubectl get pods -n kube-system | grep traefik

# Check Traefik service
kubectl get svc -n kube-system traefik

# Output shows:
# NAME      TYPE           CLUSTER-IP    EXTERNAL-IP   PORT(S)
# traefik   LoadBalancer   10.43.x.x     NODE_IP       80:30XXX/TCP,443:30XXX/TCP

Basic Ingress Example

Create an ingress to route traffic by hostname:

# basic-ingress.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
      - name: nginx
        image: nginx:alpine
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: web-app-svc
spec:
  selector:
    app: web
  ports:
  - port: 80
    targetPort: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: web-app-ingress
spec:
  rules:
  - host: web.local
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: web-app-svc
            port:
              number: 80

Deploy and test:

# Apply manifests
kubectl apply -f basic-ingress.yaml

# Add to /etc/hosts for testing
echo "127.0.0.1 web.local" | sudo tee -a /etc/hosts

# Test the ingress
curl http://web.local

# Should return nginx welcome page

Path-Based Routing

Route different paths to different services:

# multi-path-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: multi-path
spec:
  rules:
  - host: api.local
    http:
      paths:
      - path: /v1
        pathType: Prefix
        backend:
          service:
            name: api-v1
            port:
              number: 8080
      - path: /v2
        pathType: Prefix
        backend:
          service:
            name: api-v2
            port:
              number: 8080
      - path: /admin
        pathType: Prefix
        backend:
          service:
            name: admin-panel
            port:
              number: 3000

HTTPS/TLS with cert-manager

For production deployments, secure your ingress with TLS certificates.

Install cert-manager:

# Install cert-manager
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.13.0/cert-manager.yaml

# Wait for cert-manager pods
kubectl wait --for=condition=ready pod -l app.kubernetes.io/instance=cert-manager -n cert-manager --timeout=120s

Create Let’s Encrypt issuer:

# letsencrypt-issuer.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: your-[email protected]  # Change this!
    privateKeySecretRef:
      name: letsencrypt-prod-key
    solvers:
    - http01:
        ingress:
          class: traefik

Secure ingress with TLS:

# secure-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: secure-app
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
  tls:
  - hosts:
    - myapp.example.com
    secretName: myapp-tls
  rules:
  - host: myapp.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: my-app
            port:
              number: 80

Apply and verify:

kubectl apply -f letsencrypt-issuer.yaml
kubectl apply -f secure-ingress.yaml

# Watch certificate creation
kubectl get certificate -w

# Once ready, access via HTTPS
curl https://myapp.example.com

Traefik Middleware

Add functionality like authentication, rate limiting, and request transformation:

Basic Authentication:

# auth-middleware.yaml
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: basic-auth
spec:
  basicAuth:
    secret: authsecret
---
apiVersion: v1
kind: Secret
metadata:
  name: authsecret
type: Opaque
data:
  # Generated with: htpasswd -nb admin password | base64
  users: YWRtaW46JGFwcjEkSDY1dnp1Nk4kSkc1RXFLNEZaY2FyenNPZ0hzamZhLwoK
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: protected-app
  annotations:
    traefik.ingress.kubernetes.io/router.middlewares: default-basic-auth@kubernetescrd
spec:
  rules:
  - host: admin.local
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: admin-panel
            port:
              number: 80

Rate Limiting:

# rate-limit-middleware.yaml
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: rate-limit
spec:
  rateLimit:
    average: 100  # Requests per period
    burst: 50     # Burst capacity
    period: 1m    # Time period
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: api-rate-limited
  annotations:
    traefik.ingress.kubernetes.io/router.middlewares: default-rate-limit@kubernetescrd
spec:
  rules:
  - host: api.local
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: api-service
            port:
              number: 8080

ServiceLB Load Balancer

K3s includes ServiceLB (formerly Klipper) that provides LoadBalancer services without requiring a cloud provider.

How ServiceLB Works

When you create a LoadBalancer service:

1. ServiceLB automatically assigns an external IP (uses node IP)
2. Creates a DaemonSet to proxy traffic
3. Distributes traffic to backend pods

Example LoadBalancer service:

# loadbalancer-example.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
    spec:
      containers:
      - name: api
        image: kennethreitz/httpbin
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: api-lb
spec:
  type: LoadBalancer
  ports:
  - port: 8080
    targetPort: 80
    protocol: TCP
  selector:
    app: api

Deploy and test:

kubectl apply -f loadbalancer-example.yaml

# Get external IP
kubectl get svc api-lb

# Output shows:
# NAME     TYPE           CLUSTER-IP    EXTERNAL-IP     PORT(S)
# api-lb   LoadBalancer   10.43.x.x     192.168.1.100   8080:30XXX/TCP

# Test the service
EXTERNAL_IP=$(kubectl get svc api-lb -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
curl http://$EXTERNAL_IP:8080/get

# Should return JSON response from httpbin

DNS and Service Discovery

CoreDNS provides automatic service discovery within the cluster.

Service DNS naming:

# Format:
SERVICE_NAME.NAMESPACE.svc.cluster.local

# Examples:
# Service "mysql" in namespace "default"
mysql.default.svc.cluster.local

# Service "api" in namespace "production"
api.production.svc.cluster.local

# Short form within same namespace:
mysql

Test DNS resolution:

# Create a debug pod
kubectl run -it --rm debug --image=busybox:1.36 --restart=Never -- /bin/sh

# Inside the pod, test DNS
nslookup kubernetes.default
nslookup kube-dns.kube-system

# Test a service (if you have the api-lb service from earlier)
nslookup api-lb.default.svc.cluster.local

# Exit
exit

Practical Exercise: Complete Ingress Setup

Deploy a multi-service application with ingress routing:

# complete-ingress-example.yaml
# Web frontend
apiVersion: apps/v1
kind: Deployment
metadata:
  name: frontend
spec:
  replicas: 2
  selector:
    matchLabels:
      app: frontend
  template:
    metadata:
      labels:
        app: frontend
    spec:
      containers:
      - name: nginx
        image: nginx:alpine
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: frontend
spec:
  selector:
    app: frontend
  ports:
  - port: 80
---
# API backend
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 3
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
    spec:
      containers:
      - name: api
        image: kennethreitz/httpbin
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: api
spec:
  selector:
    app: api
  ports:
  - port: 8080
    targetPort: 80
---
# Ingress with multiple routes
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app-ingress
spec:
  rules:
  - host: myapp.local
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: frontend
            port:
              number: 80
      - path: /api
        pathType: Prefix
        backend:
          service:
            name: api
            port:
              number: 8080

Deploy and test:

# Deploy all resources
kubectl apply -f complete-ingress-example.yaml

# Add to /etc/hosts
echo "127.0.0.1 myapp.local" | sudo tee -a /etc/hosts

# Test frontend
curl http://myapp.local/

# Test API
curl http://myapp.local/api/get

# View Traefik dashboard
kubectl port-forward -n kube-system deployment/traefik 9000:9000
# Visit http://localhost:9000/dashboard/

---

Application Deployment with Helm

Helm is the package manager for Kubernetes, simplifying application deployment and management. It’s essential for efficient homelab operations.

Helm Fundamentals

What is Helm?

Helm packages Kubernetes manifests into reusable “charts” that can be:

• Deployed with a single command
• Versioned and rolled back
• Configured with values files
• Shared via repositories

Key concepts:

• Chart: Package of Kubernetes manifests (like a .deb or .rpm)
• Release: Deployed instance of a chart
• Repository: Collection of charts (like apt repositories)
• Values: Configuration parameters for customization

Installing Helm

# Download and install Helm 3
curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

# Verify installation
helm version

# Expected output:
# version.BuildInfo{Version:"v3.13.x", ...}

Working with Helm Charts

Adding Repositories

# Add popular Helm repositories
helm repo add stable https://charts.helm.sh/stable
helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo add jetstack https://charts.jetstack.io
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

# Update repositories
helm repo update

# List repositories
helm repo list

Searching for Charts

# Search for charts
helm search repo nginx

# Search with versions
helm search repo nginx --versions

# Show chart information
helm show chart bitnami/nginx

# View default values
helm show values bitnami/nginx

Installing Charts

Basic installation:

# Install with default values
helm install my-nginx bitnami/nginx

# Install in specific namespace
helm install my-nginx bitnami/nginx --namespace web --create-namespace

# Dry run (preview what will be created)
helm install my-nginx bitnami/nginx --dry-run --debug

Custom installation:

# Override values via command line
helm install my-db bitnami/mysql   --set auth.rootPassword=supersecret   --set auth.database=myapp   --set primary.persistence.size=20Gi

# Or use a values file
cat > custom-values.yaml <<EOF
auth:
  rootPassword: supersecret
  database: myapp
primary:
  persistence:
    size: 20Gi
EOF

helm install my-db bitnami/mysql -f custom-values.yaml

Managing Releases

# List releases
helm list
helm list -A  # All namespaces

# Check release status
helm status my-nginx

# Get release values
helm get values my-nginx

# View release history
helm history my-nginx

# Upgrade release
helm upgrade my-nginx bitnami/nginx --set replicaCount=3

# Rollback to previous release
helm rollback my-nginx

# Rollback to specific revision
helm rollback my-nginx 2

# Uninstall release
helm uninstall my-nginx

Popular Charts for Homelabs

WordPress Blog

Complete blog platform with MySQL:

helm install my-blog bitnami/wordpress   --namespace blog --create-namespace   --set wordpressUsername=admin   --set wordpressPassword=MySecurePass123   --set [email protected]   --set wordpressBlogName="My Homelab Blog"   --set service.type=LoadBalancer   --set mariadb.auth.rootPassword=DBRootPass123   --set mariadb.auth.password=DBPass123   --set persistence.size=20Gi

# Get credentials
kubectl get secret --namespace blog my-blog-wordpress -o jsonpath="{.data.wordpress-password}" | base64 -d

# Access blog
kubectl get svc --namespace blog my-blog-wordpress
# Visit EXTERNAL-IP in browser

Nextcloud Personal Cloud

Self-hosted file sync and collaboration:

helm install nextcloud bitnami/nextcloud   --namespace cloud --create-namespace   --set nextcloudUsername=admin   --set nextcloudPassword=admin123   --set [email protected]   --set service.type=LoadBalancer   --set persistence.size=100Gi   --set postgresql.auth.password=postgres123

# Access Nextcloud
kubectl get svc -n cloud nextcloud

Gitea Git Server

Self-hosted Git service:

helm install gitea bitnami/gitea   --namespace git --create-namespace   --set service.type=LoadBalancer   --set postgresql.auth.password=gitea123   --set persistence.size=20Gi

# Access Gitea
kubectl get svc -n git gitea

PostgreSQL Database

Production-grade PostgreSQL:

helm install postgres bitnami/postgresql   --namespace databases --create-namespace   --set global.postgresql.auth.postgresPassword=postgres123   --set global.postgresql.auth.username=myapp   --set global.postgresql.auth.password=myapp123   --set global.postgresql.auth.database=myapp   --set primary.persistence.size=20Gi

# Get connection info
export POSTGRES_PASSWORD=$(kubectl get secret --namespace databases postgres-postgresql -o jsonpath="{.data.postgres-password}" | base64 -d)
echo "Connection: postgresql://postgres:$POSTGRES_PASSWORD@postgres-postgresql.databases.svc.cluster.local:5432/myapp"

MinIO S3-Compatible Storage

Self-hosted object storage:

helm install minio bitnami/minio   --namespace storage --create-namespace   --set auth.rootUser=admin   --set auth.rootPassword=minio123   --set persistence.size=100Gi   --set service.type=LoadBalancer

# Access MinIO console
kubectl get svc -n storage minio

Creating Custom Helm Charts

For your own applications, create custom Helm charts.

Chart Structure

# Create a new chart
helm create myapp

# Structure:
myapp/
├── Chart.yaml              # Chart metadata
├── values.yaml             # Default configuration
├── templates/
│   ├── deployment.yaml     # Deployment template
│   ├── service.yaml        # Service template
│   ├── ingress.yaml        # Ingress template
│   ├── _helpers.tpl        # Template helpers
│   └── NOTES.txt           # Post-install notes
└── .helmignore

Example Custom Chart

Chart.yaml:

apiVersion: v2
name: myapp
description: My custom application
type: application
version: 0.1.0
appVersion: "1.0.0"
keywords:
  - myapp
  - custom
maintainers:
  - name: Your Name
    email: [email protected]

values.yaml:

replicaCount: 3

image:
  repository: myregistry/myapp
  tag: "1.0.0"
  pullPolicy: IfNotPresent

service:
  type: LoadBalancer
  port: 80
  targetPort: 8080

ingress:
  enabled: true
  hosts:
    - host: myapp.local
      paths:
        - path: /
          pathType: Prefix

persistence:
  enabled: true
  storageClass: longhorn
  size: 10Gi
  mountPath: /data

resources:
  requests:
    cpu: 250m
    memory: 256Mi
  limits:
    cpu: 500m
    memory: 512Mi

autoscaling:
  enabled: true
  minReplicas: 2
  maxReplicas: 5
  targetCPUUtilizationPercentage: 70

templates/deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: [[ include "myapp.fullname" . ]]
  labels:
    [[- include "myapp.labels" . | nindent 4 ]]
spec:
  [[- if not .Values.autoscaling.enabled ]]
  replicas: [[ .Values.replicaCount ]]
  [[- end ]]
  selector:
    matchLabels:
      [[- include "myapp.selectorLabels" . | nindent 6 ]]
  template:
    metadata:
      labels:
        [[- include "myapp.selectorLabels" . | nindent 8 ]]
    spec:
      containers:
      - name: [[ .Chart.Name ]]
        image: "[[ .Values.image.repository ]]:[[ .Values.image.tag | default .Chart.AppVersion ]]"
        imagePullPolicy: [[ .Values.image.pullPolicy ]]
        ports:
        - name: http
          containerPort: [[ .Values.service.targetPort ]]
          protocol: TCP
        resources:
          [[- toYaml .Values.resources | nindent 12 ]]
        [[- if .Values.persistence.enabled ]]
        volumeMounts:
        - name: data
          mountPath: [[ .Values.persistence.mountPath ]]
        [[- end ]]
      [[- if .Values.persistence.enabled ]]
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: [[ include "myapp.fullname" . ]]-pvc
      [[- end ]]

Testing and Deploying Custom Charts

# Validate chart syntax
helm lint myapp/

# Test template rendering
helm template myapp myapp/ --debug

# Dry run installation
helm install test-release myapp/ --dry-run --debug

# Install the chart
helm install my-release myapp/ --namespace myapp --create-namespace

# Override values
helm install my-release myapp/   --set replicaCount=5   --set persistence.size=20Gi

Helm Hooks

Execute tasks at specific points in the release lifecycle:

# templates/pre-install-job.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: [[ include "myapp.fullname" . ]]-migration
  annotations:
    helm.sh/hook: pre-install,pre-upgrade
    helm.sh/hook-weight: "-5"
    helm.sh/hook-delete-policy: before-hook-creation
spec:
  template:
    spec:
      containers:
      - name: migration
        image: "[[ .Values.image.repository ]]:[[ .Values.image.tag ]]"
        command: ["/bin/sh", "-c", "database-migration.sh"]
      restartPolicy: Never
  backoffLimit: 3

Available hooks:

• pre-install: Before resources are created
• post-install: After all resources are loaded
• pre-upgrade: Before upgrade
• post-upgrade: After upgrade
• pre-delete: Before deletion
• post-delete: After deletion
• pre-rollback: Before rollback
• post-rollback: After rollback

---

Monitoring and Observability

Effective monitoring is crucial for understanding cluster health, identifying issues, and optimizing performance. Prometheus and Grafana provide enterprise-grade observability for K3s.

Monitoring Architecture

Components:

• Prometheus: Time-series database for metrics storage
• Grafana: Visualization and dashboarding
• AlertManager: Alert routing and notification
• Node Exporter: Node-level metrics (CPU, memory, disk)
• kube-state-metrics: Kubernetes object metrics
• Prometheus Operator: Manages Prometheus deployments

Data flow:

Applications & Infrastructure
    ↓ (expose metrics)
Prometheus (scrapes /metrics endpoints)
    ↓ (stores time-series)
Time-Series Database
    ↓ (queries with PromQL)
Grafana (visualization)
AlertManager (notifications)

Installing the Prometheus Stack

The easiest way to get comprehensive monitoring is via the kube-prometheus-stack Helm chart:

# Add Prometheus community repository
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

# Create monitoring namespace
kubectl create namespace monitoring

# Install kube-prometheus-stack
helm install prometheus prometheus-community/kube-prometheus-stack   --namespace monitoring   --set prometheus.prometheusSpec.retention=30d   --set prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.resources.requests.storage=50Gi   --set grafana.adminPassword=admin   --set grafana.persistence.enabled=true   --set grafana.persistence.size=10Gi

# This installs:
# - Prometheus server
# - Grafana
# - AlertManager
# - Node exporter (on all nodes)
# - kube-state-metrics
# - Prometheus operator

Verify installation:

# Check all monitoring pods
kubectl get pods -n monitoring

# Should see:
# - prometheus-prometheus-kube-prometheus-prometheus-0
# - prometheus-grafana-xxx
# - prometheus-kube-prometheus-operator-xxx
# - prometheus-kube-state-metrics-xxx
# - prometheus-prometheus-node-exporter-xxx (one per node)
# - alertmanager-prometheus-kube-prometheus-alertmanager-0

Accessing Prometheus and Grafana

Port Forwarding

# Access Prometheus
kubectl port-forward -n monitoring svc/prometheus-kube-prometheus-prometheus 9090:9090 &

# Visit http://localhost:9090
# - Execute PromQL queries
# - View targets and alerts
# - Check service discovery

# Access Grafana
kubectl port-forward -n monitoring svc/prometheus-grafana 3000:80 &

# Visit http://localhost:3000
# Default credentials: admin / admin (or the password you set)

# Access AlertManager
kubectl port-forward -n monitoring svc/prometheus-kube-prometheus-alertmanager 9093:9093 &

# Visit http://localhost:9093

Via Ingress

For permanent access, create ingress resources:

# monitoring-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: monitoring
  namespace: monitoring
spec:
  rules:
  - host: grafana.local
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: prometheus-grafana
            port:
              number: 80
  - host: prometheus.local
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: prometheus-kube-prometheus-prometheus
            port:
              number: 9090
  - host: alertmanager.local
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: prometheus-kube-prometheus-alertmanager
            port:
              number: 9093

Apply and add to /etc/hosts:

kubectl apply -f monitoring-ingress.yaml

# Add to /etc/hosts
echo "127.0.0.1 grafana.local prometheus.local alertmanager.local" | sudo tee -a /etc/hosts

Key Metrics to Monitor

Cluster-Level Metrics

Essential PromQL queries for cluster monitoring:

# Node CPU usage (percentage)
100 * (1 - avg by (instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])))

# Node memory usage (percentage)
100 * (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes))

# Node disk usage (percentage)
100 * (1 - (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}))

# Pod count by namespace
count by (namespace) (kube_pod_info)

# Container restart rate
rate(kube_pod_container_status_restarts_total[1h])

# Pods not running
count by (namespace) (kube_pod_status_phase{phase!="Running"} == 1)

Application Metrics

# Pod CPU usage
sum(rate(container_cpu_usage_seconds_total{container!=""}[5m])) by (pod, namespace)

# Pod memory usage
sum(container_memory_working_set_bytes{container!=""}) by (pod, namespace)

# Pod network received
sum(rate(container_network_receive_bytes_total[5m])) by (pod, namespace)

# Pod network transmitted
sum(rate(container_network_transmit_bytes_total[5m])) by (pod, namespace)

Creating Dashboards in Grafana

Import Pre-Built Dashboards

Grafana has thousands of community dashboards:

# Popular dashboard IDs for K3s:
# 15282 - K3s Cluster Monitoring
# 15398 - Kubernetes Cluster Monitoring (via Prometheus)
# 12006 - Kubernetes API Server
# 1860 - Node Exporter Full

# In Grafana UI:
# 1. Click "+" → "Import"
# 2. Enter dashboard ID (e.g., 15282)
# 3. Select "Prometheus" as data source
# 4. Click "Import"

Create Custom Dashboard

Example: Simple cluster overview

# In Grafana UI:
# 1. Click "+" → "Dashboard"
# 2. Click "Add visualization"
# 3. Select "Prometheus" data source
# 4. Enter PromQL query
# 5. Configure visualization type
# 6. Add more panels
# 7. Save dashboard

# Example panels:

# Panel 1: Cluster CPU Usage
# Query: sum(rate(container_cpu_usage_seconds_total[5m]))
# Visualization: Gauge

# Panel 2: Memory Usage by Namespace
# Query: sum(container_memory_working_set_bytes) by (namespace)
# Visualization: Bar chart

# Panel 3: Pod Count
# Query: count(kube_pod_info)
# Visualization: Stat

# Panel 4: Network Traffic
# Query: sum(rate(container_network_receive_bytes_total[5m]))
# Visualization: Time series

Alerting

Configure alerts to proactively identify issues:

Create Alert Rules

# k3s-alerts.yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: k3s-alerts
  namespace: monitoring
  labels:
    prometheus: kube-prometheus
spec:
  groups:
  - name: k3s-cluster
    interval: 30s
    rules:
    # Alert: High node CPU
    - alert: HighNodeCPU
      expr: 100 * (1 - avg by (instance) (irate(node_cpu_seconds_total{mode="idle"}[5m]))) > 80
      for: 10m
      labels:
        severity: warning
      annotations:
        summary: "High CPU usage on [[ $labels.instance ]]"
        description: "Node [[ $labels.instance ]] has CPU usage above 80% for 10 minutes"

    # Alert: High node memory
    - alert: HighNodeMemory
      expr: 100 * (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) > 85
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "High memory usage on [[ $labels.instance ]]"
        description: "Node [[ $labels.instance ]] has memory usage above 85%"

    # Alert: Pod crash looping
    - alert: PodCrashLooping
      expr: rate(kube_pod_container_status_restarts_total[15m]) > 0
      for: 5m
      labels:
        severity: critical
      annotations:
        summary: "Pod [[ $labels.namespace ]]/[[ $labels.pod ]] is crash looping"
        description: "Pod has restarted [[ $value ]] times in the last 15 minutes"

    # Alert: Node not ready
    - alert: NodeNotReady
      expr: kube_node_status_condition{condition="Ready",status="true"} == 0
      for: 5m
      labels:
        severity: critical
      annotations:
        summary: "Node [[ $labels.node ]] is not ready"
        description: "Node has been in NotReady state for 5 minutes"

    # Alert: High disk usage
    - alert: HighDiskUsage
      expr: 100 * (1 - (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"})) > 85
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "High disk usage on [[ $labels.instance ]]"
        description: "Disk usage is above 85% on [[ $labels.instance ]]"

    # Alert: PVC almost full
    - alert: PersistentVolumeUsageHigh
      expr: 100 * (kubelet_volume_stats_used_bytes / kubelet_volume_stats_capacity_bytes) > 80
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "PVC [[ $labels.persistentvolumeclaim ]] is almost full"
        description: "PVC usage is above 80% in namespace [[ $labels.namespace ]]"

Apply the alert rules:

kubectl apply -f k3s-alerts.yaml

# Verify alerts are loaded
kubectl get prometheusrules -n monitoring

# Check in Prometheus UI
# Visit http://localhost:9090/alerts

Configure AlertManager

Send alerts to various destinations:

# alertmanager-config.yaml
apiVersion: v1
kind: Secret
metadata:
  name: alertmanager-prometheus-kube-prometheus-alertmanager
  namespace: monitoring
type: Opaque
stringData:
  alertmanager.yaml: |
    global:
      resolve_timeout: 5m

    route:
      receiver: 'default'
      group_by: ['alertname', 'cluster', 'service']
      group_wait: 10s
      group_interval: 10s
      repeat_interval: 12h
      routes:
      - match:
          severity: critical
        receiver: 'critical'
      - match:
          severity: warning
        receiver: 'warning'

    receivers:
    - name: 'default'
      webhook_configs:
      - url: 'http://webhook-receiver:5000/alerts'

    - name: 'critical'
      # Email configuration
      email_configs:
      - to: '[email protected]'
        from: '[email protected]'
        smarthost: 'smtp.gmail.com:587'
        auth_username: '[email protected]'
        auth_password: 'your-app-password'
        headers:
          Subject: 'CRITICAL: [[ .GroupLabels.alertname ]]'

    - name: 'warning'
      # Slack webhook (optional)
      slack_configs:
      - api_url: 'https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK'
        channel: '#alerts'
        title: 'Warning Alert'
        text: '[[ range .Alerts ]][[ .Annotations.description ]][[ end ]]'

Apply configuration:

kubectl apply -f alertmanager-config.yaml

# Restart AlertManager to pick up changes
kubectl rollout restart statefulset/alertmanager-prometheus-kube-prometheus-alertmanager -n monitoring

K3s-Specific Monitoring

K3s requires some additional configuration to expose control plane metrics:

# Edit K3s service to expose metrics
sudo vi /etc/systemd/system/k3s.service

# Add these flags to ExecStart:
--kube-controller-manager-arg='bind-address=0.0.0.0' --kube-proxy-arg='metrics-bind-address=0.0.0.0' --kube-scheduler-arg='bind-address=0.0.0.0'

# Reload and restart
sudo systemctl daemon-reload
sudo systemctl restart k3s

# Verify metrics are accessible
curl http://127.0.0.1:10257/metrics  # Controller manager
curl http://127.0.0.1:10259/metrics  # Scheduler

---

AI Automation and Intelligent Management

Artificial intelligence and machine learning can significantly enhance K3s cluster operations through predictive scaling, anomaly detection, and automated remediation.

Introduction to AI in Kubernetes

Benefits of AI automation:

• Predictive scaling: Scale resources before demand spikes
• Anomaly detection: Identify unusual patterns automatically
• Cost optimization: Right-size resources based on actual usage
• Self-healing: Automatic remediation of common issues
• Capacity planning: Forecast future resource needs

AI/ML use cases in K3s:

1. Load prediction and proactive scaling
2. Anomaly detection in metrics and logs
3. Intelligent pod placement and scheduling
4. Resource optimization recommendations
5. Security threat detection
6. Automated troubleshooting

Intelligent Pod Autoscaling

Vertical Pod Autoscaler (VPA)

VPA automatically adjusts pod resource requests based on actual usage:

# Install VPA
git clone https://github.com/kubernetes/autoscaler.git
cd autoscaler/vertical-pod-autoscaler
./hack/vpa-up.sh

# Or via Helm (if available)
helm install vpa fairwinds-stable/vpa --namespace vpa --create-namespace

Configure VPA for a deployment:

# vpa-example.yaml
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: myapp-vpa
spec:
  targetRef:
    apiVersion: "apps/v1"
    kind: Deployment
    name: myapp
  updatePolicy:
    updateMode: "Auto"  # Options: Off, Initial, Recreate, Auto
  resourcePolicy:
    containerPolicies:
    - containerName: '*'
      minAllowed:
        cpu: 100m
        memory: 128Mi
      maxAllowed:
        cpu: 2000m
        memory: 2Gi

Deploy and observe:

kubectl apply -f vpa-example.yaml

# Check VPA recommendations
kubectl describe vpa myapp-vpa

# Output shows:
# - Current resource usage
# - Recommended requests
# - VPA will automatically update pod requests

Horizontal Pod Autoscaler with Custom Metrics

Scale based on custom application metrics:

# hpa-custom-metrics.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: myapp-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: myapp
  minReplicas: 2
  maxReplicas: 10
  metrics:
  # CPU-based scaling
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  # Memory-based scaling
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 80
  # Custom metric (requires metrics server)
  - type: Pods
    pods:
      metric:
        name: http_requests_per_second
      target:
        type: AverageValue
        averageValue: "1000"
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
      - type: Percent
        value: 50
        periodSeconds: 60
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
      - type: Percent
        value: 100
        periodSeconds: 30
      - type: Pods
        value: 2
        periodSeconds: 30

Anomaly Detection

Detect unusual cluster behavior using machine learning:

Python-based anomaly detector:

# anomaly_detector.py
import numpy as np
from sklearn.ensemble import IsolationForest
from prometheus_api_client import PrometheusConnect
import time

class K3sAnomalyDetector:
    def __init__(self, prometheus_url='http://localhost:9090'):
        self.prom = PrometheusConnect(url=prometheus_url, disable_ssl=True)
        self.detector = IsolationForest(contamination=0.05, random_state=42)
        self.trained = False

    def fetch_metrics(self, query, duration='1h'):
        """Fetch metrics from Prometheus"""
        result = self.prom.custom_query_range(
            query=query,
            start_time=(time.time() - 3600),
            end_time=time.time(),
            step=60
        )
        # Extract values
        values = [float(point[1]) for metric in result for point in metric['values']]
        return np.array(values).reshape(-1, 1)

    def train(self):
        """Train anomaly detector on historical data"""
        # Fetch various metrics
        cpu_data = self.fetch_metrics('sum(rate(container_cpu_usage_seconds_total[5m]))')
        mem_data = self.fetch_metrics('sum(container_memory_working_set_bytes)')
        net_data = self.fetch_metrics('sum(rate(container_network_receive_bytes_total[5m]))')

        # Combine features
        X = np.column_stack([cpu_data, mem_data, net_data])

        # Train detector
        self.detector.fit(X)
        self.trained = True
        print("Anomaly detector trained successfully")

    def detect_anomalies(self):
        """Detect current anomalies"""
        if not self.trained:
            print("Detector not trained yet")
            return []

        # Fetch current metrics
        current_cpu = self.fetch_metrics('sum(rate(container_cpu_usage_seconds_total[5m]))', '5m')
        current_mem = self.fetch_metrics('sum(container_memory_working_set_bytes)', '5m')
        current_net = self.fetch_metrics('sum(rate(container_network_receive_bytes_total[5m]))', '5m')

        # Combine current features
        X_current = np.column_stack([current_cpu[-1], current_mem[-1], current_net[-1]])

        # Predict
        prediction = self.detector.predict(X_current)

        anomalies = []
        if prediction[0] == -1:
            anomalies.append({
                'timestamp': time.time(),
                'cpu': current_cpu[-1][0],
                'memory': current_mem[-1][0],
                'network': current_net[-1][0],
                'severity': 'warning'
            })

        return anomalies

    def run_continuous_monitoring(self, interval=60):
        """Continuously monitor for anomalies"""
        print("Starting continuous anomaly detection...")

        while True:
            anomalies = self.detect_anomalies()

            if anomalies:
                for anomaly in anomalies:
                    print(f"ANOMALY DETECTED at {time.ctime(anomaly['timestamp'])}")
                    print(f"  CPU: {anomaly['cpu']:.2f}")
                    print(f"  Memory: {anomaly['memory'] / 1e9:.2f} GB")
                    print(f"  Network: {anomaly['network']:.2f} bytes/s")

                    # Trigger alert or remediation
                    self.trigger_alert(anomaly)

            time.sleep(interval)

    def trigger_alert(self, anomaly):
        """Send alert when anomaly detected"""
        # Implement alerting (email, Slack, webhook, etc.)
        pass

if __name__ == '__main__':
    detector = K3sAnomalyDetector()

    # Train on historical data
    detector.train()

    # Start monitoring
    detector.run_continuous_monitoring(interval=60)

Deploy as a Kubernetes CronJob:

# anomaly-detector-cronjob.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: anomaly-detector
  namespace: monitoring
spec:
  schedule: "*/5 * * * *"  # Every 5 minutes
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: detector
            image: your-registry/anomaly-detector:latest
            env:
            - name: PROMETHEUS_URL
              value: "http://prometheus-kube-prometheus-prometheus.monitoring.svc.cluster.local:9090"
          restartPolicy: OnFailure

Automated Remediation

Self-healing cluster that automatically fixes common issues:

# self_healer.py
import subprocess
import json
import time
import logging

logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
logger = logging.getLogger(__name__)

class K3sSelfHealer:
    def __init__(self):
        self.restart_threshold = 5
        self.cpu_threshold = 90
        self.memory_threshold = 90

    def run_kubectl(self, command):
        """Execute kubectl command"""
        try:
            result = subprocess.run(
                f"kubectl {command}",
                shell=True,
                capture_output=True,
                text=True,
                check=True
            )
            return result.stdout
        except subprocess.CalledProcessError as e:
            logger.error(f"Command failed: {e.stderr}")
            return None

    def check_crash_looping_pods(self):
        """Find pods that are crash looping"""
        output = self.run_kubectl("get pods -A -o json")
        if not output:
            return []

        pods_data = json.loads(output)
        crash_looping = []

        for item in pods_data.get('items', []):
            name = item['metadata']['name']
            namespace = item['metadata']['namespace']
            container_statuses = item.get('status', {}).get('containerStatuses', [])

            for status in container_statuses:
                restart_count = status.get('restartCount', 0)
                if restart_count > self.restart_threshold:
                    crash_looping.append({
                        'name': name,
                        'namespace': namespace,
                        'restarts': restart_count
                    })

        return crash_looping

    def fix_crash_looping_pod(self, pod):
        """Attempt to fix crash looping pod"""
        logger.info(f"Attempting to fix crash looping pod: {pod['namespace']}/{pod['name']}")

        # Get pod details
        details = self.run_kubectl(f"get pod {pod['name']} -n {pod['namespace']} -o json")
        if not details:
            return

        pod_data = json.loads(details)

        # Strategy 1: Check resource constraints
        containers = pod_data['spec']['containers']
        for container in containers:
            resources = container.get('resources', {})
            if not resources.get('limits'):
                logger.info(f"Pod has no resource limits. Consider adding them.")

        # Strategy 2: Check for OOMKilled
        container_statuses = pod_data.get('status', {}).get('containerStatuses', [])
        for status in container_statuses:
            last_state = status.get('lastState', {}).get('terminated', {})
            if last_state.get('reason') == 'OOMKilled':
                logger.warning(f"Pod was OOMKilled. Needs more memory.")
                # Could automatically increase memory here

        # Strategy 3: Delete pod to trigger fresh restart
        logger.info(f"Deleting pod {pod['namespace']}/{pod['name']} for fresh restart")
        self.run_kubectl(f"delete pod {pod['name']} -n {pod['namespace']}")

    def check_node_resources(self):
        """Check for nodes with resource pressure"""
        output = self.run_kubectl("top nodes --no-headers")
        if not output:
            return []

        high_usage = []
        for line in output.strip().split('\n'):
            parts = line.split()
            if len(parts) >= 5:
                node = parts[0]
                cpu_pct = int(parts[1].rstrip('%'))
                mem_pct = int(parts[3].rstrip('%'))

                if cpu_pct > self.cpu_threshold or mem_pct > self.memory_threshold:
                    high_usage.append({
                        'node': node,
                        'cpu': cpu_pct,
                        'memory': mem_pct
                    })

        return high_usage

    def rebalance_node(self, node_info):
        """Rebalance pods on high-usage node"""
        logger.info(f"Rebalancing node {node_info['node']}")

        # Cordon node
        self.run_kubectl(f"cordon {node_info['node']}")

        # Get low-priority pods
        pods_output = self.run_kubectl(
            f"get pods --field-selector spec.nodeName={node_info['node']} -A -o json"
        )

        if pods_output:
            pods_data = json.loads(pods_output)
            # Delete lowest priority pods to trigger rescheduling
            for item in pods_data.get('items', [])[:3]:  # Limit to 3 pods
                name = item['metadata']['name']
                namespace = item['metadata']['namespace']
                logger.info(f"Deleting pod {namespace}/{name} for rebalancing")
                self.run_kubectl(f"delete pod {name} -n {namespace}")

        # Wait a bit then uncordon
        time.sleep(30)
        self.run_kubectl(f"uncordon {node_info['node']}")

    def check_disk_usage(self):
        """Check for high disk usage"""
        output = self.run_kubectl("get pv -o json")
        if not output:
            return []

        pv_data = json.loads(output)
        high_usage = []

        for item in pv_data.get('items', []):
            name = item['metadata']['name']
            capacity = item['spec']['capacity']['storage']
            # Would need metrics to get actual usage
            # This is a simplified check

        return high_usage

    def run_health_checks(self):
        """Run all health checks and fix issues"""
        logger.info("Running cluster health checks...")

        # Check 1: Crash looping pods
        crash_looping = self.check_crash_looping_pods()
        if crash_looping:
            logger.warning(f"Found {len(crash_looping)} crash looping pods")
            for pod in crash_looping:
                self.fix_crash_looping_pod(pod)

        # Check 2: Node resource pressure
        high_usage_nodes = self.check_node_resources()
        if high_usage_nodes:
            logger.warning(f"Found {len(high_usage_nodes)} nodes with high resource usage")
            for node in high_usage_nodes:
                self.rebalance_node(node)

        # Check 3: Disk usage
        # high_disk = self.check_disk_usage()

        logger.info("Health checks completed")

    def run_continuous_healing(self, interval=300):
        """Continuously monitor and heal cluster"""
        logger.info(f"Starting continuous healing with {interval}s interval")

        while True:
            self.run_health_checks()
            time.sleep(interval)

if __name__ == '__main__':
    healer = K3sSelfHealer()
    healer.run_continuous_healing(interval=300)  # Every 5 minutes

Deploy as Kubernetes Deployment:

# self-healer-deployment.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: self-healer
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: self-healer
rules:
- apiGroups: [""]
  resources: ["pods", "nodes"]
  verbs: ["get", "list", "watch", "delete", "patch"]
- apiGroups: [""]
  resources: ["pods/eviction"]
  verbs: ["create"]
- apiGroups: ["apps"]
  resources: ["deployments", "statefulsets"]
  verbs: ["get", "list", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: self-healer
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: self-healer
subjects:
- kind: ServiceAccount
  name: self-healer
  namespace: kube-system
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: self-healer
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: self-healer
  template:
    metadata:
      labels:
        app: self-healer
    spec:
      serviceAccountName: self-healer
      containers:
      - name: healer
        image: your-registry/k3s-self-healer:latest
        imagePullPolicy: Always
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 500m
            memory: 256Mi

Cost Optimization with AI

Analyze and optimize resource utilization:

# cost_optimizer.py
import subprocess
import json
from collections import defaultdict

class K3sCostOptimizer:
    def analyze_cluster(self):
        """Analyze cluster for cost optimization opportunities"""
        recommendations = []

        # Check 1: Overprovisioned pods
        recommendations.extend(self.find_overprovisioned_pods())

        # Check 2: Unused persistent volumes
        recommendations.extend(self.find_unused_pvs())

        # Check 3: Idle deployments
        recommendations.extend(self.find_idle_deployments())

        return recommendations

    def find_overprovisioned_pods(self):
        """Find pods with excessive resource requests"""
        recommendations = []

        # Get pod metrics
        result = subprocess.run(
            "kubectl top pods -A --no-headers",
            shell=True,
            capture_output=True,
            text=True
        )

        for line in result.stdout.strip().split('\n'):
            parts = line.split()
            if len(parts) >= 4:
                namespace = parts[0]
                pod = parts[1]
                cpu = int(parts[2].rstrip('m'))
                memory = int(parts[3].rstrip('Mi'))

                # Get pod spec
                spec = subprocess.run(
                    f"kubectl get pod {pod} -n {namespace} -o json",
                    shell=True,
                    capture_output=True,
                    text=True
                )

                pod_data = json.loads(spec.stdout)
                containers = pod_data['spec']['containers']

                for container in containers:
                    requests = container.get('resources', {}).get('requests', {})
                    requested_cpu = int(requests.get('cpu', '0').rstrip('m'))
                    requested_mem = int(requests.get('memory', '0').rstrip('Mi'))

                    # If using less than 30% of requested resources
                    if requested_cpu &gt; 0 and cpu < requested_cpu * 0.3:
                        recommendations.append({
                            'type': 'overprovisioned_cpu',
                            'pod': f"{namespace}/{pod}",
                            'container': container['name'],
                            'requested': f"{requested_cpu}m",
                            'actual': f"{cpu}m",
                            'recommendation': f"Reduce CPU request to {int(cpu * 1.2)}m"
                        })

                    if requested_mem &gt; 0 and memory < requested_mem * 0.3:
                        recommendations.append({
                            'type': 'overprovisioned_memory',
                            'pod': f"{namespace}/{pod}",
                            'container': container['name'],
                            'requested': f"{requested_mem}Mi",
                            'actual': f"{memory}Mi",
                            'recommendation': f"Reduce memory request to {int(memory * 1.2)}Mi"
                        })

        return recommendations

    def find_unused_pvs(self):
        """Find persistent volumes with low usage"""
        recommendations = []

        # Would need to query volume metrics
        # This is a simplified example

        return recommendations

    def find_idle_deployments(self):
        """Find deployments with no traffic"""
        recommendations = []

        # Would need to query traffic metrics from ingress/service mesh
        # This is a simplified example

        return recommendations

    def generate_report(self):
        """Generate optimization report"""
        recommendations = self.analyze_cluster()

        print("=" * 60)
        print("K3s Cost Optimization Report")
        print("=" * 60)
        print()

        if not recommendations:
            print("No optimization opportunities found. Cluster is well-optimized!")
            return

        by_type = defaultdict(list)
        for rec in recommendations:
            by_type[rec['type']].append(rec)

        for rec_type, recs in by_type.items():
            print(f"\n{rec_type.replace('_', ' ').title()}")
            print("-" * 60)
            for rec in recs:
                print(f"  Pod: {rec['pod']}")
                print(f"  Container: {rec['container']}")
                print(f"  Current: {rec['actual']} / Requested: {rec['requested']}")
                print(f"  Recommendation: {rec['recommendation']}")
                print()

if __name__ == '__main__':
    optimizer = K3sCostOptimizer()
    optimizer.generate_report()

---

Troubleshooting and Optimization

Effective troubleshooting skills are essential for maintaining healthy K3s clusters.

Common Issues and Solutions

Essential Debugging Commands

# Cluster overview
kubectl cluster-info
kubectl get nodes -o wide
kubectl get pods -A

# Resource usage
kubectl top nodes
kubectl top pods -A
kubectl top pods -A --sort-by=memory
kubectl top pods -A --sort-by=cpu

# Pod diagnostics
kubectl describe pod POD_NAME -n NAMESPACE
kubectl logs POD_NAME -n NAMESPACE
kubectl logs POD_NAME -n NAMESPACE --previous  # Previous crashed container
kubectl logs POD_NAME -c CONTAINER_NAME -n NAMESPACE  # Specific container
kubectl logs -f POD_NAME -n NAMESPACE  # Follow logs

# Execute commands in pods
kubectl exec -it POD_NAME -n NAMESPACE -- /bin/sh
kubectl exec POD_NAME -n NAMESPACE -- env

# Events (crucial for debugging)
kubectl get events -A --sort-by='.lastTimestamp'
kubectl get events -n NAMESPACE --sort-by='.lastTimestamp'

# Deployment troubleshooting
kubectl describe deployment DEPLOYMENT_NAME
kubectl rollout status deployment/DEPLOYMENT_NAME
kubectl rollout history deployment/DEPLOYMENT_NAME
kubectl rollout undo deployment/DEPLOYMENT_NAME

# Service troubleshooting
kubectl describe service SERVICE_NAME
kubectl get endpoints SERVICE_NAME

# Network troubleshooting
kubectl describe ingress INGRESS_NAME
kubectl get networkpolicies -A

# Storage troubleshooting
kubectl get pv
kubectl get pvc -A
kubectl describe pv PV_NAME
kubectl describe pvc PVC_NAME

# Node troubleshooting
kubectl describe node NODE_NAME
kubectl get node NODE_NAME -o yaml

# Check K3s service
sudo systemctl status k3s
sudo journalctl -u k3s -f
sudo journalctl -u k3s -n 100 --no-pager

Performance Optimization

Resource Right-Sizing

Properly set resource requests and limits:

# Example: Well-configured pod resources
apiVersion: apps/v1
kind: Deployment
metadata:
  name: optimized-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: optimized
  template:
    metadata:
      labels:
        app: optimized
    spec:
      containers:
      - name: app
        image: myapp:latest
        resources:
          requests:
            cpu: 250m        # Guaranteed CPU
            memory: 256Mi    # Guaranteed memory
          limits:
            cpu: 500m        # Maximum CPU (throttled if exceeded)
            memory: 512Mi    # Maximum memory (OOMKilled if exceeded)
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /ready
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 5

Guidelines:

• Set requests based on average usage
• Set limits 2x requests for CPU, 1.5x for memory
• Monitor actual usage with kubectl top
• Adjust over time based on metrics

Node Optimization

# Label nodes for specific workloads
kubectl label nodes NODE_NAME workload-type=compute-intensive

# Taint nodes to dedicate them
kubectl taint nodes NODE_NAME dedicated=gpu:NoSchedule

# Use nodeSelector in pods
spec:
  nodeSelector:
    workload-type: compute-intensive

# Use affinity for advanced placement
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: workload-type
            operator: In
            values:
            - compute-intensive

Storage Optimization

# Clean up unused persistent volumes
kubectl get pv | grep Released
kubectl delete pv PV_NAME

# Clean up completed jobs
kubectl delete jobs --field-selector status.successful=1

# Clean up old pods
kubectl delete pods --field-selector status.phase=Succeeded
kubectl delete pods --field-selector status.phase=Failed

# Check storage usage
df -h /var/lib/rancher/k3s/storage

# Clean Docker images (if using Docker)
docker system prune -a

Diagnostic Scenarios

Scenario 1: Pod Not Starting

# Step 1: Check pod status
kubectl get pod POD_NAME -n NAMESPACE

# Step 2: Describe pod for events
kubectl describe pod POD_NAME -n NAMESPACE

# Common causes and fixes:

# ImagePullBackOff
# - Check image name spelling
# - Verify image exists in registry
# - Check image pull secrets

# Pending (Insufficient resources)
# - Check node resources: kubectl describe nodes
# - Scale down other workloads or add nodes

# Pending (PVC not bound)
# - Check PVC status: kubectl get pvc
# - Verify StorageClass exists
# - Check PV availability

# CrashLoopBackOff
# - Check logs: kubectl logs POD_NAME
# - Check previous logs: kubectl logs POD_NAME --previous
# - Verify application configuration

Scenario 2: Service Not Accessible

# Step 1: Check service
kubectl get svc SERVICE_NAME -n NAMESPACE
kubectl describe svc SERVICE_NAME -n NAMESPACE

# Step 2: Check endpoints
kubectl get endpoints SERVICE_NAME -n NAMESPACE

# If no endpoints:
# - Verify pod selectors match labels
# - Check if pods are running
# - Verify pod ports match service targetPort

# Step 3: Test from within cluster
kubectl run -it --rm debug --image=busybox --restart=Never -- /bin/sh
wget -O- http://SERVICE_NAME.NAMESPACE.svc.cluster.local

# Step 4: Check ingress (if using)
kubectl describe ingress INGRESS_NAME

Scenario 3: High Memory Usage

# Step 1: Identify high memory pods
kubectl top pods -A --sort-by=memory | head -20

# Step 2: Describe high memory pods
kubectl describe pod POD_NAME -n NAMESPACE

# Step 3: Check for memory leaks
kubectl logs POD_NAME -n NAMESPACE | grep -i "out of memory|oom"

# Step 4: Check node memory
kubectl top nodes

# Solutions:
# - Increase pod memory limits
# - Scale horizontally (more replicas, less memory each)
# - Fix memory leaks in application
# - Add more nodes

Cluster Health Checklist

Regular checks to maintain cluster health:

#!/bin/bash
# cluster-health-check.sh

echo "=== K3s Cluster Health Check ==="
echo

# 1. Check nodes
echo "Nodes:"
kubectl get nodes
echo

# 2. Check system pods
echo "System Pods:"
kubectl get pods -n kube-system
echo

# 3. Check resource usage
echo "Node Resources:"
kubectl top nodes
echo

# 4. Check persistent volumes
echo "Persistent Volumes:"
kubectl get pv
echo

# 5. Check for crash looping pods
echo "Crash Looping Pods:"
kubectl get pods -A | grep -i "CrashLoop|Error"
echo

# 6. Check recent events
echo "Recent Events:"
kubectl get events -A --sort-by='.lastTimestamp' | head -20
echo

# 7. Check K3s service
echo "K3s Service:"
systemctl status k3s --no-pager -l
echo

# 8. Check disk space
echo "Disk Space:"
df -h / /var/lib/rancher/k3s/storage
echo

# 9. Check for pending pods
echo "Pending Pods:"
kubectl get pods -A --field-selector=status.phase=Pending
echo

echo "Health check complete!"

Run regularly:

chmod +x cluster-health-check.sh
./cluster-health-check.sh

# Or schedule with cron
crontab -e
# Add: 0 */6 * * * /path/to/cluster-health-check.sh > /var/log/k3s-health.log 2>&1

---

Next Steps

Congratulations on deploying K3s! Here are paths forward to deepen your Kubernetes expertise.

Advanced Topics to Explore

1. Service Mesh (Istio/Linkerd)

• Advanced traffic management
• Mutual TLS (mTLS) security
• Enhanced observability
• Circuit breaking and retries

2. GitOps with ArgoCD

• Infrastructure as code
• Declarative application deployment
• Automatic sync from Git repositories
• Rollback capabilities

3. Backup & Disaster Recovery

• Velero for cluster backups
• etcd/SQLite backup strategies
• Disaster recovery planning
• Multi-cluster failover

4. Security Hardening

• Pod Security Standards (PSS)
• Network Policies
• RBAC best practices
• Secret management with Vault
• Image scanning

5. CI/CD Integration

• Jenkins on K3s
• GitLab CI/CD
• GitHub Actions with K3s
• Tekton pipelines

Practical Project Ideas

Beginner Projects:

1. Personal Blog Platform

   • WordPress on K3s
   • MySQL database with persistent storage
   • Ingress with SSL
   • Automated backups

2. Development Environment

   • Gitea for Git hosting
   • Jenkins for CI/CD
   • SonarQube for code quality
   • Harbor for container registry

Intermediate Projects:

3. Home Media Server

   • Jellyfin for media streaming
   • Sonarr/Radarr for automation
   • Transmission for downloads
   • NFS storage backend

4. Monitoring Stack

   • Prometheus + Grafana
   • Loki for log aggregation
   • AlertManager for notifications
   • Custom dashboards

Advanced Projects:

5. Multi-Service Platform

   • Nextcloud (file sync)
   • Gitea (Git)
   • Rocket.Chat (communication)
   • Shared authentication (OAuth/LDAP)
   • Single ingress controller

6. AI/ML Workload Platform

   • Jupyter notebooks
   • TensorFlow serving
   • MLflow for experiment tracking
   • GPU support for training

Learning Resources

Official Documentation:

• K3s Docs: https://docs.k3s.io/
• Kubernetes Docs: https://kubernetes.io/docs/
• Helm Docs: https://helm.sh/docs/

Community Resources:

• K3s GitHub: https://github.com/k3s-io/k3s
• r/kubernetes subreddit
• Kubernetes Slack community
• CNCF Landscape: https://landscape.cncf.io/

Courses & Certifications:

• Certified Kubernetes Administrator (CKA)
• Certified Kubernetes Application Developer (CKAD)
• Cloud Native Computing Foundation courses

Homelab Optimization Tips

1. Start Small: Begin with single-node, add nodes as needed
2. Document Everything: Keep notes on configuration decisions
3. Automate: Use GitOps, CI/CD, and scripts
4. Monitor Proactively: Set up alerts before problems occur
5. Regular Maintenance: Update K3s, clean up resources, review logs
6. Backup Strategy: Test backups regularly
7. Security First: Keep secrets secure, use RBAC, scan images

Quick Reference Commands

# Cluster Management
kubectl cluster-info
kubectl get nodes
kubectl top nodes

# Application Management
kubectl get pods -A
kubectl logs POD_NAME -f
kubectl exec -it POD_NAME -- /bin/sh
kubectl describe pod POD_NAME

# Debugging
kubectl get events -A --sort-by='.lastTimestamp'
kubectl describe node NODE_NAME
kubectl top pods -A

# Helm
helm list -A
helm upgrade RELEASE CHART
helm rollback RELEASE

# System Maintenance
systemctl status k3s
journalctl -u k3s -f
/usr/local/bin/k3s-uninstall.sh

---

Conclusion

K3s provides a production-grade Kubernetes experience optimized for resource-constrained environments like homelabs. Through this guide, you’ve learned:

• Fundamentals: What K3s is and why it’s ideal for homelabs
• Installation: From single-node to HA multi-node clusters
• Storage: Local path and distributed Longhorn storage
• Networking: Traefik ingress, LoadBalancer services, TLS configuration
• Applications: Helm-based deployment and management
• Monitoring: Prometheus and Grafana observability
• Automation: AI-powered scaling, anomaly detection, and self-healing
• Troubleshooting: Diagnosing and resolving common issues

Key Takeaways:

1. K3s brings enterprise Kubernetes to resource-constrained environments
2. Start simple and scale as your needs grow
3. Monitoring and observability are essential from day one
4. Automation reduces operational overhead
5. The Kubernetes ecosystem is vast—explore tools that fit your needs

Your K3s Journey:

• Week 1: Master single-node deployments
• Month 1: Add multi-node capabilities and storage
• Month 3: Implement monitoring and automation
• Month 6: Deploy production-grade applications
• Year 1: Contribute to the community!

Whether you’re learning Kubernetes for career development, hosting personal services, or experimenting with cloud-native technologies, K3s provides the perfect foundation. The skills you’ve gained are directly applicable to enterprise Kubernetes environments.

Happy homelabbing!

---

Related Guides

• Docker Fundamentals for Homelabs
• VictoriaMetrics: Prometheus-Compatible Monitoring
• Home Assistant: Docker Setup
• Self-Hosted GitOps with ArgoCD
• K0s: Another Lightweight Kubernetes Option

---

Last updated: January 2025 Guide version: 1.0 Tested with K3s v1.28.5+k3s1