Ollama: Self-Hosted Local LLMs for Your Homelab

Running large language models (LLMs) locally has never been easier. Ollama brings the power of AI directly to your homelab—no cloud dependencies, no API costs, and complete privacy. This comprehensive guide will walk you through everything from installation to advanced automation workflows.

What You’ll Learn:

• How to install and configure Ollama on any platform
• Select the right models for your hardware and use cases
• Integrate AI into your homelab workflows
• Build automated systems with local LLMs
• Troubleshoot common issues and optimize performance

Time Investment: 45 minutes for basic setup | 2-3 hours for advanced integrations

Hardware Requirements: 8GB RAM minimum (16GB recommended) | GPU optional but recommended

---

Table of Contents

1. Understanding Ollama
2. Installation & Setup
3. Model Selection Guide
4. First Steps
5. API Integration
6. Web Interface with Open WebUI
7. Custom Models with Modelfiles
8. AI Automation Workflows
9. Troubleshooting
10. Next Steps

---

Understanding Ollama

What is Ollama?

Ollama is an open-source platform that simplifies running large language models on your local hardware. Think of it as Docker for AI models—a single command like ollama run llama3.3 downloads, configures, and runs a complete LLM on your machine.

Core Features:

• 100% Local Processing: Your data never leaves your machine
• Free & Open Source: MIT licensed with no subscription fees
• OpenAI-Compatible API: Drop-in replacement for cloud services
• Extensive Model Library: 100+ pre-configured models ready to use
• Hardware Flexible: Runs on CPU or accelerates with NVIDIA, AMD, or Apple GPUs

Why Choose Local LLMs?

The decision to run local LLMs comes down to three key factors: privacy, cost, and control.

Privacy & Security: When you use cloud AI services, your data is transmitted to external servers. For homelabs handling sensitive information—proprietary code, personal documents, internal knowledge bases—this isn’t acceptable. With Ollama, every inference happens locally. Your prompts, responses, and model weights stay on your hardware, making it HIPAA and GDPR-friendly by default.

Cost Savings: Cloud AI services charge per token. A moderate user running 10 queries daily (200 tokens each) spends approximately $1,460/year on GPT-3.5 API costs. An Ollama setup with a mid-range GPU costs ~$1,800 upfront plus $131/year in electricity. You break even in 18 months, then save $1,300+ annually. Scale to 100 queries daily, and cloud costs explode to $14,600/year while Ollama remains at $131/year.

Control & Customization: Local LLMs give you complete control over system prompts, parameters, fine-tuning, and model behavior. You can create specialized models for your exact use case, adjust performance parameters in real-time, and integrate deeply with your existing infrastructure—all without vendor limitations.

How Ollama Works

User Request
    ↓
Ollama CLI/API
    ↓
Check Local Cache (if model exists)
    ↓
Load Model to RAM/VRAM (Quantized GGUF format)
    ↓
Inference Engine (llama.cpp-based)
    ↓
Hardware Acceleration (GPU/CPU)
    ↓
Streaming Response

Key Architecture Concepts:

1. Quantization: Ollama uses GGUF format with 4-bit and 8-bit quantization to reduce memory footprint by 70%+ while preserving 95%+ model quality. A 7B parameter model that originally requires 14GB RAM runs in just 4GB with Q4_K_M quantization.

2. Model Registry: All models are hosted at ollama.com/library with pre-configured settings, eliminating complex setup. Just run ollama pull model-name and you’re ready.

3. Hardware Abstraction: Ollama automatically detects and utilizes available acceleration—NVIDIA CUDA, AMD ROCm, Apple Metal, or CPU fallback—without manual configuration.

4. REST API: Exposes OpenAI-compatible endpoints at http://localhost:11434/v1/, allowing seamless integration with existing tools built for OpenAI’s API.

Ollama vs Cloud AI Services

When to Choose Ollama:

• You handle sensitive data (code, health records, internal docs)
• You make 10+ AI queries daily (ROI within 18 months)
• You need offline AI capabilities
• You want full control over model behavior
• You have compliance requirements (HIPAA, GDPR)

When Cloud APIs Work Better:

• Occasional use (< 5 queries/week)
• Need latest cutting-edge models immediately
• Require massive scale beyond single-machine capacity
• Prefer zero hardware management

---

Installation & Setup

Hardware Requirements

Before installing, verify your system meets these requirements:

Minimum Configuration (CPU-Only):

• RAM: 8GB (allows 3B models) | 16GB recommended (7B models)
• CPU: Modern 8-core processor (Intel 11th gen+ or AMD Zen4+)
• Storage: 20GB free space (OS + 1-2 models)
• Performance: 2-5 tokens/second with 7B models

Recommended Configuration (GPU-Accelerated):

• RAM: 16-32GB system memory
• GPU Options:
  • NVIDIA: RTX 4060 (8GB) or better | RTX 4070 (12GB) ideal | RTX 4090 (24GB) for large models
  • AMD: RX 6600 (8GB) or RX 7800 XT (16GB+)
  • Apple Silicon: M1/M2/M3 with 16GB+ unified memory
• Storage: NVMe SSD with 50GB+ free (faster model loading)
• Performance: 40-150 tokens/second depending on GPU

GPU Memory Guidelines:

• 7B models: 4GB VRAM (Q4 quantization) to 8GB (Q8)
• 13B models: 8GB VRAM (Q4) to 16GB (Q8)
• 34B models: 16GB VRAM (Q4) to 32GB (Q8)
• 70B models: 32GB+ VRAM (Q4) or multi-GPU setup
• Rule of Thumb: Model size × 1.5 = minimum VRAM needed

Installation Methods

Choose the installation method that fits your environment:

Native Installation (Recommended for Most Users)

Linux Installation:

# Download and install Ollama
curl -fsSL https://ollama.ai/install.sh | sh

# Start the service
systemctl start ollama
systemctl enable ollama  # Auto-start on boot

# Verify installation
ollama --version
# Expected output: ollama version X.X.X

GPU Setup for NVIDIA (Linux):

# Install NVIDIA Container Toolkit (for Docker later)
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list |   sudo tee /etc/apt/sources.list.d/nvidia-docker.list

# Install toolkit
sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit
sudo systemctl restart docker

# Verify GPU detection
nvidia-smi
# Should show your GPU(s) with available memory

macOS Installation:

1. Download Ollama.app from ollama.com
2. Drag to Applications folder
3. Launch Ollama from Applications
4. Verify in Terminal:

   ollama --version

Notes for macOS:

• Automatic Metal acceleration on M1/M2/M3 Macs
• Runs as background service at http://localhost:11434
• No Docker required for native installation

Windows Installation:

1. Download OllamaSetup.exe from ollama.com
2. Run installer and follow wizard
3. Ollama starts automatically on system boot
4. Access at http://localhost:11434

For GPU Support (Windows):

• Install latest NVIDIA drivers from nvidia.com
• CUDA support is auto-detected for compatible GPUs
• For Docker usage: Enable WSL2 in Docker Desktop settings

Verify Installation (All Platforms):

# Test basic connectivity
curl http://localhost:11434/api/tags
# Expected: {"models":[]} (or list of existing models)

Docker Installation (Best for Production)

Docker provides consistent deployment across platforms and simplifies management, especially for multi-service setups.

Docker Compose Setup (Recommended):

Create docker-compose.yml:

version: '3.8'

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: always
    ports:
      - "11434:11434"
    volumes:
      - ollama_data:/root/.ollama
    environment:
      - OLLAMA_NUM_PARALLEL=4        # Concurrent requests
      - OLLAMA_NUM_THREADS=8         # CPU threads (match cores)
      - OLLAMA_GPU_LEVELS=50         # GPU utilization %
      - OLLAMA_KEEP_ALIVE=24h        # Keep models in memory
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

  open-webui:
    image: ghcr.io/open-webui/open-webui:latest
    container_name: open-webui
    restart: always
    ports:
      - "3000:8080"
    volumes:
      - open-webui_data:/app/backend/data
    environment:
      - OLLAMA_API_BASE_URL=http://ollama:11434/api
    depends_on:
      - ollama

volumes:
  ollama_data:
  open-webui_data:

Deploy the stack:

docker-compose up -d

# Verify services are running
docker-compose ps

# Check logs
docker logs ollama -f
docker logs open-webui -f

GPU Support Across Platforms:

Important: macOS does not support GPU passthrough with Docker Desktop. Use native installation for Metal acceleration.

Initial Configuration

Configure Resource Limits (Linux):

# Edit systemd service
sudo systemctl edit ollama

Add to [Service] section:

[Service]
Environment="OLLAMA_NUM_PARALLEL=4"      # Parallel requests
Environment="OLLAMA_NUM_THREADS=8"       # CPU threads
Environment="OLLAMA_GPU_LEVELS=50"       # GPU utilization %
Environment="OLLAMA_KEEP_ALIVE=24h"      # Model memory retention
Environment="OLLAMA_HOST=0.0.0.0:11434"  # Bind to all interfaces (if needed)

Restart service:

sudo systemctl restart ollama

Monitor Resources During First Run:

# Watch real-time resource usage
watch -n 1 'free -h && nvidia-smi'  # GPU systems
watch -n 1 'free -h'                # CPU-only systems

---

Model Selection Guide

Choosing the right model is critical for balancing performance, resource usage, and output quality.

Popular Models for Homelabs (2025)

General Purpose: Llama 3.3

Overview:

• Released: December 2024 (Meta’s latest flagship)
• Sizes: 1B, 3B, 8B, 70B parameters
• Context Length: 128K tokens (handles long documents)
• Strengths: Balanced reasoning, multilingual, excellent instruction-following

Recommended Variant: Llama 3.3 8B

ollama pull llama3.3:8b
ollama run llama3.3:8b

Use Cases:

• General chatbot for family/team
• Content generation and writing assistance
• Document summarization
• Knowledge synthesis

Hardware Requirements:

• VRAM: 6GB (Q4) to 12GB (Q8)
• Speed: 60-80 tokens/sec on RTX 4070
• Best For: Primary general-purpose model

Code Generation: CodeLlama

Overview:

• Specialization: Code generation, debugging, review
• Sizes: 7B, 13B, 34B parameters
• Supports: 20+ programming languages
• Strengths: Multi-file editing, function implementation, technical docs

Recommended Variant: CodeLlama 34B (if hardware allows) or 7B (for efficiency)

# Professional grade
ollama pull codellama:34b
ollama run codellama:34b

# Efficient alternative
ollama pull codellama:7b
ollama run codellama:7b

Use Cases:

• IDE integration (Continue.dev, Cursor)
• Code review automation
• Refactoring assistance
• Documentation generation

Hardware Requirements:

• 7B: 4GB VRAM, 50+ tokens/sec
• 34B: 16GB VRAM, 20-40 tokens/sec
• Best For: Developers wanting local coding assistant

Advanced Reasoning: DeepSeek-R1

Overview:

• Released: January 2025
• Sizes: 1.5B, 7B, 32B, 70B parameters
• Performance: Approaches OpenAI O3 and Gemini 2.5 Pro
• Strengths: Chain-of-thought, complex math/logic, problem decomposition

Recommended Variant: DeepSeek-R1 7B

ollama pull deepseek-r1:7b
ollama run deepseek-r1:7b

Use Cases:

• Complex technical problem-solving
• Mathematical reasoning
• Multi-step planning
• Research assistance

Hardware Requirements:

• 7B: 8GB VRAM
• 32B: 16GB VRAM
• Speed: Slower than general models (reasoning overhead)
• Best For: Complex analytical tasks

Vision & Documents: Qwen-VL

Overview:

• Capability: Image analysis, OCR, document layout understanding
• Sizes: 7B+ parameters
• Strengths: Technical diagram interpretation, screenshot analysis, visual Q&A

Recommended Variant: Qwen-VL 7B

ollama pull qwen-vl:7b
ollama run qwen-vl:7b

Use Cases:

• Screenshot troubleshooting (“What’s this error?“)
• Network diagram analysis
• Document OCR and extraction
• Visual technical documentation

Hardware Requirements:

• VRAM: 8GB+
• Speed: Slower (vision processing overhead)
• Best For: Homelab documentation and diagram analysis

Lightweight: Phi-3-Mini

Overview:

• Size: 3.8B parameters
• Designed For: Constrained environments (Raspberry Pi, low-power systems)
• Quality: Excellent for size; punches above its weight

Recommended for: Learning, testing, CPU-only systems

ollama pull phi-3-mini
ollama run phi-3-mini

Use Cases:

• Learning and experimentation
• Edge computing (low-power servers)
• Raspberry Pi homelabs
• Quick testing without GPU

Hardware Requirements:

• VRAM/RAM: 2GB
• Speed: 10-20 tokens/sec on CPU
• Best For: Resource-constrained setups

Development Assistant: Devstral-Small

Overview:

• Released: 2025 (Mistral AI)
• Size: 24B parameters
• Strengths: Multi-file editing, codebase exploration, software engineering agents

ollama pull devstral-small

Use Cases:

• Large codebase refactoring
• Automated code review pipelines
• Complex software architecture analysis

Hardware Requirements:

• VRAM: 16GB+
• Best For: Advanced development workflows

Model Selection Matrix

Understanding Quantization

Quantization is what makes running large models on consumer hardware possible. It reduces the precision of model weights from 32-bit floating point to 4-8 bit integers.

Impact on Memory:

Quantization Methods:

• Q4_K_M (4-bit with K-means, Medium): Best balance; 75% size reduction with minimal quality loss. Start here.
• Q8_0 (8-bit): Higher quality, larger files. Use when you have VRAM headroom.
• Q5_K (5-bit): Compromise between Q4 and Q8.
• Q2_K (2-bit): Maximum compression; quality tradeoff. Experimental.

Selecting Quantization:

# List available quantizations for a model
ollama list

# Pull specific quantization
ollama pull llama3.3:70b-q4_K_M

# Run quantized model
ollama run llama3.3:70b-q4_K_M

Recommendation: Start with Q4_K_M variants. They offer 95%+ quality retention with 70%+ memory savings.

---

First Steps

Downloading Your First Model

Step 1: Choose a starter model

For your first experience, we recommend Mistral 7B—it’s fast, capable, and runs on modest hardware.

# Pull (download) Mistral 7B
ollama pull mistral

# Output shows progress:
# pulling manifest
# pulling 8b747ebfeecf
# downloading [████████████] 4.8 GB / 4.8 GB
# verifying sha256 digest
# writing manifest
# success

Step 2: Run interactive session

ollama run mistral
# Starts interactive prompt

Step 3: Test with a question

>>> What is the difference between Docker and Kubernetes?

[Model streams response in real-time...]

Docker is a containerization platform that packages applications with their dependencies
into isolated containers. Kubernetes is a container orchestration system that manages
and scales multiple Docker containers across clusters of machines...

>>> /bye

Interactive Commands:

• /help - Show available commands
• /bye or Ctrl+D - Exit interactive mode
• /set parameter value - Adjust model parameters

Monitoring Performance

While the model runs, open another terminal:

# Monitor GPU and RAM usage
watch -n 1 'ollama list && echo "---" && nvidia-smi && echo "---" && free -h'

What to look for:

• GPU memory should ramp up when model loads
• VRAM usage should stabilize during inference
• CPU usage varies based on batch processing
• Tokens/second displayed in interactive mode

Non-Interactive Usage

# Pipe input to ollama (useful for scripts)
echo "Explain quantum entanglement in one sentence" | ollama run mistral

# Or use API directly
curl http://localhost:11434/api/generate   -d '{
    "model": "mistral",
    "prompt": "What is cloud computing?",
    "stream": false
  }' | jq .response

Testing Multiple Models

# Pull additional models
ollama pull llama3.3:8b
ollama pull codellama:7b

# List all local models
ollama list
# Shows: NAME, ID, SIZE, MODIFIED

# Switch between models
ollama run llama3.3:8b
# Ask a question, then exit

ollama run codellama:7b
# Ask for code generation

---

API Integration

Ollama exposes an OpenAI-compatible REST API, making it a drop-in replacement for cloud services.

Available Endpoints

POST /api/generate              # Text generation
POST /api/chat                  # Chat completions
POST /v1/chat/completions       # OpenAI-compatible endpoint
GET  /api/tags                  # List available models
POST /api/pull                  # Download model
POST /api/show                  # Model details
POST /api/create                # Create custom model
DELETE /api/delete              # Remove model

Testing with cURL

Basic chat completion:

curl http://localhost:11434/v1/chat/completions   -H "Content-Type: application/json"   -d '{
    "model": "mistral",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "Explain Kubernetes in 3 sentences"}
    ],
    "temperature": 0.7,
    "max_tokens": 200
  }'

Expected response:

{
  "id": "chatcmpl-xyz",
  "object": "chat.completion",
  "created": 1704067200,
  "model": "mistral",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Kubernetes is an open-source container orchestration platform..."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 25,
    "completion_tokens": 62,
    "total_tokens": 87
  }
}

Python Integration

Installation:

pip install openai

Basic chat example:

from openai import OpenAI

# Initialize with Ollama endpoint
client = OpenAI(
    base_url="http://localhost:11434/v1/",
    api_key="ollama"  # Required but not validated
)

# Single message
response = client.chat.completions.create(
    model="mistral",
    messages=[
        {"role": "system", "content": "You are a Kubernetes expert."},
        {"role": "user", "content": "How do I debug a pod stuck in CrashLoopBackOff?"}
    ],
    temperature=0.3,
    max_tokens=500
)

print(response.choices[0].message.content)

Multi-turn conversation:

from openai import OpenAI

def chat_session():
    client = OpenAI(
        base_url="http://localhost:11434/v1/",
        api_key="ollama"
    )

    messages = []
    system_prompt = "You are a helpful coding assistant specializing in Python."

    print("Chat started. Type 'exit' to quit.\n")

    while True:
        user_input = input("You: ").strip()

        if user_input.lower() == 'exit':
            break

        messages.append({"role": "user", "content": user_input})

        response = client.chat.completions.create(
            model="mistral",
            messages=[{"role": "system", "content": system_prompt}] + messages,
            temperature=0.7
        )

        assistant_msg = response.choices[0].message.content
        messages.append({"role": "assistant", "content": assistant_msg})

        print(f"\nAssistant: {assistant_msg}\n")

# Run
chat_session()

Streaming responses:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:11434/v1/",
    api_key="ollama"
)

stream = client.chat.completions.create(
    model="mistral",
    messages=[{"role": "user", "content": "Explain Docker in detail"}],
    stream=True
)

print("Response: ", end="", flush=True)
for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)
print()  # Newline after stream

Node.js Integration

Installation:

npm install openai

Basic usage:

const OpenAI = require("openai");

const client = new OpenAI({
    baseURL: "http://localhost:11434/v1/",
    apiKey: "ollama",
});

async function generateCode() {
    const response = await client.chat.completions.create({
        model: "codellama:7b",
        messages: [
            {
                role: "user",
                content: "Write a Python function that validates email addresses using regex"
            }
        ],
        temperature: 0.3,
    });

    console.log(response.choices[0].message.content);
}

generateCode();

Streaming in Node.js:

const OpenAI = require("openai");

const client = new OpenAI({
    baseURL: "http://localhost:11434/v1/",
    apiKey: "ollama",
});

async function streamResponse() {
    const stream = await client.chat.completions.create({
        model: "mistral",
        messages: [{ role: "user", content: "Explain machine learning" }],
        stream: true,
    });

    process.stdout.write("Response: ");
    for await (const chunk of stream) {
        process.stdout.write(chunk.choices[0]?.delta?.content || "");
    }
    console.log("\n");
}

streamResponse();

Building a CLI Tool

Save as ollama-cli.py:

#!/usr/bin/env python3
"""Simple Ollama CLI with conversation history"""

import sys
import json
from openai import OpenAI

class OllamaChat:
    def __init__(self, model="mistral"):
        self.client = OpenAI(
            base_url="http://localhost:11434/v1/",
            api_key="ollama"
        )
        self.model = model
        self.history = []

    def chat(self, user_message, system_prompt=None):
        """Send message and return response"""
        self.history.append({"role": "user", "content": user_message})

        messages = self.history
        if system_prompt:
            messages = [{"role": "system", "content": system_prompt}] + messages

        response = self.client.chat.completions.create(
            model=self.model,
            messages=messages,
            temperature=0.7
        )

        assistant_response = response.choices[0].message.content
        self.history.append({"role": "assistant", "content": assistant_response})

        return assistant_response

    def clear_history(self):
        """Reset conversation"""
        self.history = []

    def save_history(self, filename):
        """Save conversation to JSON"""
        with open(filename, 'w') as f:
            json.dump(self.history, f, indent=2)

# Main execution
if __name__ == "__main__":
    chat = OllamaChat(model="mistral")

    print("Ollama Chat Tool")
    print("Commands: 'exit' to quit | 'clear' to reset | 'save' to export\n")

    while True:
        user_input = input("You: ").strip()

        if user_input.lower() == 'exit':
            print("Goodbye!")
            break
        elif user_input.lower() == 'clear':
            chat.clear_history()
            print("Conversation cleared.\n")
            continue
        elif user_input.lower() == 'save':
            chat.save_history("conversation.json")
            print("Saved to conversation.json\n")
            continue

        response = chat.chat(user_input)
        print(f"\nAssistant: {response}\n")

Usage:

chmod +x ollama-cli.py
./ollama-cli.py

---

Web Interface with Open WebUI

Open WebUI provides a modern, user-friendly interface for Ollama—no command-line required.

What is Open WebUI?

Features:

• ChatGPT-like web interface
• Model download and management
• Conversation history and organization
• User authentication (multi-user support)
• RAG (Retrieval-Augmented Generation) integration
• Model parameter customization
• Export conversations as markdown/PDF

Installation

Option 1: Docker (Easiest)

docker run -d   -p 3000:8080   -v open-webui:/app/backend/data   --name open-webui   ghcr.io/open-webui/open-webui:latest

# Wait 10 seconds for startup
sleep 10

# Access at http://localhost:3000

Option 2: Docker Compose (With Ollama)

Use the docker-compose.yml from the Installation section. This deploys both Ollama and Open WebUI together.

Option 3: Native Installation (Linux)

git clone https://github.com/open-webui/open-webui.git
cd open-webui

pip install -r requirements.txt

python main.py
# Access at http://localhost:8080

First-Time Setup

1. Open browser to http://localhost:3000
2. Sign up: Create username and password (becomes admin account)
3. Configure Ollama connection:
   • Go to Settings → Admin Panel → Connections
   • Ollama API URL:
     • Docker Compose: http://ollama:11434 (service name)
     • Native: http://localhost:11434
   • Click “Test Connection” → Should show “Connected”
4. Download models:
   • Click ”+” button in chat interface
   • Search “mistral” or other model names
   • Click to download from Ollama library

Using Open WebUI

Starting a conversation:

1. Select model from dropdown (top of chat)
2. Optionally set system prompt:
   • Click settings icon
   • Enter prompt: “You are a Kubernetes expert”
3. Type message and press Enter
4. View streaming response

Managing conversations:

• New chat: Click ”+” button
• Rename: Click conversation name
• Delete: Right-click → Delete
• Export: Settings → Export as Markdown/PDF

Adjusting model parameters:

Click model settings icon:

• Temperature: 0.1 (focused) to 1.0 (creative)
• Max Tokens: Limit response length
• Top P: Nucleus sampling (0.1-1.0)
• System Prompt: Define model behavior

Multi-user setup:

1. Admin creates user accounts: Settings → Admin → Users → Add User
2. Share URL with team: http://your-server-ip:3000
3. Each user gets own conversation history
4. Admin controls model access and settings

---

Custom Models with Modelfiles

Modelfiles let you create specialized AI assistants tailored to specific tasks.

Understanding Modelfiles

Think of a Modelfile as a Dockerfile for AI models. It defines:

• Base model to build from
• System prompt (personality/instructions)
• Default parameters (temperature, token limits)
• Few-shot examples (optional)

Basic syntax:

FROM base_model

SYSTEM "Your system prompt here"

PARAMETER temperature 0.7
PARAMETER top_p 0.9
PARAMETER num_predict 2048

MESSAGE user "Example user message"
MESSAGE assistant "Example assistant response"

Example 1: Homelab Expert

Create Modelfile-homelab:

FROM mistral

SYSTEM """You are an expert in self-hosted infrastructure for homelabs.

Specialties:
- Docker and Kubernetes orchestration
- Network configuration and troubleshooting
- Hardware recommendations and optimization
- Cost-effective infrastructure design
- Security best practices for homelabs

Guidelines:
- Provide practical, actionable advice
- Include specific configuration examples
- Consider resource constraints (budget, power)
- Prioritize reliability and maintainability
- Keep responses concise and technical"""

PARAMETER temperature 0.3
PARAMETER top_p 0.8
PARAMETER num_predict 1500

Build and use:

ollama create homelab-expert -f Modelfile-homelab

ollama run homelab-expert
>>> What's the best way to set up a 3-node Kubernetes homelab on a budget?

Example 2: Code Review Specialist

Create Modelfile-codereview:

FROM codellama:34b

SYSTEM """You are an expert code reviewer specializing in Python and Go.

Review Checklist:
1. Security vulnerabilities (SQL injection, XSS, auth issues)
2. Performance bottlenecks (N+1 queries, memory leaks)
3. Code style violations (PEP 8, Go conventions)
4. Best practice violations (error handling, logging)
5. Testability and maintainability concerns

Output Format:
SECURITY: [critical findings]
PERFORMANCE: [optimization opportunities]
STYLE: [convention violations]
BEST PRACTICES: [recommendations]
REFACTORING: [suggested improvements]

Be direct, specific, and actionable."""

PARAMETER temperature 0.2
PARAMETER top_p 0.8

Build and use:

ollama create code-reviewer -f Modelfile-codereview

ollama run code-reviewer
>>> [Paste code to review]

Example 3: RAG-Enabled Knowledge Assistant

Create Modelfile-knowledge:

FROM llama3.3:8b

SYSTEM """You are a knowledge base assistant with access to internal documentation.

Instructions:
1. ONLY use information from provided context documents
2. If context doesn't contain the answer, say: "This information isn't in our knowledge base"
3. Always cite which document provides the information
4. Provide concise, accurate answers
5. For ambiguous questions, ask for clarification

Context Format:
User queries will include <CONTEXT> tags with relevant documents.
Base your answer ONLY on this context."""

PARAMETER temperature 0.1
PARAMETER top_p 0.7
PARAMETER num_predict 1024

Build and use:

ollama create knowledge-assistant -f Modelfile-knowledge

# Use with context
ollama run knowledge-assistant
>>> <CONTEXT>
[Paste relevant documentation]
</CONTEXT>

What is our paternity leave policy?

Example 4: Technical Writing Assistant

Create Modelfile-techwriter:

FROM llama3.3:8b

SYSTEM """You are a technical writing assistant for infrastructure documentation.

Documentation Standards:
- Use clear, concise language
- Include prerequisites and requirements
- Provide step-by-step instructions
- Add troubleshooting sections
- Include code examples with syntax highlighting
- Use tables for comparisons
- Follow markdown best practices

Structure:
1. Overview (what and why)
2. Prerequisites
3. Step-by-step guide
4. Configuration examples
5. Verification steps
6. Troubleshooting common issues
7. Next steps"""

PARAMETER temperature 0.4
PARAMETER top_p 0.85

# Few-shot examples
MESSAGE user "Document how to set up Prometheus monitoring"
MESSAGE assistant "# Prometheus Monitoring Setup\n\n## Overview\nPrometheus is an open-source monitoring system...\n\n## Prerequisites\n- Docker installed\n- 2GB available RAM\n..."

Build and use:

ollama create tech-writer -f Modelfile-techwriter

ollama run tech-writer
>>> Document how to configure Traefik reverse proxy for my homelab

Advanced: Multi-Model Orchestration

You can create specialized models for different tasks and route queries intelligently.

Models:

• homelab-expert - Infrastructure questions
• code-reviewer - Code analysis
• tech-writer - Documentation
• knowledge-assistant - Internal knowledge base

Router script (route.py):

#!/usr/bin/env python3
import ollama

def classify_query(query):
    """Determine which model to use"""
    response = ollama.generate(
        model="mistral",
        prompt=f"""Classify this query into ONE category:
1. INFRASTRUCTURE - Homelab, networking, servers
2. CODE - Programming, code review, debugging
3. DOCUMENTATION - Writing guides, technical docs
4. KNOWLEDGE - Company/internal information

Query: {query}

Respond with ONLY the category name.""",
        stream=False
    )
    return response['response'].strip().upper()

def route_to_model(query):
    """Route query to specialized model"""
    category = classify_query(query)

    routing = {
        'INFRASTRUCTURE': 'homelab-expert',
        'CODE': 'code-reviewer',
        'DOCUMENTATION': 'tech-writer',
        'KNOWLEDGE': 'knowledge-assistant',
    }

    model = routing.get(category, 'mistral')

    response = ollama.generate(
        model=model,
        prompt=query,
        stream=False
    )

    return {
        'category': category,
        'model': model,
        'response': response['response']
    }

# Test
result = route_to_model("How do I set up a VLAN for my homelab?")
print(f"Category: {result['category']}")
print(f"Model: {result['model']}")
print(f"Answer: {result['response']}")

---

AI Automation Workflows

This section demonstrates practical automation workflows that leverage local LLMs to save time and enhance your homelab.

1. Automated Document Analysis Pipeline

Use Case: Process technical documents, extract summaries, key concepts, and generate study materials automatically.

Implementation (analyze_docs.py):

#!/usr/bin/env python3
"""Automated document analysis pipeline"""

import ollama
from pathlib import Path
import sys

def analyze_document(doc_path):
    """Analyze single document and extract insights"""

    print(f"Processing: {doc_path.name}")

    with open(doc_path, 'r') as f:
        content = f.read()[:3000]  # Limit to first 3000 chars

    analyses = {}

    # Generate summary
    print("  → Generating summary...")
    summary = ollama.generate(
        model="llama3.3:8b",
        prompt=f"Summarize this technical document in 3 bullet points:\n\n{content}",
        stream=False
    )
    analyses['summary'] = summary['response']

    # Extract key concepts
    print("  → Extracting key concepts...")
    concepts = ollama.generate(
        model="llama3.3:8b",
        prompt=f"Extract 5 key technical concepts from this document:\n\n{content}",
        stream=False
    )
    analyses['concepts'] = concepts['response']

    # Generate study questions
    print("  → Generating study questions...")
    questions = ollama.generate(
        model="llama3.3:8b",
        prompt=f"Generate 3 challenging study questions based on:\n\n{content}",
        stream=False
    )
    analyses['questions'] = questions['response']

    return analyses

def process_folder(folder_path):
    """Process all documents in folder"""

    doc_count = 0
    for doc in Path(folder_path).glob("*.txt"):
        doc_count += 1
        analysis = analyze_document(doc)

        # Save results
        result_file = doc.with_suffix(".analysis.md")
        with open(result_file, 'w') as f:
            f.write(f"# Analysis of {doc.name}\n\n")
            f.write(f"## Summary\n{analysis['summary']}\n\n")
            f.write(f"## Key Concepts\n{analysis['concepts']}\n\n")
            f.write(f"## Study Questions\n{analysis['questions']}\n")

        print(f"  ✓ Saved to {result_file}\n")

    print(f"Processed {doc_count} documents")

if __name__ == "__main__":
    folder = sys.argv[1] if len(sys.argv) > 1 else "./documents"
    process_folder(folder)

Usage:

chmod +x analyze_docs.py

# Create test documents
mkdir documents
echo "Kubernetes is a container orchestration platform..." > documents/k8s.txt
echo "Docker provides containerization..." > documents/docker.txt

# Process
./analyze_docs.py documents

# View results
cat documents/k8s.analysis.md

2. Private Knowledge Base with RAG

Use Case: Build a searchable knowledge base where the AI answers questions based solely on your internal documents.

Architecture:

Your Documents
    ↓ (embedding)
Vector Store (ChromaDB)
    ↓ (similarity search)
Retrieved Context
    ↓ (augmented prompt)
Ollama LLM
    ↓
Grounded Answer

Installation:

pip install chromadb ollama

Implementation (knowledge_base.py):

#!/usr/bin/env python3
"""RAG-powered knowledge base"""

import ollama
import chromadb
from chromadb.utils import embedding_functions

# Initialize ChromaDB
client = chromadb.Client()

# Use Ollama for embeddings (requires nomic-embed-text model)
# ollama pull nomic-embed-text
embedding_fn = embedding_functions.OllamaEmbeddingFunction(
    url="http://localhost:11434/api/embeddings",
    model_name="nomic-embed-text"
)

collection = client.create_collection(
    name="homelab_docs",
    embedding_function=embedding_fn
)

def index_documents(docs):
    """Add documents to vector store"""
    for i, doc in enumerate(docs):
        collection.add(
            ids=[str(i)],
            documents=[doc],
            metadatas=[{"source": f"doc_{i}"}]
        )
    print(f"Indexed {len(docs)} documents")

def answer_question(question):
    """Answer using RAG"""

    # Retrieve relevant documents
    results = collection.query(
        query_texts=[question],
        n_results=3
    )

    context = "\n\n".join(results['documents'][0])

    # Generate answer with context
    prompt = f"""Using this context from our knowledge base:

<CONTEXT>
{context}
</CONTEXT>

Answer the question: {question}

Important: Only use information from the context above. If the context doesn't contain the answer, say "This information isn't in our knowledge base."
"""

    response = ollama.generate(
        model="llama3.3:8b",
        prompt=prompt,
        stream=False
    )

    return response['response']

# Example usage
if __name__ == "__main__":
    # Index your homelab documentation
    docs = [
        "Kubernetes cluster runs on 3 nodes: node1 (master), node2, node3 (workers). Each has 16GB RAM and 4 cores.",
        "Docker registry is hosted at registry.homelab.local:5000 with basic authentication.",
        "Proxmox hosts are pve1 (192.168.1.10) and pve2 (192.168.1.11). Storage is on NFS at 192.168.1.100.",
        "Backup schedule: Daily at 2 AM to /mnt/backup using restic. Retention: 7 daily, 4 weekly, 12 monthly.",
        "Network VLAN setup: VLAN 10 (servers), VLAN 20 (IoT), VLAN 30 (guest). Firewall rules block inter-VLAN by default.",
    ]

    print("Indexing documents...")
    index_documents(docs)

    # Query knowledge base
    questions = [
        "What are the Kubernetes node specs?",
        "Where is the Docker registry?",
        "What's the backup retention policy?",
        "How many VLANs do we have?",
    ]

    for q in questions:
        print(f"\nQ: {q}")
        answer = answer_question(q)
        print(f"A: {answer}")

Usage:

# Pull embedding model
ollama pull nomic-embed-text

# Run knowledge base
python knowledge_base.py

3. Automated Code Review Bot

Use Case: Analyze code changes in Git commits and provide automated review feedback.

Implementation (code_review.py):

#!/usr/bin/env python3
"""Git pre-commit code review hook"""

import subprocess
import ollama
import sys

def get_staged_changes():
    """Get code diff from git"""
    result = subprocess.run(
        ['git', 'diff', '--cached'],
        capture_output=True,
        text=True
    )
    return result.stdout

def review_code(diff):
    """Analyze code changes"""

    if not diff.strip():
        print("No staged changes to review")
        return None

    prompt = f"""Review this code diff for potential issues:

{diff[:2000]}  # Limit to 2000 chars

Analyze for:
1. SECURITY: Vulnerabilities, injection risks, auth issues
2. PERFORMANCE: Inefficiencies, memory leaks, N+1 queries
3. STYLE: Conventions, readability, naming
4. BUGS: Logic errors, edge cases, error handling

Format:
SECURITY: [findings or "None"]
PERFORMANCE: [findings or "None"]
STYLE: [findings or "None"]
BUGS: [findings or "None"]
RECOMMENDATION: [overall recommendation: APPROVE/REVIEW/REJECT]

Be concise and specific."""

    response = ollama.generate(
        model="codellama:34b",
        prompt=prompt,
        stream=False
    )

    return response['response']

# Main execution
if __name__ == "__main__":
    print("Running AI code review...\n")

    diff = get_staged_changes()
    review = review_code(diff)

    if review:
        print(review)

        # Check for REJECT recommendation
        if "REJECT" in review:
            print("\n⚠️  Code review recommends rejection. Fix issues before committing.")
            sys.exit(1)
        elif "REVIEW" in review:
            print("\n⚠️  Code review suggests manual review. Proceed with caution.")
        else:
            print("\n✓ Code review passed")

    sys.exit(0)

Setup as Git hook:

# Copy to Git hooks directory
cp code_review.py .git/hooks/pre-commit
chmod +x .git/hooks/pre-commit

# Now every commit triggers AI review
git add myfile.py
git commit -m "Add new feature"
# → Runs AI code review automatically

4. System Log Analysis & Alerting

Use Case: Monitor system logs for anomalies, security issues, and errors using AI.

Implementation (log_analyzer.py):

#!/usr/bin/env python3
"""AI-powered log analysis"""

import ollama
import subprocess
from datetime import datetime, timedelta
import sys

def get_recent_logs(hours=1):
    """Get system logs from last N hours"""
    cutoff = datetime.now() - timedelta(hours=hours)

    result = subprocess.run(
        ['journalctl', '-S', str(cutoff), '--no-pager'],
        capture_output=True,
        text=True
    )

    # Return last 3000 characters (avoid overwhelming model)
    return result.stdout[-3000:]

def analyze_logs(logs):
    """AI analysis of log entries"""

    prompt = f"""Analyze these system logs for issues:

{logs}

Identify:
- ERRORS: Failures, crashes, exceptions
- SECURITY: Authentication failures, unauthorized access attempts
- PERFORMANCE: Resource exhaustion, slow operations
- ANOMALIES: Unusual patterns, unexpected behavior

Format:
SEVERITY: [CRITICAL/WARNING/INFO]
FINDINGS: [specific issues found]
RECOMMENDATIONS: [actionable next steps]

Be concise. Only report significant issues."""

    response = ollama.generate(
        model="llama3.3:8b",
        prompt=prompt,
        stream=False
    )

    return response['response']

def send_alert(message):
    """Send alert (customize for your notification system)"""
    # Example: Write to file, send email, post to Slack, etc.
    with open('/var/log/ai-alerts.log', 'a') as f:
        f.write(f"[{datetime.now()}] {message}\n\n")
    print("Alert logged to /var/log/ai-alerts.log")

# Main execution
if __name__ == "__main__":
    hours = int(sys.argv[1]) if len(sys.argv) > 1 else 1

    print(f"Analyzing logs from last {hours} hour(s)...\n")

    logs = get_recent_logs(hours)
    analysis = analyze_logs(logs)

    print(analysis)

    # Alert on critical issues
    if "CRITICAL" in analysis or "SECURITY" in analysis:
        print(f"\n🚨 ALERT: Critical issues detected at {datetime.now()}")
        send_alert(analysis)
    else:
        print(f"\n✓ No critical issues detected")

Schedule with cron:

# Add to crontab (runs every 10 minutes)
crontab -e

# Add line:
*/10 * * * * /usr/bin/python3 /path/to/log_analyzer.py 1 >> /var/log/ai-log-analysis.log 2>&1

5. Intelligent Task Router

Use Case: Route user queries to specialized models based on intent classification.

Implementation (task_router.py):

#!/usr/bin/env python3
"""Multi-model task router"""

import ollama

class TaskRouter:
    def __init__(self):
        self.models = {
            'CODE': 'codellama:34b',
            'INFRASTRUCTURE': 'homelab-expert',  # Custom model
            'GENERAL': 'mistral',
            'REASONING': 'deepseek-r1:7b',
            'DOCUMENT': 'llama3.3:8b',
        }

    def classify(self, query):
        """Determine task category"""
        prompt = f"""Classify this query into ONE category:

1. CODE - Programming, debugging, code generation
2. INFRASTRUCTURE - Servers, networking, DevOps, homelab
3. GENERAL - General knowledge, conversation
4. REASONING - Complex logic, math, multi-step problems
5. DOCUMENT - Text analysis, summarization, writing

Query: {query}

Respond with ONLY the category name (e.g., "CODE")."""

        response = ollama.generate(
            model="mistral",
            prompt=prompt,
            stream=False
        )

        return response['response'].strip().upper()

    def process(self, query):
        """Route and process query"""
        category = self.classify(query)
        model = self.models.get(category, 'mistral')

        print(f"Category: {category} | Model: {model}\n")

        response = ollama.generate(
            model=model,
            prompt=query,
            stream=False
        )

        return response['response']

# Usage
if __name__ == "__main__":
    router = TaskRouter()

    test_queries = [
        "Write a Python function to parse JSON",
        "How do I configure a VLAN on my homelab switch?",
        "Explain the concept of entropy",
        "If A is twice B, and B is half of C=20, what is A?",
        "Summarize this text: [long document]",
    ]

    for query in test_queries:
        print(f"Query: {query}")
        result = router.process(query)
        print(f"Response: {result}\n")
        print("-" * 80 + "\n")

6. Continuous Documentation Generator

Use Case: Automatically generate and update documentation from code comments and README files.

Implementation (doc_generator.py):

#!/usr/bin/env python3
"""Automated documentation generator"""

import ollama
import os
from pathlib import Path

def generate_module_docs(code_file):
    """Generate documentation for Python module"""

    with open(code_file, 'r') as f:
        code = f.read()[:2000]  # First 2000 chars

    prompt = f"""Generate comprehensive documentation for this Python module:

{code}

Include:
1. Module Overview (what it does)
2. Functions/Classes (purpose and parameters)
3. Usage Examples
4. Dependencies

Format as markdown."""

    response = ollama.generate(
        model="codellama:7b",
        prompt=prompt,
        stream=False
    )

    return response['response']

def generate_project_readme(project_dir):
    """Generate README.md for project"""

    # Scan directory structure
    structure = []
    for root, dirs, files in os.walk(project_dir):
        level = root.replace(project_dir, '').count(os.sep)
        indent = ' ' * 2 * level
        structure.append(f"{indent}{os.path.basename(root)}/")
        subindent = ' ' * 2 * (level + 1)
        for file in files[:5]:  # Limit files
            structure.append(f"{subindent}{file}")

    structure_str = "\n".join(structure[:30])  # Limit lines

    prompt = f"""Generate a README.md for this project:

Directory Structure:
{structure_str}

Include:
1. Project Title and Description
2. Features
3. Installation Instructions
4. Usage Examples
5. Project Structure Explanation
6. Contributing Guidelines

Format as markdown."""

    response = ollama.generate(
        model="llama3.3:8b",
        prompt=prompt,
        stream=False
    )

    return response['response']

# Usage
if __name__ == "__main__":
    # Generate module docs
    module_doc = generate_module_docs("analyze_docs.py")
    with open("docs/analyze_docs.md", 'w') as f:
        f.write(module_doc)
    print("✓ Generated module documentation")

    # Generate project README
    readme = generate_project_readme(".")
    with open("README.md", 'w') as f:
        f.write(readme)
    print("✓ Generated project README")

---

Troubleshooting

Common issues and their solutions.

Issue 1: Out of Memory (OOM) Errors

Symptoms:

• “CUDA out of memory” error
• Process killed during inference
• System freeze or slowdown

Diagnostic:

# Check GPU memory
nvidia-smi

# Check system RAM
free -h

# Monitor during inference
watch -n 1 'nvidia-smi && echo "---" && free -h'

Solutions (in order):

1. Use smaller quantization:

   # Instead of Q8, use Q4
   ollama pull mistral:7b-q4_K_M
   ollama run mistral:7b-q4_K_M

2. Use smaller model:

   # Instead of 70B, use 8B
   ollama pull llama3.3:8b

3. Limit output length:

   curl http://localhost:11434/api/generate -d '{
     "model": "mistral",
     "prompt": "...",
     "num_predict": 128
   }'

4. Adjust GPU layer offloading:

   export OLLAMA_GPU_LAYERS=35  # Reduce from default
   systemctl restart ollama

5. Close other GPU applications:

   # Check GPU usage
   nvidia-smi
   # Kill other processes using GPU

Issue 2: Slow Inference Speed

Symptoms:

• < 5 tokens/second generation
• Interactive use feels sluggish

Diagnostic:

# Benchmark current performance
time echo "Explain quantum computing" | ollama run mistral

# Check GPU utilization
watch -n 1 nvidia-smi
# Should show GPU memory in use during inference

Solutions:

1. Verify GPU acceleration:

   ollama list
   # Check if models show GPU memory usage
   
   # Test GPU
   nvidia-smi
   # Should show processes using GPU

2. Optimize threading (CPU):

   export OLLAMA_NUM_THREADS=16  # Match your CPU core count
   export OLLAMA_NUM_PARALLEL=4
   systemctl restart ollama

3. Use faster quantization:

   • Q2_K: Fastest, lower quality
   • Q4_K_M: Fast, good quality ← Recommended
   • Q8: Slower, best quality

4. Select faster model:

   • Phi-3-Mini: Fastest (3.8B)
   • Mistral 7B: Fast (7B)
   • Llama 3.3 8B: Medium (8B)
   • CodeLlama 34B: Slower (34B)

Expected Performance:

Issue 3: Docker GPU Not Working

Symptoms:

• Container can’t access GPU
• Falls back to CPU (slow performance)

Linux + NVIDIA:

# Test if Docker sees GPU
docker run --rm --gpus=all nvidia/cuda:11.8.0-runtime-ubuntu22.04 nvidia-smi

# If fails, install NVIDIA Container Toolkit
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list |   sudo tee /etc/apt/sources.list.d/nvidia-docker.list

sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit
sudo systemctl restart docker

# Test again
docker run --rm --gpus=all nvidia/cuda:11.8.0-runtime-ubuntu22.04 nvidia-smi

Windows 11 + NVIDIA:

1. Enable WSL2 in Docker Desktop: Settings → General → Use WSL2 based engine
2. Install latest NVIDIA drivers from nvidia.com
3. Restart Docker Desktop
4. Test: wsl --list -v (should show WSL2)

macOS:

GPU passthrough is NOT supported in Docker Desktop on macOS.

Solution: Use native Ollama installation (automatic Metal acceleration)

# Download from ollama.com
# Drag to Applications
# Run normally - Metal acceleration automatic

Issue 4: First Token Delay

Symptoms:

• 2-5 second delay before response starts
• Then inference is fast

Cause: Model loading from disk into VRAM

Solutions:

1. Keep model in memory longer:

   export OLLAMA_KEEP_ALIVE=24h  # Default is 5 minutes
   systemctl restart ollama

2. Pre-load model before use:

   # Start background warmup
   ollama run mistral < /dev/null &
   sleep 5  # Wait for loading
   
   # Now use normally (already loaded)
   ollama run mistral

3. Use smaller model (faster load):

   • Phi-3-Mini: <1 second load
   • Mistral 7B: ~2 seconds load
   • Llama 3.3 70B: 5-10 seconds load

Issue 5: Network Access Blocked

Symptoms:

• Can’t access Ollama from other machines on network
• Connection refused errors

Solution 1: Bind to all interfaces

# Linux systemd
sudo systemctl edit ollama

[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"

sudo systemctl restart ollama

Solution 2: Configure firewall

# Ubuntu/Debian
sudo ufw allow 11434/tcp

# RHEL/CentOS/Fedora
sudo firewall-cmd --permanent --add-port=11434/tcp
sudo firewall-cmd --reload

Solution 3: Reverse proxy (Most Secure)

Create /etc/nginx/sites-available/ollama:

server {
    listen 8080;
    server_name ollama.homelab.local;

    # Basic authentication
    auth_basic "Ollama API";
    auth_basic_user_file /etc/nginx/.htpasswd;

    location / {
        proxy_pass http://localhost:11434;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }
}

Enable and restart:

sudo ln -s /etc/nginx/sites-available/ollama /etc/nginx/sites-enabled/
sudo nginx -t
sudo systemctl restart nginx

Issue 6: Models Won’t Download

Symptoms:

• ollama pull stuck or very slow
• Download errors

Solutions:

1. Check network connectivity:

   curl -I https://registry.ollama.ai

2. Resume interrupted download:

   # Ollama automatically resumes
   ollama pull mistral

3. Check disk space:

   df -h
   # Need 10GB+ free per model

4. Try during off-peak hours:

   • Registry sometimes congested
   • Try 2-4 AM local time

Issue 7: Open WebUI Won’t Connect

Symptoms:

• Open WebUI can’t reach Ollama
• Connection errors in web interface

Diagnostic:

# Check Ollama is running
curl http://localhost:11434/api/tags

# From Open WebUI container
docker exec open-webui curl http://ollama:11434/api/tags

Solutions:

1. Verify connection URL:

   • Docker Compose: http://ollama:11434 (service name)
   • Native: http://localhost:11434
   • Remote: http://your-ip:11434

2. Check Docker network:

   docker network inspect bridge
   # Verify both containers on same network

3. Restart services:

   docker-compose restart
   # Or individually:
   docker restart ollama
   docker restart open-webui

Performance Benchmarking

Benchmark script (benchmark.sh):

#!/bin/bash
MODEL="mistral"
PROMPT="Explain artificial intelligence in 500 words"

echo "=== Ollama Performance Benchmark ==="
echo "Model: $MODEL"
echo ""

# Test 1: First token latency
echo "Test 1: First Token Latency"
time curl -s http://localhost:11434/api/generate   -d "{"model":"$MODEL","prompt":"$PROMPT","stream":true}" | head -1

# Test 2: Throughput
echo ""
echo "Test 2: Throughput (tokens/sec)"
curl -s http://localhost:11434/api/generate   -d "{"model":"$MODEL","prompt":"$PROMPT","stream":false}" |   jq -r '.eval_duration, .eval_count' |   awk 'NR==1{dur=$1} NR==2{tokens=$1} END{print tokens*1000000000/dur " tokens/sec"}'

# Test 3: Resource usage
echo ""
echo "Test 3: Peak Resource Usage"
echo "GPU VRAM:"
nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits
echo "System RAM:"
free -h | grep Mem | awk '{print $3 " / " $2}'

Usage:

chmod +x benchmark.sh
./benchmark.sh

---

Next Steps

Congratulations! You now have a working Ollama installation with the knowledge to:

• Select appropriate models for your hardware
• Integrate AI into applications via API
• Build custom models with Modelfiles
• Create automated workflows
• Troubleshoot common issues

Immediate Next Actions

1. Deploy your first production model:

   ollama pull llama3.3:8b
   ollama run llama3.3:8b

2. Set up Open WebUI for easier access:

   docker run -d -p 3000:8080 -v open-webui:/app/backend/data   ghcr.io/open-webui/open-webui:latest

3. Create a custom model for your use case:

   • Write a Modelfile with specialized system prompt
   • Build: ollama create my-assistant -f Modelfile

4. Build your first automation:

   • Start with document analysis or log monitoring
   • Integrate into existing homelab workflows

Advanced Topics to Explore

Fine-Tuning: Train models on your specific data for specialized tasks.

Multi-GPU Setup: Distribute large models across multiple GPUs for better performance.

Kubernetes Deployment: Scale Ollama across cluster for production workloads.

RAG Integration: Add vector databases (ChromaDB, Milvus) for knowledge-grounded responses.

Related Homelab Guides

• Self-Hosted Vector Databases: Weaviate, Milvus for RAG
• Container Orchestration: Kubernetes on homelabs
• Network Security: Secure homelab with VLANs and firewalls
• Storage Infrastructure: NAS setup for model storage

Learning Resources

Official:

• Ollama Website
• GitHub Repository
• Model Library
• Open WebUI Documentation

Community:

• Ollama Discord server
• r/ollama subreddit
• r/LocalLLaMA subreddit
• r/homelab subreddit

Model Performance:

• Hugging Face Leaderboards
• Community benchmarks for hardware comparisons

Cost-Benefit Analysis

Break-even calculator:

# Your usage
queries_per_day = 10
tokens_per_query = 200
days = 365

# Cloud cost (OpenAI GPT-3.5)
total_tokens = queries_per_day * tokens_per_query * days
cloud_cost_annual = (total_tokens / 1_000_000) * 1.50

# Ollama cost
hardware_cost = 1800  # Mid-range GPU setup
electricity_annual = 131  # ~30W average

# Break-even
breakeven_months = (hardware_cost + electricity_annual) / (cloud_cost_annual / 12)

print(f"Cloud cost (1 year): ${cloud_cost_annual:.2f}")
print(f"Ollama cost (1 year): ${hardware_cost + electricity_annual:.2f}")
print(f"Break-even: {breakeven_months:.1f} months")
print(f"5-year savings: ${cloud_cost_annual * 5 - (hardware_cost + electricity_annual * 5):.2f}")

When Ollama makes sense:

• 10+ queries daily → 18-month breakeven
• Privacy requirements → Immediate value
• Offline needs → No alternative
• Fine-tuning needs → Significant value

When cloud APIs work better:

• < 5 queries/week → Cloud cheaper
• Need latest models immediately → Cloud advantage
• No technical expertise → Cloud simpler

---

Conclusion

Ollama represents the democratization of AI for homelabbers and privacy-conscious developers. By running models locally, you gain:

Privacy: Your data never leaves your infrastructure Cost Savings: Break-even in 18 months, then $1,300+/year savings Control: Full customization, fine-tuning, and integration Autonomy: No dependency on cloud services or internet connectivity

The homelab community has embraced Ollama as the standard for local LLMs, with active development, extensive model support, and growing ecosystem integrations.

Start building today. Your homelab is ready for AI.

---

Additional Resources

Hardware Recommendations by Budget

Budget Tier ($500-800):

• NVIDIA RTX 4060 (8GB) + existing system
• Runs: Mistral 7B, Llama 3.3 8B, Phi-3-Mini
• Performance: 40-50 tokens/sec

Mid-Range Tier ($1200-1800):

• NVIDIA RTX 4070 (12GB) + 32GB RAM
• Runs: CodeLlama 34B, Llama 3.3 8B, Qwen3 32B
• Performance: 60-100 tokens/sec

High-End Tier ($3000+):

• NVIDIA RTX 4090 (24GB) + 64GB RAM
• Runs: Llama 3.3 70B, Qwen3 110B, all models
• Performance: 100+ tokens/sec

Multi-GPU Setup ($6000+):

• 2x RTX 4090 (48GB total) + 128GB RAM
• Runs: Llama 3.3 405B (with quantization), largest models
• Performance: 150+ tokens/sec

Quick Reference: Model Comparison

Environment Variables Reference

# Core settings
OLLAMA_HOST="0.0.0.0:11434"       # Bind address
OLLAMA_MODELS="/path/to/models"   # Model storage location
OLLAMA_KEEP_ALIVE="24h"           # Model memory retention

# Performance tuning
OLLAMA_NUM_PARALLEL=4             # Concurrent requests
OLLAMA_NUM_THREADS=8              # CPU threads
OLLAMA_GPU_LAYERS=50              # GPU utilization %

# Advanced
OLLAMA_MAX_LOADED_MODELS=3        # Max models in memory
OLLAMA_FLASH_ATTENTION=1          # Enable flash attention
OLLAMA_DEBUG=1                    # Enable debug logging

Apply to systemd service:

sudo systemctl edit ollama
# Add Environment= lines
sudo systemctl restart ollama

---

Guide Version: 1.0 Last Updated: December 2024 Author: homelabbing.io community License: CC BY-SA 4.0

Feedback? Submit issues or improvements to our GitHub repository.