Whisper: Self-Hosted Speech-to-Text Transcription

OpenAI’s Whisper is a state-of-the-art speech recognition model that rivals commercial transcription services - but runs entirely on your hardware. Whether you’re transcribing meetings, creating podcast summaries, or building voice interfaces for your homelab, Whisper delivers professional-quality results without sending your audio to the cloud.

This comprehensive guide covers deploying Whisper, choosing the right model size, optimizing for performance, and integrating with other services in your homelab ecosystem.

What You’ll Learn:

• Install Whisper and faster-whisper for optimized inference
• Choose model sizes for accuracy vs performance
• Create a REST API for transcription services
• Integrate with Home Assistant, n8n, and other tools
• Optimize for GPU and CPU performance
• Build practical automation workflows

Time Investment: 30 minutes for basic setup | 1-2 hours for integrations

Hardware Requirements: 4GB RAM minimum | 8GB+ recommended | GPU optional but faster

---

Understanding Whisper

What is Whisper?

Whisper is OpenAI’s open-source automatic speech recognition (ASR) system that:

Core Capabilities:

• Multilingual: Supports 99 languages
• Transcription: Speech to text in original language
• Translation: Speech to English text (any language)
• Robust: Handles accents, background noise, technical terminology
• Timestamps: Word-level and phrase-level timing
• Formats: Support for MP3, WAV, M4A, WebM, and more

Model Sizes:

Whisper vs Commercial Services

---

Installation

Method 1: faster-whisper (Recommended)

Optimized implementation with 4x speedup and lower memory:

# Install dependencies
pip install faster-whisper

# Python script
from faster_whisper import WhisperModel

model = WhisperModel("base", device="cuda", compute_type="float16")

segments, info = model.transcribe("audio.mp3")

for segment in segments:
    print(f"[{segment.start:.2f}s -> {segment.end:.2f}s] {segment.text}")

Method 2: Docker API Service (Best for Homelab)

Create docker-compose.yml:

version: '3.8'

services:
  whisper:
    image: onerahmet/openai-whisper-asr-webservice:latest
    container_name: whisper-api
    restart: unless-stopped
    ports:
      - "9000:9000"
    environment:
      - ASR_MODEL=base
      - ASR_ENGINE=faster_whisper
      - ASR_DEVICE=cuda
    volumes:
      - whisper_cache:/root/.cache/whisper
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

volumes:
  whisper_cache:

Deploy:

docker compose up -d

Test API:

curl -F "[email protected]" http://localhost:9000/asr

Method 3: Native Installation

# Install PyTorch
pip install torch torchvision torchaudio

# Install Whisper
pip install openai-whisper

# Transcribe
whisper audio.mp3 --model base --language English

---

Model Selection Guide

Performance vs Accuracy

For Quick Transcription (Meetings, Notes):

• Model: base or small
• Speed: 10-15x realtime (CPU)
• Use case: Fast drafts, real-time transcription

For Accurate Transcription (Podcasts, Interviews):

• Model: medium or large-v3
• Speed: 1-2x realtime (CPU) or 10x (GPU)
• Use case: Publication-ready transcripts

For Resource-Constrained (Raspberry Pi, Old Hardware):

• Model: tiny
• Speed: 20-30x realtime
• Use case: Good enough for basic transcription

Language-Specific Optimizations

English-Only:

model = WhisperModel("base.en", device="cuda")

Faster and more accurate for English-only content.

Multilingual:

segments, info = model.transcribe(
    "audio.mp3",
    language="es"  # Specify Spanish
)

Auto-Detect Language:

segments, info = model.transcribe("audio.mp3")
print(f"Detected language: {info.language}")

---

API Usage

REST API Examples

Transcribe Audio:

curl -X POST http://localhost:9000/asr   -F "[email protected]"   -F "task=transcribe"   -F "language=en"   -F "output=json"

Response:

{
  "text": "Welcome to today's meeting. Let's discuss the quarterly results...",
  "segments": [
    {
      "start": 0.0,
      "end": 3.5,
      "text": "Welcome to today's meeting."
    },
    ...
  ],
  "language": "en"
}

Translation to English:

curl -X POST http://localhost:9000/asr   -F "audio_file=@spanish_audio.mp3"   -F "task=translate"   -F "output=txt"

Returns English translation of Spanish audio.

Python Integration

import requests

def transcribe_audio(file_path, model="base"):
    url = "http://localhost:9000/asr"
    
    with open(file_path, "rb") as f:
        files = {"audio_file": f}
        data = {
            "task": "transcribe",
            "language": "en",
            "output": "json"
        }
        response = requests.post(url, files=files, data=data)
    
    return response.json()

# Usage
result = transcribe_audio("interview.mp3")
print(result["text"])

---

Practical Integrations

Home Assistant Voice Control

Create automation:

automation:
  - alias: "Voice Command Transcription"
    trigger:
      platform: webhook
      webhook_id: voice_command
    action:
      - service: shell_command.transcribe_audio
        data:
          file: "/config/www/voice_command.wav"
      - service: script.process_command
        data_template:
          command: "{{ state_attr('sensor.last_transcription', 'text') }}"

n8n Workflow (Meeting Transcription)

Workflow:

1. Trigger: File Upload (Nextcloud/Dropbox)
   ↓
2. HTTP Request: POST to Whisper API
   ↓
3. Process JSON Response
   ↓
4. Save to Notion/Obsidian/Google Docs
   ↓
5. Send Summary Email

n8n HTTP Node:

{
  "url": "http://whisper:9000/asr",
  "method": "POST",
  "bodyParameters": {
    "audio_file": "{{$binary.data}}",
    "task": "transcribe",
    "output": "json"
  }
}

Podcast Transcription Pipeline

Automated script:

#!/bin/bash
# Transcribe all MP3s in a directory

PODCAST_DIR="/media/podcasts"
OUTPUT_DIR="/media/transcripts"

for audio in "$PODCAST_DIR"/*.mp3; do
  filename=$(basename "$audio" .mp3)
  
  echo "Transcribing: $filename"
  
  curl -X POST http://localhost:9000/asr     -F "audio_file=@$audio"     -F "task=transcribe"     -F "output=srt"     -o "$OUTPUT_DIR/$filename.srt"
  
  echo "Saved: $OUTPUT_DIR/$filename.srt"
done

Schedule with cron:

0 2 * * * /opt/scripts/transcribe-podcasts.sh

Meeting Assistant Bot

Real-time transcription:

from faster_whisper import WhisperModel
import sounddevice as sd
import numpy as np
import queue
import sys

# Audio recording
audio_queue = queue.Queue()

def audio_callback(indata, frames, time, status):
    audio_queue.put(indata.copy())

# Whisper model
model = WhisperModel("base", device="cuda")

# Record and transcribe
with sd.InputStream(callback=audio_callback, channels=1, samplerate=16000):
    print("Recording... (Ctrl+C to stop)")
    
    while True:
        # Collect 30 seconds of audio
        chunks = []
        for _ in range(30):
            chunks.append(audio_queue.get())
        
        audio = np.concatenate(chunks)
        
        # Transcribe
        segments, _ = model.transcribe(audio)
        
        for segment in segments:
            print(f"[{segment.start:.1f}s] {segment.text}")
            sys.stdout.flush()

---

Performance Optimization

GPU Acceleration

NVIDIA CUDA:

from faster_whisper import WhisperModel

# Use GPU with FP16
model = WhisperModel("base", device="cuda", compute_type="float16")

# Or INT8 for even faster (slight quality loss)
model = WhisperModel("base", device="cuda", compute_type="int8")

Performance Gain:

• CPU: 2x realtime
• GPU (FP16): 10-20x realtime
• GPU (INT8): 20-40x realtime

CPU Optimization

Multi-threading:

model = WhisperModel(
    "base",
    device="cpu",
    compute_type="int8",
    cpu_threads=8  # Match your CPU cores
)

Batch Processing

Process multiple files:

from faster_whisper import WhisperModel
import os

model = WhisperModel("base", device="cuda")

audio_files = [f for f in os.listdir("audio/") if f.endswith(".mp3")]

for audio_file in audio_files:
    print(f"Transcribing: {audio_file}")
    
    segments, _ = model.transcribe(f"audio/{audio_file}")
    
    with open(f"transcripts/{audio_file}.txt", "w") as f:
        for segment in segments:
            f.write(segment.text + "\n")

---

Advanced Features

Speaker Diarization

Identify different speakers:

# Install pyannote
pip install pyannote.audio

# Download diarization model
huggingface-cli login
# Get token from https://huggingface.co/settings/tokens

Diarization Script:

from pyannote.audio import Pipeline
from faster_whisper import WhisperModel

# Speaker diarization
diarization = Pipeline.from_pretrained("pyannote/speaker-diarization")
diarization_result = diarization("audio.mp3")

# Transcription
whisper = WhisperModel("base")
segments, _ = whisper.transcribe("audio.mp3")

# Combine
for segment in segments:
    # Find speaker at this time
    for turn, _, speaker in diarization_result.itertracks(yield_label=True):
        if turn.start <= segment.start <= turn.end:
            print(f"[{speaker}] {segment.text}")
            break

Subtitle Generation

SRT format:

segments, _ = model.transcribe("video.mp4")

with open("subtitles.srt", "w") as f:
    for i, segment in enumerate(segments, 1):
        start = format_time(segment.start)
        end = format_time(segment.end)
        
        f.write(f"{i}\n")
        f.write(f"{start} --> {end}\n")
        f.write(f"{segment.text}\n\n")

def format_time(seconds):
    hours = int(seconds // 3600)
    minutes = int((seconds % 3600) // 60)
    secs = int(seconds % 60)
    millis = int((seconds % 1) * 1000)
    return f"{hours:02}:{minutes:02}:{secs:02},{millis:03}"

Streaming Transcription

Real-time transcription:

from faster_whisper import WhisperModel
import pyaudio

model = WhisperModel("base.en", device="cuda")

# Audio stream
p = pyaudio.PyAudio()
stream = p.open(format=pyaudio.paInt16, channels=1, rate=16000, input=True)

buffer = []
while True:
    data = stream.read(16000)  # 1 second
    buffer.append(data)
    
    if len(buffer) >= 30:  # Every 30 seconds
        audio = np.frombuffer(b''.join(buffer), dtype=np.int16)
        segments, _ = model.transcribe(audio)
        
        for segment in segments:
            print(segment.text)
        
        buffer = buffer[-5:]  # Keep 5s overlap

---

Troubleshooting

Issue: Slow Transcription

Solutions:

1. Use smaller model: tiny or base
2. Enable GPU: device="cuda"
3. Use INT8: compute_type="int8"
4. Use faster-whisper instead of openai-whisper

Issue: Poor Accuracy

Solutions:

1. Use larger model: medium or large-v3
2. Specify language: language="en"
3. Improve audio quality (denoise, remove background)
4. Use English-specific model: base.en

Issue: Out of Memory

Solutions:

# Reduce batch size
model = WhisperModel("base", device="cuda", compute_type="int8")

# Or use CPU
model = WhisperModel("base", device="cpu")

---

Next Steps

Integration Ideas

• Automated podcast transcription and indexing
• Voice-controlled homelab commands
• Meeting minutes generator
• Multilingual customer support transcription
• Accessibility features for hearing impaired

Related Guides

• Open WebUI: ChatGPT Interface - Use transcriptions as LLM input
• n8n Automation - Build transcription workflows

Guide Version: 1.0 Last Updated: December 2024 Difficulty: Intermediate Time to Complete: 30-45 minutes