LocalAI: OpenAI API-Compatible Local Inference

LocalAI is a drop-in replacement for OpenAI’s API that runs entirely on your hardware. Unlike Ollama which focuses on simplicity, LocalAI provides maximum compatibility with the OpenAI ecosystem - meaning thousands of existing applications and tools work with it immediately, with zero code changes.

This comprehensive guide will show you how to deploy LocalAI, load models from multiple sources, leverage GPU acceleration, and integrate it with popular AI applications that expect OpenAI’s API.

What You’ll Learn:

• Install and configure LocalAI for maximum compatibility
• Load models from GGUF, Hugging Face, and other sources
• Enable GPU acceleration for inference
• Configure the model gallery for easy deployment
• Integrate with existing OpenAI-compatible tools
• Optimize performance and troubleshoot issues

Time Investment: 45 minutes for basic setup | 2-3 hours for advanced configuration

Hardware Requirements: 8GB RAM minimum (16GB+ recommended) | GPU strongly recommended

---

Table of Contents

1. Understanding LocalAI
2. Installation & Setup
3. Model Loading Strategies
4. GPU Acceleration
5. Model Gallery
6. API Compatibility
7. Practical Integrations
8. Performance Optimization
9. Troubleshooting
10. Next Steps

---

Understanding LocalAI

What is LocalAI?

LocalAI is a free, open-source, self-hosted alternative to OpenAI that provides:

Core Capabilities:

• OpenAI API Compatibility: Drop-in replacement for api.openai.com
• Multi-Model Support: GGUF, GPTQ, Safetensors, ONNX, and more
• Multi-Modal: Text generation, image generation, audio transcription, embeddings
• GPU Acceleration: NVIDIA CUDA, AMD ROCm, Apple Metal, Intel oneAPI
• No External Dependencies: Runs completely offline
• REST API: Full OpenAI-compatible REST endpoints
• Model Gallery: Pre-configured models ready to deploy

Architecture:

Application Layer (OpenAI SDK, LangChain, etc.)
    ↓
LocalAI REST API (OpenAI-compatible endpoints)
    ↓
Backend Engines (llama.cpp, diffusers, whisper.cpp, etc.)
    ↓
Hardware Acceleration (CUDA, ROCm, Metal, CPU)
    ↓
Models (GGUF, Safetensors, ONNX, etc.)

LocalAI vs Ollama vs OpenAI

When to Choose LocalAI:

• You need OpenAI API compatibility for existing apps
• You want multi-modal capabilities (text + images + audio)
• You need maximum flexibility in model selection
• You’re migrating from OpenAI to self-hosted
• You want to use tools like LangChain, AutoGPT, etc.

When to Choose Ollama:

• You want simplest possible setup
• You only need text generation
• You prefer opinionated, streamlined tools
• You don’t need OpenAI API compatibility

---

Installation & Setup

Docker Installation (Recommended)

Step 1: Create Project Directory

mkdir -p /opt/docker/localai
cd /opt/docker/localai

Step 2: Create Docker Compose File

version: '3.8'

services:
  localai:
    image: quay.io/go-skynet/local-ai:latest
    container_name: localai
    restart: unless-stopped
    ports:
      - "8080:8080"
    volumes:
      # Model storage
      - localai_models:/models
      # Configuration
      - ./localai_config:/build/models
    environment:
      # Performance tuning
      - THREADS=8
      - CONTEXT_SIZE=4096
      - MODELS_PATH=/models
      # GPU support (NVIDIA)
      - CUDA_VISIBLE_DEVICES=0
      - BUILD_TYPE=cublas
      # Features
      - GALLERIES=[{"name":"model-gallery","url":"https://raw.githubusercontent.com/mudler/LocalAI/master/embedded/model_library.yaml"}]
      - PRELOAD_MODELS=[{"id":"phi-3-mini","name":"phi-3-mini"}]
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

volumes:
  localai_models:
    driver: local

Step 3: Deploy

docker compose up -d

# Monitor logs
docker compose logs -f

# Wait for startup (30-60 seconds)

Step 4: Verify Installation

# Check health
curl http://localhost:8080/readiness

# List loaded models
curl http://localhost:8080/v1/models

# Test text generation
curl http://localhost:8080/v1/completions   -H "Content-Type: application/json"   -d '{
    "model": "phi-3-mini",
    "prompt": "The meaning of life is",
    "max_tokens": 50
  }'

GPU-Specific Images

NVIDIA CUDA:

services:
  localai:
    image: quay.io/go-skynet/local-ai:latest-gpu-nvidia-cuda-12
    # ... rest of config

AMD ROCm:

services:
  localai:
    image: quay.io/go-skynet/local-ai:latest-gpu-hipblas
    devices:
      - /dev/kfd
      - /dev/dri
    # ... rest of config

Intel oneAPI:

services:
  localai:
    image: quay.io/go-skynet/local-ai:latest-intel
    # ... rest of config

CPU-Only (No GPU):

services:
  localai:
    image: quay.io/go-skynet/local-ai:latest
    # Remove deploy.resources.reservations.devices

Binary Installation (Advanced)

For maximum performance and control:

# Download pre-built binary
curl -Lo local-ai https://github.com/mudler/LocalAI/releases/download/v2.9.0/local-ai-Linux-x86_64
chmod +x local-ai

# Or build from source
git clone https://github.com/go-skynet/LocalAI
cd LocalAI
make build

# Run
./local-ai --models-path /opt/models --threads 8 --context-size 4096

---

Model Loading Strategies

LocalAI supports multiple model formats and loading methods.

Method 1: Model Gallery (Easiest)

Pre-configured models ready to download and use.

List Available Models:

curl http://localhost:8080/models/available

Install from Gallery:

# Via API
curl -X POST http://localhost:8080/models/apply   -H "Content-Type: application/json"   -d '{
    "id": "mistral-7b-instruct",
    "name": "mistral-7b"
  }'

# Check installation progress
curl http://localhost:8080/models/jobs

Popular Gallery Models:

Method 2: Manual GGUF Models

Download and configure models manually for maximum control.

Step 1: Download Model

# Create models directory
mkdir -p /opt/docker/localai/models

# Download from Hugging Face
curl -L https://huggingface.co/TheBloke/Mistral-7B-Instruct-v0.2-GGUF/resolve/main/mistral-7b-instruct-v0.2.Q4_K_M.gguf   -o /opt/docker/localai/models/mistral-7b.gguf

Step 2: Create Configuration

Create /opt/docker/localai/localai_config/mistral-7b.yaml:

name: mistral-7b
backend: llama
parameters:
  model: mistral-7b.gguf

# Context and generation
context_size: 4096
f16: true  # Use 16-bit floats
threads: 8

# Generation parameters
temperature: 0.7
top_k: 40
top_p: 0.9
max_tokens: 2048

# GPU layers (adjust based on VRAM)
gpu_layers: 35

# Prompt template
template:
  chat: |
    <s>[INST] {{ .Input }} [/INST]
  completion: |
    {{ .Input }}

Step 3: Load Model

Restart LocalAI to pick up new configuration:

docker compose restart localai

# Verify model loaded
curl http://localhost:8080/v1/models

Method 3: Hugging Face Integration

Load models directly from Hugging Face:

Configuration file (llama3-hf.yaml):

name: llama3-hf
backend: transformers
parameters:
  model: meta-llama/Meta-Llama-3-8B-Instruct

# Quantization for efficiency
quantization: 4bit
load_in_4bit: true

# GPU configuration
device_map: auto
torch_dtype: float16

temperature: 0.7
max_tokens: 2048

This downloads the model from Hugging Face on first use (requires internet).

Method 4: Pre-loading Models

Configure models to load automatically on startup:

Update docker-compose.yml:

environment:
  - PRELOAD_MODELS=[
      {"id":"mistral-7b-instruct","name":"mistral"},
      {"id":"phi-3-mini","name":"phi3"},
      {"id":"codellama-7b","name":"codellama"}
    ]

Models download and load automatically on container start.

Model Configuration Reference

Common Parameters:

name: model-name            # API reference name
backend: llama              # Backend engine (llama, transformers, etc.)
parameters:
  model: model-file.gguf    # Model file name

# Performance
context_size: 4096          # Context window
threads: 8                  # CPU threads
gpu_layers: 35              # Layers offloaded to GPU
batch_size: 512             # Batch processing size
f16: true                   # Use 16-bit precision

# Generation
temperature: 0.7            # Randomness (0.0-2.0)
top_k: 40                   # Top-K sampling
top_p: 0.9                  # Nucleus sampling
max_tokens: 2048            # Max response length
repeat_penalty: 1.1         # Prevent repetition

# Prompt formatting
template:
  chat: "template here"
  completion: "template here"
  system: "system template"

---

GPU Acceleration

NVIDIA CUDA Setup

Verify GPU Detection:

# From host
nvidia-smi

# From container
docker exec localai nvidia-smi

Configure GPU Layers:

# In model config
gpu_layers: 35  # Adjust based on VRAM

# All layers (maximum GPU usage)
gpu_layers: 999

# No GPU (CPU only)
gpu_layers: 0

GPU Memory Management:

# Limit GPU memory usage
environment:
  - CUDA_VISIBLE_DEVICES=0        # Use first GPU only
  - CUDA_MEM_FRACTION=0.8         # Use 80% of VRAM

AMD ROCm Setup

Install ROCm drivers on host:

# Ubuntu/Debian
wget https://repo.radeon.com/amdgpu-install/latest/ubuntu/jammy/amdgpu-install_*.deb
sudo dpkg -i amdgpu-install_*.deb
sudo amdgpu-install --usecase=rocm --no-dkms

# Verify
rocm-smi

Docker configuration:

services:
  localai:
    image: quay.io/go-skynet/local-ai:latest-gpu-hipblas
    devices:
      - /dev/kfd
      - /dev/dri
    group_add:
      - video
    environment:
      - HSA_OVERRIDE_GFX_VERSION=10.3.0  # For RX 6000 series
      - BUILD_TYPE=hipblas

Apple Metal Setup

Only works with native installation (not Docker):

# Build with Metal support
git clone https://github.com/go-skynet/LocalAI
cd LocalAI
BUILD_TYPE=metal make build

# Run
./local-ai --models-path ./models

Metal acceleration automatic on M1/M2/M3 Macs.

Performance Tuning

Optimal Settings by Hardware:

RTX 4060 (8GB VRAM):

gpu_layers: 30
context_size: 4096
batch_size: 512
# Runs 7B models at 60-80 tokens/sec

RTX 4070 (12GB VRAM):

gpu_layers: 35
context_size: 8192
batch_size: 1024
# Runs 13B models at 40-60 tokens/sec

RTX 4090 (24GB VRAM):

gpu_layers: 999  # All layers
context_size: 16384
batch_size: 2048
# Runs 34B models at 30-50 tokens/sec

CPU-Only (16 cores, 32GB RAM):

gpu_layers: 0
threads: 16
context_size: 4096
batch_size: 128
# Runs 7B models at 5-10 tokens/sec

---

Model Gallery

LocalAI’s model gallery provides pre-configured models for various tasks.

Text Generation Models

General Purpose:

# Mistral 7B (best balanced model)
curl -X POST http://localhost:8080/models/apply -d '{"id":"mistral-7b-instruct"}'

# Llama 3.1 8B (high quality)
curl -X POST http://localhost:8080/models/apply -d '{"id":"llama3.1-8b"}'

# Phi-3 Mini (fast, lightweight)
curl -X POST http://localhost:8080/models/apply -d '{"id":"phi-3-mini"}'

Code Generation:

# CodeLlama 7B
curl -X POST http://localhost:8080/models/apply -d '{"id":"codellama-7b"}'

# WizardCoder 7B
curl -X POST http://localhost:8080/models/apply -d '{"id":"wizardcoder-7b"}'

# DeepSeek Coder 6.7B
curl -X POST http://localhost:8080/models/apply -d '{"id":"deepseek-coder-6.7b"}'

Specialized Models:

# SQL Generation
curl -X POST http://localhost:8080/models/apply -d '{"id":"sqlcoder-7b"}'

# Math & Reasoning
curl -X POST http://localhost:8080/models/apply -d '{"id":"mathstral-7b"}'

Image Generation Models

Stable Diffusion:

# Stable Diffusion XL
curl -X POST http://localhost:8080/models/apply -d '{"id":"stablediffusion-xl"}'

# Stable Diffusion 1.5
curl -X POST http://localhost:8080/models/apply -d '{"id":"stablediffusion"}'

Generate Images:

curl http://localhost:8080/v1/images/generations   -H "Content-Type: application/json"   -d '{
    "model": "stablediffusion",
    "prompt": "a photograph of a futuristic server room",
    "size": "512x512",
    "n": 1
  }' --output image.png

Audio Transcription

Whisper Models:

# Whisper Base (fastest)
curl -X POST http://localhost:8080/models/apply -d '{"id":"whisper-base"}'

# Whisper Medium (better quality)
curl -X POST http://localhost:8080/models/apply -d '{"id":"whisper-medium"}'

# Whisper Large (best quality)
curl -X POST http://localhost:8080/models/apply -d '{"id":"whisper-large"}'

Transcribe Audio:

curl http://localhost:8080/v1/audio/transcriptions   -H "Content-Type: multipart/form-data"   -F file="@audio.mp3"   -F model="whisper-base"

Embedding Models

# Sentence Transformers
curl -X POST http://localhost:8080/models/apply -d '{"id":"all-minilm-l6-v2"}'

# OpenAI-compatible embeddings
curl http://localhost:8080/v1/embeddings   -H "Content-Type: application/json"   -d '{
    "model": "all-minilm-l6-v2",
    "input": "Your text here"
  }'

Custom Gallery

Create your own model gallery:

Create custom-gallery.yaml:

- name: "My Custom Models"
  models:
    - name: homelab-assistant
      description: "Specialized homelab AI"
      urls:
        - https://example.com/models/homelab.gguf
      config_file:
        name: homelab-assistant
        backend: llama
        parameters:
          model: homelab.gguf
        gpu_layers: 35
        context_size: 4096
        template:
          chat: "<s>[INST] {{ .Input }} [/INST]"

Configure LocalAI:

environment:
  - GALLERIES=[{"name":"custom","url":"https://your-server/custom-gallery.yaml"}]

---

API Compatibility

LocalAI implements OpenAI’s API specification, making it compatible with thousands of existing tools.

Text Completion

OpenAI SDK (Python):

from openai import OpenAI

# Point to LocalAI instead of OpenAI
client = OpenAI(
    base_url="http://localhost:8080/v1",
    api_key="not-needed"  # LocalAI doesn't require key
)

# Exact same code as OpenAI
response = client.chat.completions.create(
    model="mistral-7b",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Explain Docker networking"}
    ]
)

print(response.choices[0].message.content)

Streaming Responses:

stream = client.chat.completions.create(
    model="mistral-7b",
    messages=[{"role": "user", "content": "Write a story"}],
    stream=True
)

for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")

Embeddings

# Generate embeddings
response = client.embeddings.create(
    model="all-minilm-l6-v2",
    input="Your text to embed"
)

embedding = response.data[0].embedding
# Use for similarity search, clustering, etc.

Function Calling

LocalAI supports OpenAI’s function calling:

functions = [
    {
        "name": "get_weather",
        "description": "Get current weather for a location",
        "parameters": {
            "type": "object",
            "properties": {
                "location": {"type": "string"},
                "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}
            },
            "required": ["location"]
        }
    }
]

response = client.chat.completions.create(
    model="mistral-7b",
    messages=[{"role": "user", "content": "What's the weather in Paris?"}],
    functions=functions,
    function_call="auto"
)

# Check if function should be called
if response.choices[0].message.function_call:
    function_name = response.choices[0].message.function_call.name
    arguments = response.choices[0].message.function_call.arguments
    # Execute function and return result to AI

Image Generation

response = client.images.generate(
    model="stablediffusion",
    prompt="a photograph of a server rack in a datacenter",
    size="512x512",
    n=1
)

image_url = response.data[0].url

Audio Transcription

audio_file = open("speech.mp3", "rb")

transcript = client.audio.transcriptions.create(
    model="whisper-base",
    file=audio_file
)

print(transcript.text)

---

Practical Integrations

LangChain Integration

from langchain.llms import OpenAI
from langchain.chains import LLMChain
from langchain.prompts import PromptTemplate

# Configure LangChain to use LocalAI
llm = OpenAI(
    openai_api_base="http://localhost:8080/v1",
    openai_api_key="not-needed",
    model_name="mistral-7b"
)

# Use normally
template = "Write a {adjective} story about {topic}"
prompt = PromptTemplate(template=template, input_variables=["adjective", "topic"])
chain = LLMChain(llm=llm, prompt=prompt)

result = chain.run(adjective="funny", topic="a robot learning to code")
print(result)

RAG with LangChain:

from langchain.embeddings import OpenAIEmbeddings
from langchain.vectorstores import Chroma
from langchain.chains import RetrievalQA

# Use LocalAI for embeddings
embeddings = OpenAIEmbeddings(
    openai_api_base="http://localhost:8080/v1",
    openai_api_key="not-needed",
    model="all-minilm-l6-v2"
)

# Create vector store
vectorstore = Chroma.from_documents(documents, embeddings)

# RAG chain
qa = RetrievalQA.from_chain_type(
    llm=llm,
    chain_type="stuff",
    retriever=vectorstore.as_retriever()
)

response = qa.run("What is Kubernetes?")

AutoGPT Integration

Configure AutoGPT to use LocalAI:

Edit .env:

OPENAI_API_BASE=http://localhost:8080/v1
OPENAI_API_KEY=not-needed
FAST_LLM_MODEL=mistral-7b
SMART_LLM_MODEL=llama3.1-8b
EMBEDDING_MODEL=all-minilm-l6-v2

Run AutoGPT normally - it uses LocalAI for all operations.

Semantic Kernel Integration

using Microsoft.SemanticKernel;

var kernel = Kernel.Builder
    .WithOpenAIChatCompletionService(
        modelId: "mistral-7b",
        apiKey: "not-needed",
        endpoint: "http://localhost:8080"
    )
    .Build();

// Use normally
var prompt = "Explain quantum computing";
var result = await kernel.InvokeSemanticFunctionAsync(prompt);
Console.WriteLine(result);

LibreChat Integration

Point LibreChat to LocalAI for a self-hosted ChatGPT alternative:

docker-compose.yml:

services:
  librechat:
    image: ghcr.io/danny-avila/librechat:latest
    environment:
      - OPENAI_API_KEY=not-needed
      - OPENAI_REVERSE_PROXY=http://localai:8080/v1

Full ChatGPT-like interface powered by LocalAI.

---

Performance Optimization

Concurrent Requests

Configure parallel processing:

environment:
  - PARALLEL_REQUESTS=4      # Process 4 requests simultaneously
  - SINGLE_ACTIVE_BACKEND=false  # Allow multiple backend instances

Model Caching

Keep models in memory:

environment:
  - PRELOAD_MODELS_CONFIG='{"models":[{"id":"mistral-7b"}]}'
  - MODELS_CACHE_ENABLED=true

Context Caching

Enable prompt caching:

# In model config
cache_prompts: true
mmap: true          # Memory-map model file
mlock: true         # Lock model in RAM

Batch Processing

Process multiple prompts efficiently:

# Instead of sequential requests
for prompt in prompts:
    response = client.completions.create(model="mistral", prompt=prompt)

# Use batch API
response = client.completions.create(
    model="mistral",
    prompt=prompts,  # List of prompts
    max_tokens=100
)

Resource Monitoring

# Monitor GPU usage
watch -n 1 nvidia-smi

# Monitor API performance
curl http://localhost:8080/metrics  # Prometheus metrics

# Container resource usage
docker stats localai

---

Troubleshooting

Issue 1: Model Won’t Load

Symptoms:

• “Model not found” errors
• API returns 404 for model

Solution:

# List available models
curl http://localhost:8080/v1/models

# Check model file exists
docker exec localai ls -la /models/

# Check configuration
docker exec localai cat /build/models/model-name.yaml

# Review logs
docker logs localai -f | grep -i error

Issue 2: Slow Inference

Diagnosis:

# Benchmark
time curl http://localhost:8080/v1/completions   -d '{"model":"mistral-7b","prompt":"Hello","max_tokens":50}'

Solutions:

1. Increase GPU layers: gpu_layers: 999
2. Reduce context: context_size: 2048
3. Lower precision: f16: true or use Q4 quantization
4. More threads: THREADS=16

Issue 3: Out of Memory

Solution:

# Reduce GPU layers
gpu_layers: 20  # Down from 35

# Smaller batch size
batch_size: 256  # Down from 512

# Smaller context
context_size: 2048  # Down from 4096

---

Next Steps

Related Guides

• Ollama: Self-Hosted Local LLMs
• Open WebUI: ChatGPT-Style Interface
• Continue.dev + Ollama

Advanced Topics

• Multi-model routing and load balancing
• Custom backend integration
• Fine-tuning and model optimization
• Production deployment with Kubernetes

Guide Version: 1.0 Last Updated: December 2024 Difficulty: Intermediate Time to Complete: 45 minutes