ComfyUI: Advanced Stable Diffusion Workflows

ComfyUI is a powerful node-based interface for Stable Diffusion that transforms image generation from simple prompts into sophisticated, reusable workflows. Unlike user-friendly tools like AUTOMATIC1111 or Fooocus, ComfyUI is designed for creators who want complete control over every aspect of the generation pipeline - from initial noise to final output.

This comprehensive guide will teach you how to install ComfyUI, understand its node-based paradigm, create custom workflows for specific use cases, and leverage advanced features like ControlNet, IP-Adapter, and AnimateDiff for professional-grade image and video generation.

What You’ll Learn:

Install and configure ComfyUI on your homelab
Understand the node-based workflow system
Create custom workflows for different artistic styles
Use ControlNet for precise image control
Implement LoRA models for style customization
Generate videos with AnimateDiff
Optimize performance for your hardware

Time Investment: 1 hour for basic setup | 3-5 hours to master advanced workflows

Hardware Requirements: 8GB VRAM minimum | 12GB+ recommended | 16GB+ for video generation

Understanding ComfyUI

What is ComfyUI?

ComfyUI is a node-based workflow system for Stable Diffusion that provides:

Core Capabilities:

Visual Programming: Connect nodes to create custom generation pipelines
Workflow Reusability: Save and share complex workflows as JSON
Advanced Control: ControlNet, IP-Adapter, regional prompting, inpainting
Model Flexibility: Support for SD 1.5, SDXL, Cascade, and custom models
Extension System: Community plugins for additional functionality
API Access: Programmatic control for automation

Why ComfyUI Instead of AUTOMATIC1111?

Feature	ComfyUI	AUTOMATIC1111
Learning Curve	Steep	Gentle
Flexibility	Maximum	Limited
Workflow Sharing	Excellent	None
Performance	Optimized	Good
VRAM Efficiency	Excellent	Good
Advanced Features	First-class	Extension-dependent
Best For	Power users, professionals	Beginners, quick generation

Installation & Setup

Docker Installation (Recommended)

Step 1: Create Project Directory

mkdir -p /opt/docker/comfyui
cd /opt/docker/comfyui

Step 2: Create Docker Compose File

version: '3.8'

services:
  comfyui:
    image: ghcr.io/ai-dock/comfyui:latest-cuda
    container_name: comfyui
    restart: unless-stopped
    ports:
      - "8188:8188"
    volumes:
      # Model storage
      - comfyui_models:/opt/ComfyUI/models
      # Custom workflows
      - ./workflows:/opt/ComfyUI/user
      # Output images
      - ./output:/opt/ComfyUI/output
    environment:
      - NVIDIA_VISIBLE_DEVICES=all
      - NVIDIA_DRIVER_CAPABILITIES=all
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

volumes:
  comfyui_models:
    driver: local

Step 3: Deploy

docker compose up -d
docker compose logs -f

Step 4: Access UI

Navigate to: http://your-server-ip:8188

You’ll see the node-based interface with a default workflow loaded.

Manual Installation

For maximum control and performance:

Install Dependencies:

# Update system
sudo apt update && sudo apt upgrade -y

# Install Python 3.10+
sudo apt install -y python3.10 python3.10-venv python3-pip git

# Install NVIDIA drivers and CUDA (if not already)
sudo apt install -y nvidia-driver-535 nvidia-cuda-toolkit

Install ComfyUI:

# Clone repository
git clone https://github.com/comfyanonymous/ComfyUI.git
cd ComfyUI

# Create virtual environment
python3.10 -m venv venv
source venv/bin/activate

# Install PyTorch with CUDA support
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121

# Install ComfyUI dependencies
pip install -r requirements.txt

# Launch
python main.py --listen 0.0.0.0 --port 8188

Auto-Start with systemd:

[Unit]
Description=ComfyUI Stable Diffusion
After=network.target

[Service]
Type=simple
User=your_username
WorkingDirectory=/opt/ComfyUI
Environment="PATH=/opt/ComfyUI/venv/bin"
ExecStart=/opt/ComfyUI/venv/bin/python main.py --listen 0.0.0.0 --port 8188
Restart=always

[Install]
WantedBy=multi-user.target

Node-Based Workflow Basics

Understanding Nodes

Node Types:

Loaders: Load models, LoRAs, VAEs, ControlNet models
Conditioning: Process prompts (positive/negative)
Sampling: Generate latent images from noise
Latent: Manipulate latent space
Image: Convert latents to images, apply effects
Output: Save results

Basic Workflow Structure:

Checkpoint Loader → CLIP Text Encode (Positive) ↘
                                                  Sampler → VAE Decode → Save Image
Checkpoint Loader → CLIP Text Encode (Negative) ↗

Default Workflow Explained

When you open ComfyUI, you see a default txt2img workflow:

Load Checkpoint (Model) - Loads your Stable Diffusion model
CLIP Text Encode (Positive) - Your main prompt
CLIP Text Encode (Negative) - What to avoid
Empty Latent Image - Blank canvas (noise)
KSampler - Generation algorithm
VAE Decode - Converts latent to visible image
Save Image - Saves the result

To Generate an Image:

Click “Load Checkpoint” → Select your model
Enter prompt in “CLIP Text Encode (Positive)”
Enter negative prompt in “CLIP Text Encode (Negative)”
Click “Queue Prompt” (top right)
Watch generation progress
View result in the output window

Essential Workflows

Text-to-Image Workflow

Basic Setup:

Load Checkpoint
  ├─> CLIP Text Encode (Positive): "professional photo of a server room"
  ├─> CLIP Text Encode (Negative): "blurry, low quality"
  └─> Model → KSampler
                 ├─> Empty Latent (512x512)
                 ├─> Seed: random
                 ├─> Steps: 20
                 ├─> CFG: 7.0
                 ├─> Sampler: euler
                 └─> VAE Decode → Save Image

Key Parameters:

Steps: 15-30 (higher = more refinement, slower)
CFG Scale: 5-10 (how closely to follow prompt)
Sampler: euler, dpmpp_2m, ddim (each has different characteristics)
Scheduler: normal, karras, exponential
Seed: Random or fixed (for reproducibility)

Image-to-Image Workflow

Transform existing images:

Load Image → VAE Encode
                ↓
Load Checkpoint → KSampler (denoise: 0.7)
  ├─> CLIP Text Encode (Positive)
  └─> CLIP Text Encode (Negative)
                ↓
          VAE Decode → Save Image

Denoise Strength:

0.0-0.3: Subtle changes
0.4-0.6: Moderate transformation
0.7-1.0: Heavy changes (mostly new image)

Inpainting Workflow

Edit specific regions:

Load Image → VAE Encode → Inpaint Model Conditioning
                              ↓
Load Inpaint Model → KSampler
  ├─> Mask (draw in UI or upload)
  ├─> CLIP Text Encode (Positive)
  └─> CLIP Text Encode (Negative)
                ↓
          VAE Decode → Save Image

Use Cases:

Remove objects
Change backgrounds
Fix details
Add elements

ControlNet Integration

ControlNet provides pixel-level control over generation.

Installing ControlNet

Download Models:

cd /opt/ComfyUI/models/controlnet/

# Canny (edge detection)
wget https://huggingface.co/lllyasviel/ControlNet-v1-1/resolve/main/control_v11p_sd15_canny.pth

# Depth
wget https://huggingface.co/lllyasviel/ControlNet-v1-1/resolve/main/control_v11f1p_sd15_depth.pth

# OpenPose (human poses)
wget https://huggingface.co/lllyasviel/ControlNet-v1-1/resolve/main/control_v11p_sd15_openpose.pth

ControlNet Workflow

Canny Edge Detection:

Load Image → Canny Edge Detection → Apply ControlNet
                                          ↓
Load Checkpoint → KSampler ← Load ControlNet Model
  ├─> CLIP Text Encode (Positive)
  └─> CLIP Text Encode (Negative)
                ↓
          VAE Decode → Save Image

Example Use:

Upload reference image (e.g., architectural sketch)
Canny extracts edges
AI generates photorealistic image following those edges

Other ControlNet Types:

Depth: Preserve 3D structure
OpenPose: Match human poses
Scribble: Hand-drawn sketches to images
Normal Map: Surface detail control
Segmentation: Semantic layout control

LoRA Models

LoRA (Low-Rank Adaptation) adds specific styles or concepts to base models.

Installing LoRAs

Download from Civitai:

cd /opt/ComfyUI/models/loras/

# Example: Pixel Art style
wget https://civitai.com/api/download/models/XXX -O pixel-art.safetensors

Using LoRAs

Workflow:

Load Checkpoint → Load LoRA
  ├─> Model Weight: 1.0 (strength)
  └─> CLIP Weight: 1.0
        ↓
  CLIP Text Encode (Positive)
        ↓
  KSampler → VAE Decode → Save Image

Multiple LoRAs:

Load Checkpoint → Load LoRA 1 (0.8) → Load LoRA 2 (0.6) → Sampler

Stack LoRAs to combine styles (e.g., anime + cyberpunk + detailed).

Popular LoRA Types

Style LoRAs: Anime, oil painting, watercolor
Character LoRAs: Specific characters or people
Concept LoRAs: Sci-fi technology, fantasy creatures
Quality LoRAs: Add detail, improve hands/faces

Advanced Workflows

Multi-Model Comparison

Generate with multiple models simultaneously:

Load Checkpoint A ↘
                   KSampler A → VAE Decode → Save Image A
CLIP Text ↗       ↗
         ↘       ↗
Load Checkpoint B ↘
                   KSampler B → VAE Decode → Save Image B

Compare different models with same prompt and seed.

Upscaling Workflow

ESRGAN Upscaling:

Generate Base Image (512x512)
         ↓
   VAE Decode
         ↓
  Upscale Image (ESRGAN 4x)
         ↓
  VAE Encode → KSampler (denoise: 0.4)
         ↓
  VAE Decode → Save Image (2048x2048)

Result: High-resolution, detailed image from low-res generation.

AnimateDiff (Video Generation)

Install AnimateDiff:

cd /opt/ComfyUI/custom_nodes/
git clone https://github.com/Kosinkadink/ComfyUI-AnimateDiff-Evolved

Video Workflow:

Load Checkpoint → AnimateDiff Loader
  ├─> Motion Model
  ├─> CLIP Text Encode (Positive)
  └─> CLIP Text Encode (Negative)
         ↓
   AnimateDiff Sampler
         ↓
   VAE Decode Video
         ↓
   Save Video (MP4)

Parameters:

Frame Count: 16-48 frames
FPS: 8-24 fps
Context: How many frames to process together

Model Management

Model Types

Checkpoints (Main Models):

models/checkpoints/
SD 1.5: 2-4GB
SDXL: 6-13GB
Cascade: 10-20GB

VAE (Image Decoders):

models/vae/
Improves color accuracy and detail
Popular: vae-ft-mse-840000-ema-pruned

LoRAs:

models/loras/
10-200MB each
Stack multiple for combined effects

ControlNet:

models/controlnet/
1-2GB per model

Embeddings (Textual Inversions):

models/embeddings/
Add specific concepts to prompts

Recommended Models

Base Checkpoints:

# Realistic Photography
wget https://civitai.com/api/download/models/XXX -O realistic-vision-v5.safetensors

# Anime/Illustration
wget https://civitai.com/api/download/models/XXX -O anything-v5.safetensors

# SDXL (High Quality)
wget https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/resolve/main/sd_xl_base_1.0.safetensors

Custom Nodes & Extensions

ComfyUI Manager

Install the manager for easy extension installation:

cd /opt/ComfyUI/custom_nodes/
git clone https://github.com/ltdrdata/ComfyUI-Manager

Restart ComfyUI, then click “Manager” button in UI.

Essential Extensions

1. Efficiency Nodes

cd custom_nodes/
git clone https://github.com/jags111/efficiency-nodes-comfyui

Simplifies complex workflows with all-in-one nodes.

2. Image Saver

Advanced save options (metadata, organization).

3. WAS Node Suite

Text utilities, image processing, file operations.

4. Ultimate SD Upscale

Tile-based upscaling for huge images without VRAM limits.

Performance Optimization

VRAM Management

Low VRAM Mode (6-8GB):

python main.py --lowvram --listen 0.0.0.0

Very Low VRAM Mode (4-6GB):

python main.py --novram --listen 0.0.0.0

Faster Loading:

python main.py --fast --listen 0.0.0.0

Batch Generation

Generate multiple images efficiently:

Batch Size: Set in “Empty Latent Image” node

Batch of 4 = 4 images at once
More efficient than 4 separate generations

Model Precision

FP16 (Half Precision):

Default for GPUs
Balances speed and quality

FP32 (Full Precision):

CPU fallback
Slower but more accurate

Exporting Workflows

Save Workflow:

Click “Save” button
Saves as JSON file
Contains all node connections and settings

Share:

Upload JSON to GitHub, Discord, etc.
Others can drag-drop into ComfyUI

Importing Workflows

Method 1: Drag & Drop

Download workflow.json
Drag into ComfyUI interface
Nodes auto-arrange

Method 2: Load

Click “Load” button
Select JSON file

Embedding Workflows in Images

Generated images contain workflow metadata:

Extract Workflow:

Drag generated PNG into ComfyUI
Workflow automatically loads
Perfect for recreating results

Troubleshooting

Issue: Model Won’t Load

Solution:

# Check model location
ls -la /opt/ComfyUI/models/checkpoints/

# Verify file integrity
sha256sum model.safetensors

# Check logs
docker logs comfyui -f

Issue: Out of VRAM

Solutions:

Use --lowvram or --novram flags
Reduce batch size
Use smaller resolution
Unload models between generations

Issue: Slow Generation

Optimizations:

Use SDXL Turbo models (4 steps vs 20)
Enable xformers: --use-xformers
Reduce steps (15 instead of 30)
Lower resolution, then upscale

Next Steps

Learning Path

Week 1: Master basic txt2img and img2img Week 2: Learn ControlNet for precise control Week 3: Experiment with LoRAs and styles Week 4: Create custom workflows for your use cases

Community Resources

r/StableDiffusion
r/comfyui
Civitai (models and tutorials)
ComfyUI Discord

Guide Version: 1.0 Last Updated: December 2024 Difficulty: Advanced Time to Complete: 1-2 hours