ROCm-Research-Archive/_TestScripts/Danis ROCm Kernel Patch Research/ZImage_Documentation.md

# Z-Image-Turbo on AMD BC-250 — Complete Guide

> **Hardware**: AMD BC-250 (Cyan Skillfish, gfx1013→gfx1010, 24 CUs, shared RAM)
> **Backend**: ROCm 7.2.0 / HIP (hipBLAS)
> **OS**: CachyOS, kernel 6.18.8-3-cachyos with v3 kernel patches
> **Server**: [stable-diffusion.cpp-restapi](https://github.com/leejet/stable-diffusion.cpp) (sdcpp-restapi)

---

## 1. Overview

Z-Image-Turbo is a Stable Diffusion model optimized for fast image generation. On the BC-250, it runs via the ROCm/HIP backend using `sdcpp-restapi` as an HTTP server. The server provides a REST API and a WebUI for image generation.

### Architecture

```
[User / WebUI] → HTTP :8080 → [sdcpp-restapi] → [stable-diffusion.cpp] → [HIP/hipBLAS] → [AMD BC-250 GPU]
```

### Pipeline Phases

Each image generation goes through three phases:

| Phase | What it does | Time (BC-250) |
|-------|-------------|---------------|
| **Text-Encoding** (CLIP) | Encodes the text prompt into embeddings | ~0.5–0.6s |
| **Sampling** (Diffusion) | Iterative denoising (8 steps with Euler sampler) | ~73–78s |
| **VAE Decode** | Decodes latent space back to pixel image | ~14s |
| **Total** | End-to-end generation | **~82–92s** |

---

## 2. Directory Layout

```
~/stable-diffusion.cpp-restapi/
├── build/
│   ├── bin/sdcpp-restapi          # Server binary
│   └── config.json                # Server configuration
├── src/                           # Server source code
└── build/_deps/stable-diffusion-src/  # SD library (fetched via CMake)

~/sd-models/
├── diffusion_models/
│   └── z_image_turbo-Q5_K_S.gguf  # Main diffusion model (5.2 GB, Q5_K_S quantized)
├── vae/
│   └── ae.safetensors             # VAE decoder (335 MB, f32)
├── llm/
│   ├── Qwen3-4B-Instruct-2507-Q5_K_S.gguf  # LLM for prompt enhancement (2.8 GB)
│   ├── Qwen3-8B-Q4_K_M.gguf                # Alternative LLM (5.0 GB)
│   └── gpt-oss-20b-Q4_K_M.gguf             # Large LLM (11.6 GB) — too big for BC-250
├── lora/
├── clip/
├── controlnet/
├── esrgan/
└── taesd/

~/sd-outputs/                      # Generated images output directory

~/start-zimage.sh                  # Launch script
~/VibeROCm/ZImage_Documentation.md # This document
```

---

## 3. Building from Source

```bash
cd ~/stable-diffusion.cpp-restapi
mkdir -p build && cd build
cmake .. -G Ninja \
  -DCMAKE_BUILD_TYPE=Release \
  -DSD_HIP=ON \
  -DSDCPP_WEBUI=ON \
  -DCMAKE_PREFIX_PATH=/opt/rocm
ninja -j$(nproc --all)
```

**Requirements**: ROCm 7.2.0, CMake, Ninja, hipBLAS/rocBLAS

---

## 4. Configuration

### Server Config (`~/stable-diffusion.cpp-restapi/build/config.json`)

```json
{
    "server": {
        "host": "0.0.0.0",
        "port": 8080,
        "threads": 8
    },
    "paths": {
        "diffusion_models": "/home/dars/sd-models/diffusion_models",
        "vae": "/home/dars/sd-models/vae",
        "llm": "/home/dars/sd-models/llm",
        "lora": "/home/dars/sd-models/lora",
        "clip": "/home/dars/sd-models/clip",
        "controlnet": "/home/dars/sd-models/controlnet",
        "esrgan": "/home/dars/sd-models/esrgan",
        "taesd": "/home/dars/sd-models/taesd",
        "output": "/home/dars/sd-outputs"
    },
    "sd_defaults": {
        "n_threads": 10,
        "keep_clip_on_cpu": true,
        "keep_vae_on_cpu": true,
        "flash_attn": false,
        "offload_to_cpu": true,
        "free_params_immediately": true
    }
}
```

### Environment Variables (set in `start-zimage.sh`)

#### Required

| Variable | Value | Why |
|----------|-------|-----|
| `HSA_OVERRIDE_GFX_VERSION` | `10.1.0` | BC-250 (gfx1013) needs gfx1010 spoof for ROCm |
| `HSA_ENABLE_SDMA` | `0` | SDMA engine has hardware bugs on gfx1013 |
| `HIP_VISIBLE_DEVICES` | `0` | Select the BC-250 GPU |
| `ROCM_PATH` | `/opt/rocm` | ROCm installation path |
| `HSA_TOOLS_LIB` | `""` | Disable profiling tools (stability) |
| `HSA_TOOLS_REPORT_LOAD_FAILURE` | `0` | Suppress tool warnings |

#### Explicitly Unset (performance-critical!)

These old workaround variables were used during development and **destroy performance** if set. The start script explicitly unsets them:

| Variable | Effect if set | Impact |
|----------|--------------|--------|
| `GPU_MAX_HW_QUEUES=1` | Serializes all GPU operations to 1 queue | Severe slowdown |
| `HIP_LAUNCH_BLOCKING=1` | Forces synchronous kernel launches | Prevents pipelining |
| `GGML_CUDA_ENABLE_UNIFIED_MEMORY=1` | Uses hipMallocManaged (page fault overhead) | +20% slower |
| `GGML_HIP_HOST_ALLOC=1` | Uses hipHostMalloc (zero-copy over PCIe) | +40% slower |
| `GGML_CUDA_NO_PINNED=1` | Disables pinned memory pools | Not needed with v3 patches |
| `GGML_HIP_NO_COARSE_GRAIN=1` | Disables coarse-grain memory | Not needed with v3 patches |
| `HSA_DISABLE_FRAGMENT_ALLOCATOR=1` | Disables memory fragment allocator | Not needed with v3 patches |

---

## 5. Starting the Server

### Using the Launch Script (recommended)

```bash
bash ~/start-zimage.sh
```

Or in background with logging:

```bash
nohup bash ~/start-zimage.sh > /tmp/zimage.log 2>&1 &
```

The script handles:
- GPU health check (aborts if KIQ fence timeout detected — reboot needed)
- Unsetting old workaround variables
- Setting correct ROCm environment
- Starting the server

### Manual Start (if needed)

```bash
export HSA_OVERRIDE_GFX_VERSION=10.1.0
export HSA_ENABLE_SDMA=0
export HSA_TOOLS_LIB=""
export HIP_VISIBLE_DEVICES=0
export ROCM_PATH=/opt/rocm
export PATH="/opt/rocm/bin:$PATH"
export LD_LIBRARY_PATH="/opt/rocm/lib"

# CRITICAL: unset old workarounds
unset GPU_MAX_HW_QUEUES HIP_LAUNCH_BLOCKING
unset GGML_CUDA_ENABLE_UNIFIED_MEMORY GGML_HIP_HOST_ALLOC
unset GGML_CUDA_NO_PINNED GGML_HIP_NO_COARSE_GRAIN
unset HSA_DISABLE_FRAGMENT_ALLOCATOR

cd ~/stable-diffusion.cpp-restapi/build
./bin/sdcpp-restapi --config config.json
```

### Endpoints

| Endpoint | Description |
|----------|-------------|
| `http://localhost:8080/ui` | Web UI |
| `http://localhost:8080` | API root |
| `ws://localhost:8081` | WebSocket (live progress) |

---

## 6. Loading Models

After the server starts, you must load a model before generating images.

### Load Z-Image-Turbo (recommended command)

```bash
curl -s -X POST http://localhost:8080/models/load \
  -H "Content-Type: application/json" \
  -d '{
    "model_name": "z_image_turbo-Q5_K_S.gguf",
    "model_type": "diffusion",
    "vae": "ae.safetensors",
    "llm": "Qwen3-4B-Instruct-2507-Q5_K_S.gguf",
    "options": {
      "flash_attn": false
    }
  }'
```

**Expected response** (takes ~25 seconds):

```json
{
  "success": true,
  "message": "Model loaded successfully",
  "model_name": "z_image_turbo-Q5_K_S.gguf",
  "model_type": "diffusion",
  "loaded_components": {
    "vae": "ae.safetensors",
    "llm": "Qwen3-4B-Instruct-2507-Q5_K_S.gguf"
  }
}
```

### Model Load API Reference

**POST** `/models/load`

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `model_name` | string | **Yes** | Filename of the diffusion model (must be in `diffusion_models/` dir) |
| `model_type` | string | **Yes** | `"diffusion"` for SD models |
| `vae` | string | Yes | VAE filename (from `vae/` dir) |
| `llm` | string | Optional | LLM for prompt enhancement (from `llm/` dir) |
| `options` | object | Optional | Model options (see below) |

### Model Load Options

| Option | Type | Default | Recommended | Description |
|--------|------|---------|-------------|-------------|
| `flash_attn` | bool | `false` | **`false`** | Flash attention — gfx1010 does NOT support it |
| `keep_clip_on_cpu` | bool | `true` | `true` | CLIP runs fine on CPU, saves GPU memory |
| `keep_vae_on_cpu` | bool | `false` | `false` | VAE on GPU is faster |
| `offload_to_cpu` | bool | `false` | `false` | CPU offload hurts performance |
| `vae_decode_only` | bool | `true` | `true` | Only need decode for txt2img |
| `free_params_immediately` | bool | `false` | `false` | Frees memory faster but slower load |
| `vae_conv_direct` | bool | `false` | **`false`** | Direct convolutions — model load never finishes! |
| `diffusion_conv_direct` | bool | `false` | **`false`** | Direct convolutions — model load never finishes! |
| `n_threads` | int | `-1` (auto) | `-1` | CPU threads for compute |
| `enable_mmap` | bool | `true` | `true` | Memory-mapped file loading |

### Unload Model

```bash
curl -s -X POST http://localhost:8080/models/unload
```

---

## 7. Generating Images

### Via API (curl)

**Step 1: Submit a generation job**

```bash
curl -s -X POST http://localhost:8080/txt2img \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "blonde woman",
    "width": 512,
    "height": 1024,
    "steps": 8,
    "cfg_scale": 1,
    "sampler": "euler",
    "scheduler": "smoothstep",
    "seed": 42
  }'
```

Response:

```json
{
  "job_id": "e95872f8-15cb-48ce-a10f-19f2de430d9e",
  "position": 1,
  "status": "pending"
}
```

**Step 2: Poll for completion**

```bash
curl -s http://localhost:8080/queue/<job_id>
```

Response when done:

```json
{
  "status": "completed",
  "duration": "92.2s",
  "outputs": ["image_filename.png"]
}
```

### Via WebUI

Open `http://localhost:8080/ui` in a browser.

### Generation Parameters

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `prompt` | string | Required | Text description of the image |
| `negative_prompt` | string | `""` | What to avoid |
| `width` | int | 512 | Image width (pixels) |
| `height` | int | 512 | Image height (pixels) |
| `steps` | int | 20 | Sampling steps (8 is optimal for Z-Image-Turbo) |
| `cfg_scale` | float | 7.0 | Classifier-free guidance scale (1.0 for turbo models) |
| `sampler` | string | `"euler"` | Sampling method |
| `scheduler` | string | `"smoothstep"` | Noise schedule |
| `seed` | int | random | Seed for reproducibility (-1 = random) |

### Recommended Settings for Z-Image-Turbo

```json
{
  "steps": 8,
  "cfg_scale": 1,
  "sampler": "euler",
  "scheduler": "smoothstep"
}
```

Z-Image-Turbo is a distilled model — 8 steps is the sweet spot. More steps don't improve quality.

---

## 8. Performance Benchmarks

All benchmarks: 512×1024 image, 8 steps, Euler sampler, smoothstep scheduler, seed 42, prompt "blonde woman".

### Best Result: ~82s (measured warm run, 3rd consecutive generation)

### Systematic Benchmark Matrix

| Config | Text-Enc | Sampling | VAE Decode | Total | vs Baseline |
|--------|----------|----------|------------|-------|-------------|
| **Clean (cold, 1st gen)** | 0.54s | 77.4s | 13.9s | **92s** | Baseline |
| **Clean (warm, 2nd gen)** | — | — | — | **85s** | -8% |
| **Clean (warm, 3rd gen)** | — | — | — | **82s** | -11% |
| `GGML_CUDA_ENABLE_UNIFIED_MEMORY=1` | 0.52s | 82.7s | 26.1s | **109s** | +18% slower |
| `GGML_HIP_HOST_ALLOC=1` | 0.55s | 98.3s | 29.8s | **129s** | +40% slower |
| `conv_direct=true` (both) | — | — | — | **N/A** | Model load stalls (>13 min, aborted) |
| Vulkan backend (RADV) | — | — | — | **~79s** | Reference |

### Key Findings

1. **Clean config is best**: No GGML environment variables. The v3 kernel patches make all old workarounds unnecessary.

2. **Warm vs cold run**: First generation after model load takes ~92s. Subsequent runs improve to ~85s (2nd) → ~82s (3rd+) as GPU caches warm up.

3. **GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 hurts**: `hipMallocManaged` causes page fault overhead → +18% slower sampling, +88% slower VAE.

4. **GGML_HIP_HOST_ALLOC=1 worst**: Zero-copy over PCIe is terrible → +27% slower sampling, +114% slower VAE.

5. **conv_direct is broken**: Setting `vae_conv_direct=true` and/or `diffusion_conv_direct=true` causes the model load to hang indefinitely (13+ minutes, never completes). These bypass rocBLAS im2col+GEMM path for direct convolutions, but the compute graph build is too expensive for this model size (244 VAE f32 tensors + 180 diffusion q5_K tensors).

6. **ROCm vs Vulkan gap**: ROCm (82–92s) vs Vulkan (79s). ROCm warm runs are very close to Vulkan (~82s vs ~79s, only ~4% gap). The remaining overhead is ggml compute graph scheduling, not GPU compute itself (~10s actual GPU time vs ~67s framework overhead in sampling).

### Timing Breakdown

The 92s total breaks down as:

```
Text-Encoding (CLIP on CPU):        0.5s  (  0.5%)  — NOT a bottleneck
Sampling (8 steps diffusion):      77.4s  ( 84.1%)  — main bottleneck
  ├── Actual GPU compute:          ~10s   ( 10.9%)
  └── ggml framework overhead:     ~67s   ( 73.2%)  — graph scheduling, kernel launches
VAE Decode:                        13.9s  ( 15.1%)
```

The 67s ggml overhead is an upstream limitation of the stable-diffusion.cpp framework, not something fixable via configuration.

---

## 9. Tested Configurations (Full History)

All the environment variable combinations tested during optimization:

### Config 1: "All Workarounds" (pre-v3 era)

```bash
GPU_MAX_HW_QUEUES=1
HIP_LAUNCH_BLOCKING=1
GGML_CUDA_ENABLE_UNIFIED_MEMORY=1
GGML_HIP_HOST_ALLOC=1
GGML_CUDA_NO_PINNED=1
GGML_HIP_NO_COARSE_GRAIN=1
HSA_DISABLE_FRAGMENT_ALLOCATOR=1
```

**Result**: ~155s initially, ~82s with warm cache. These variables were needed before v3 kernel patches but are harmful now.

### Config 2: Clean (current optimal)

```bash
# NO GGML variables set at all
# Only ROCm basics:
HSA_OVERRIDE_GFX_VERSION=10.1.0
HSA_ENABLE_SDMA=0
HIP_VISIBLE_DEVICES=0
```

**Result**: 92s cold / ~82s warm (3rd gen). **This is the recommended config.**

### Config 3: UNIFIED_MEMORY only

```bash
GGML_CUDA_ENABLE_UNIFIED_MEMORY=1
```

**Result**: 109s (+18%). hipMallocManaged page faults add overhead.

### Config 4: HIP_HOST_ALLOC only

```bash
GGML_HIP_HOST_ALLOC=1
```

**Result**: 129s (+40%). Zero-copy over PCIe kills performance.

### Config 5: Conv Direct

```bash
# Model load options:
vae_conv_direct=true
diffusion_conv_direct=true
```

**Result**: Model load never completes (>13 minutes at 100% CPU, aborted). Not viable.

---

## 10. Troubleshooting

### Server won't start

```
[ERROR] GPU is in a broken state (KIQ fence timeout detected)
```

**Fix**: Reboot the system. The BC-250 GPU cannot recover from KIQ timeouts.

```bash
sudo reboot
```

### Model load fails

```json
{"error": "Main model not found: ''"}
```

**Fix**: Use `model_name` field (not `model` or `model_path`). Value must be the **filename only** (not full path). The server looks in the configured `diffusion_models/` directory.

### Generation hangs or crashes

- **Do NOT run** `rocm-smi`, `clinfo`, or any other GPU query tool while the server is running
- **Do NOT start** multiple GPU processes simultaneously
- If the server crashes, **reboot before restarting** — the GPU state may be corrupted

### Old environment variables leaking from .bashrc

If generation is unexpectedly slow (>100s), check for leftover variables:

```bash
env | grep -E "GGML_|GPU_MAX|HIP_LAUNCH"
```

If any of the old workaround vars are set, unset them:

```bash
unset GPU_MAX_HW_QUEUES HIP_LAUNCH_BLOCKING
unset GGML_CUDA_ENABLE_UNIFIED_MEMORY GGML_HIP_HOST_ALLOC
unset GGML_CUDA_NO_PINNED GGML_HIP_NO_COARSE_GRAIN
unset HSA_DISABLE_FRAGMENT_ALLOCATOR
```

### flash_attn errors

```
gfx1010 does not support flash attention
```

**Fix**: Always load with `"flash_attn": false`. The gfx1010 architecture does not have the required hardware. This is already the default in the patched code.

---

## 11. Quick Reference

### Start Server + Load Model + Generate Image (one-liner workflow)

```bash
# 1. Start server
nohup bash ~/start-zimage.sh > /tmp/zimage.log 2>&1 &
sleep 5

# 2. Load model
curl -s -X POST http://localhost:8080/models/load \
  -H "Content-Type: application/json" \
  -d '{
    "model_name": "z_image_turbo-Q5_K_S.gguf",
    "model_type": "diffusion",
    "vae": "ae.safetensors",
    "llm": "Qwen3-4B-Instruct-2507-Q5_K_S.gguf",
    "options": {"flash_attn": false}
  }'

# 3. Generate image
JOB=$(curl -s -X POST http://localhost:8080/txt2img \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "blonde woman",
    "width": 512, "height": 1024,
    "steps": 8, "cfg_scale": 1,
    "sampler": "euler", "scheduler": "smoothstep"
  }' | python3 -c "import sys,json; print(json.load(sys.stdin)['job_id'])")

echo "Job: $JOB"

# 4. Poll until done
while true; do
  STATUS=$(curl -s http://localhost:8080/queue/$JOB | python3 -c "import sys,json; d=json.load(sys.stdin); print(d.get('status','?'))")
  echo "Status: $STATUS"
  [ "$STATUS" = "completed" ] && break
  sleep 10
done

# 5. Check result
curl -s http://localhost:8080/queue/$JOB | python3 -m json.tool
```

### Check Server Logs

```bash
tail -f /tmp/zimage.log
```

### Check Timing Breakdown

```bash
grep -a "get_learned\|sampling completed\|decode_first\|generate_image completed" /tmp/zimage.log | tail -4
```

---

## 12. Important Warnings

1. **Do NOT run `rocm-smi`** or any GPU monitoring tool while the server is running — it can crash the GPU
2. **Reboot after any crash** — the BC-250 GPU state cannot be recovered without a full reboot
3. **Do NOT set old GGML environment variables** — they are not needed with v3 kernel patches and severely hurt performance
4. **Do NOT use `conv_direct`** — model load hangs indefinitely
5. **Do NOT use `flash_attn: true`** — gfx1010 does not support flash attention
6. **One GPU process at a time** — the BC-250 cannot handle concurrent GPU workloads