Uploaded sanitized BC250/ROCm Repository.
This commit is contained in:
Fabian
@@ -0,0 +1,637 @@
|
||||
# ComfyUI + Z-Image Turbo on AMD BC-250 — Complete Setup Guide
|
||||
|
||||
> **Hardware**: AMD BC-250 (Cyan Skillfish, gfx1013→gfx1010, 24 CUs, shared RAM)
|
||||
> **Backend**: ROCm 7.2.0 / PyTorch 2.5.1+rocm6.2
|
||||
> **OS**: CachyOS, kernel 6.18.8-3-cachyos
|
||||
> **ComfyUI Version**: 0.15.1
|
||||
> **Date**: 2026-03-02
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. [Overview](#1-overview)
|
||||
2. [Architecture](#2-architecture)
|
||||
3. [Prerequisites](#3-prerequisites)
|
||||
4. [Installation — Step by Step](#4-installation--step-by-step)
|
||||
5. [Model Setup](#5-model-setup)
|
||||
6. [Launch Script](#6-launch-script)
|
||||
7. [ComfyUI Workflow — Z-Image Turbo](#7-comfyui-workflow--z-image-turbo)
|
||||
8. [BC-250 Specific Tuning](#8-bc-250-specific-tuning)
|
||||
9. [Troubleshooting](#9-troubleshooting)
|
||||
10. [File Inventory](#10-file-inventory)
|
||||
11. [Performance Notes](#11-performance-notes)
|
||||
|
||||
---
|
||||
|
||||
## 1. Overview
|
||||
|
||||
ComfyUI is a node-based Stable Diffusion GUI that runs Z-Image Turbo (a Lumina2-architecture model) via PyTorch with ROCm/HIP on the AMD BC-250 GPU. The model uses a GGUF-quantized diffusion model (Q5_K_S) loaded via the ComfyUI-GGUF custom node, with a Gemma 2 2B text encoder and a Flux-compatible VAE.
|
||||
|
||||
### What's Running
|
||||
|
||||
| Component | File | Size | Format |
|
||||
|-----------|------|------|--------|
|
||||
| Diffusion Model | `z_image_turbo-Q5_K_S.gguf` | 5.2 GB | GGUF Q5_K_S |
|
||||
| Text Encoder | `gemma2_2b_lumina2.safetensors` | 9.8 GB | Safetensors (f32) |
|
||||
| VAE | `ae.safetensors` | 335 MB | Safetensors (f32) |
|
||||
|
||||
### Pipeline
|
||||
|
||||
```
|
||||
[ComfyUI WebUI :8188] → [PyTorch] → [ROCm/HIP] → [AMD BC-250 GPU]
|
||||
↓
|
||||
[Gemma 2 2B Text Encoder] → CLIP Encode → [Z-Image Turbo Diffusion] → [VAE Decode] → Image
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. Architecture
|
||||
|
||||
### Z-Image Turbo Details
|
||||
|
||||
- **Architecture**: Lumina2 (Lumina-Image 2.0 family)
|
||||
- **Base**: Z-Image by Freepik, turbo-distilled variant
|
||||
- **Text Encoder**: Gemma 2 2B (Google, 2304-dim embeddings)
|
||||
- **VAE**: Flux-compatible autoencoder (`ae.safetensors`)
|
||||
- **Sampler**: Euler with SGM Uniform scheduler, 8 steps (turbo)
|
||||
- **CFG Scale**: 3.0 (turbo models use low CFG)
|
||||
- **Latent Format**: Flux-style latent space
|
||||
|
||||
### Why GGUF?
|
||||
|
||||
The BC-250 has ~14.7 GB shared system RAM. The full FP16 diffusion model would be too large. GGUF Q5_K_S quantization reduces the model from ~12+ GB to 5.2 GB, making it feasible alongside the text encoder and VAE.
|
||||
|
||||
---
|
||||
|
||||
## 3. Prerequisites
|
||||
|
||||
Before starting, you need ROCm working on the BC-250. See `ROCm_BC250_Documentation.md` for the full ROCm setup.
|
||||
|
||||
### Required
|
||||
|
||||
- ROCm 7.2.0 installed and working (`rocminfo` detects BC-250)
|
||||
- Python 3.11 (`/usr/bin/python3.11`)
|
||||
- Git
|
||||
- ~30 GB free disk space
|
||||
|
||||
### Verify ROCm
|
||||
|
||||
```bash
|
||||
rocminfo | grep "Name:"
|
||||
# Should show: gfx1010 and AMD BC-250
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. Installation — Step by Step
|
||||
|
||||
### 4.1 Clone ComfyUI
|
||||
|
||||
```bash
|
||||
cd ~
|
||||
git clone https://github.com/comfyanonymous/ComfyUI.git
|
||||
cd ComfyUI
|
||||
```
|
||||
|
||||
### 4.2 Create Python 3.11 Virtual Environment
|
||||
|
||||
Python 3.11 is required — Python 3.14 (system default) is too new for PyTorch ROCm wheels.
|
||||
|
||||
```bash
|
||||
python3.11 -m venv venv
|
||||
source venv/bin/activate
|
||||
```
|
||||
|
||||
### 4.3 Install PyTorch with ROCm Support
|
||||
|
||||
```bash
|
||||
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/rocm6.2
|
||||
```
|
||||
|
||||
This downloads ~4 GB. The ROCm 6.2 PyTorch wheel is compatible with the ROCm 7.2 runtime.
|
||||
|
||||
**Verify installation:**
|
||||
```bash
|
||||
python -c "import torch; print(torch.version.cuda); print(torch.cuda.is_available())"
|
||||
# Should print: 6.2 and True
|
||||
```
|
||||
|
||||
### 4.4 Install ComfyUI Dependencies
|
||||
|
||||
```bash
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
### 4.5 Install ComfyUI-GGUF Custom Node
|
||||
|
||||
This enables loading GGUF-quantized models in ComfyUI.
|
||||
|
||||
```bash
|
||||
cd ~/ComfyUI/custom_nodes
|
||||
git clone https://github.com/city96/ComfyUI-GGUF.git
|
||||
source ~/ComfyUI/venv/bin/activate
|
||||
pip install gguf
|
||||
```
|
||||
|
||||
### 4.6 Install huggingface-hub (for model downloads)
|
||||
|
||||
```bash
|
||||
pip install huggingface-hub
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Model Setup
|
||||
|
||||
### 5.1 Directory Structure
|
||||
|
||||
ComfyUI looks for models in `~/ComfyUI/models/`. Our models live in `~/sd-models/` and are symlinked.
|
||||
|
||||
```
|
||||
~/ComfyUI/models/
|
||||
├── unet/
|
||||
│ └── z_image_turbo-Q5_K_S.gguf → ~/sd-models/diffusion_models/z_image_turbo-Q5_K_S.gguf
|
||||
├── text_encoders/
|
||||
│ └── gemma2_2b_lumina2.safetensors (merged from 3 shards, 9.8 GB)
|
||||
├── vae/
|
||||
│ └── ae.safetensors → ~/sd-models/vae/ae.safetensors
|
||||
└── ...
|
||||
```
|
||||
|
||||
### 5.2 Symlink Diffusion Model (GGUF)
|
||||
|
||||
The Z-Image Turbo GGUF model must go in `models/unet/` (ComfyUI-GGUF's `UnetLoaderGGUF` node reads from there):
|
||||
|
||||
```bash
|
||||
ln -sf /home/dars/sd-models/diffusion_models/z_image_turbo-Q5_K_S.gguf \
|
||||
~/ComfyUI/models/unet/z_image_turbo-Q5_K_S.gguf
|
||||
```
|
||||
|
||||
### 5.3 Text Encoder — Gemma 2 2B
|
||||
|
||||
Z-Image Turbo uses the Gemma 2 2B text encoder from the Lumina-Image-2.0 family. The original model is sharded into 3 safetensors files. We merge them into a single file for ComfyUI.
|
||||
|
||||
**Download from Alpha-VLLM (not gated, no login required):**
|
||||
|
||||
```bash
|
||||
source ~/ComfyUI/venv/bin/activate
|
||||
python3 -c "
|
||||
from huggingface_hub import hf_hub_download
|
||||
import os
|
||||
|
||||
repo = 'Alpha-VLLM/Lumina-Image-2.0'
|
||||
dest = os.path.expanduser('~/sd-models/text_encoders/lumina2_gemma2_2b')
|
||||
os.makedirs(dest, exist_ok=True)
|
||||
|
||||
files = [
|
||||
'text_encoder/config.json',
|
||||
'text_encoder/model.safetensors.index.json',
|
||||
'text_encoder/model-00001-of-00003.safetensors',
|
||||
'text_encoder/model-00002-of-00003.safetensors',
|
||||
'text_encoder/model-00003-of-00003.safetensors',
|
||||
]
|
||||
for f in files:
|
||||
print(f'Downloading {f}...')
|
||||
hf_hub_download(repo, f, local_dir=dest)
|
||||
print('Done!')
|
||||
"
|
||||
```
|
||||
|
||||
**Merge shards into single file:**
|
||||
|
||||
```bash
|
||||
source ~/ComfyUI/venv/bin/activate
|
||||
python3 << 'EOF'
|
||||
import safetensors.torch
|
||||
import torch
|
||||
import os, json
|
||||
|
||||
base_dir = os.path.expanduser("~/sd-models/text_encoders/lumina2_gemma2_2b/text_encoder")
|
||||
output = os.path.expanduser("~/ComfyUI/models/text_encoders/gemma2_2b_lumina2.safetensors")
|
||||
os.makedirs(os.path.dirname(output), exist_ok=True)
|
||||
|
||||
with open(os.path.join(base_dir, "model.safetensors.index.json")) as f:
|
||||
index = json.load(f)
|
||||
|
||||
all_tensors = {}
|
||||
shards = set(index["weight_map"].values())
|
||||
print(f"Loading {len(shards)} shards with {len(index['weight_map'])} tensors...")
|
||||
for shard in sorted(shards):
|
||||
path = os.path.join(base_dir, shard)
|
||||
print(f" Loading {shard}...")
|
||||
tensors = safetensors.torch.load_file(path, device="cpu")
|
||||
all_tensors.update(tensors)
|
||||
|
||||
print(f"Total tensors: {len(all_tensors)}")
|
||||
print(f"Saving merged file...")
|
||||
safetensors.torch.save_file(all_tensors, output)
|
||||
print(f"Done! Size: {os.path.getsize(output)/1e9:.2f} GB")
|
||||
EOF
|
||||
```
|
||||
|
||||
**Clean up shards (optional):**
|
||||
```bash
|
||||
rm -rf ~/sd-models/text_encoders/lumina2_gemma2_2b/
|
||||
```
|
||||
|
||||
### 5.4 VAE
|
||||
|
||||
```bash
|
||||
ln -sf /home/dars/sd-models/vae/ae.safetensors \
|
||||
~/ComfyUI/models/vae/ae.safetensors
|
||||
```
|
||||
|
||||
### 5.5 Verify All Models in Place
|
||||
|
||||
```bash
|
||||
ls -lh ~/ComfyUI/models/unet/*.gguf \
|
||||
~/ComfyUI/models/text_encoders/*.safetensors \
|
||||
~/ComfyUI/models/vae/*.safetensors
|
||||
```
|
||||
|
||||
Expected output:
|
||||
```
|
||||
9.8G ~/ComfyUI/models/text_encoders/gemma2_2b_lumina2.safetensors
|
||||
5.2G ~/ComfyUI/models/unet/z_image_turbo-Q5_K_S.gguf (symlink)
|
||||
335M ~/ComfyUI/models/vae/ae.safetensors (symlink)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. Launch Script
|
||||
|
||||
### Location: `~/start-comfyui.sh`
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# =============================================================
|
||||
# ComfyUI Launch Script for AMD BC-250 (ROCm / gfx1013)
|
||||
# =============================================================
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
echo "=========================================="
|
||||
echo " ComfyUI — BC-250 ROCm Launcher"
|
||||
echo "=========================================="
|
||||
|
||||
# --- GPU Health Check ---
|
||||
if dmesg 2>/dev/null | tail -50 | grep -qi "KIQ fence timeout"; then
|
||||
echo "[ABORT] KIQ fence timeout detected in dmesg — reboot required!"
|
||||
exit 1
|
||||
fi
|
||||
echo "[OK] GPU health check passed"
|
||||
|
||||
# --- ROCm Environment for BC-250 (gfx1013 → gfx1010 spoof) ---
|
||||
export HSA_OVERRIDE_GFX_VERSION=10.1.0
|
||||
export HSA_ENABLE_SDMA=0
|
||||
export HIP_VISIBLE_DEVICES=0
|
||||
export ROCM_PATH=/opt/rocm
|
||||
export HSA_TOOLS_LIB=""
|
||||
export HSA_TOOLS_REPORT_LOAD_FAILURE=0
|
||||
export PATH="/opt/rocm/bin:$PATH"
|
||||
export LD_LIBRARY_PATH="/opt/rocm/lib"
|
||||
|
||||
# --- Unset old workaround variables that destroy performance ---
|
||||
unset GPU_MAX_HW_QUEUES 2>/dev/null || true
|
||||
unset HIP_LAUNCH_BLOCKING 2>/dev/null || true
|
||||
unset GGML_CUDA_ENABLE_UNIFIED_MEMORY 2>/dev/null || true
|
||||
unset GGML_HIP_HOST_ALLOC 2>/dev/null || true
|
||||
unset GGML_CUDA_NO_PINNED 2>/dev/null || true
|
||||
unset GGML_HIP_NO_COARSE_GRAIN 2>/dev/null || true
|
||||
unset HSA_DISABLE_FRAGMENT_ALLOCATOR 2>/dev/null || true
|
||||
|
||||
# --- PyTorch ROCm tuning ---
|
||||
export PYTORCH_HIP_ALLOC_CONF="expandable_segments:False"
|
||||
|
||||
echo "[OK] ROCm environment configured"
|
||||
|
||||
# --- Activate venv ---
|
||||
cd ~/ComfyUI
|
||||
source venv/bin/activate
|
||||
|
||||
# --- Launch ComfyUI ---
|
||||
echo "[START] Launching ComfyUI on http://0.0.0.0:8188"
|
||||
echo "=========================================="
|
||||
python main.py \
|
||||
--listen 0.0.0.0 \
|
||||
--port 8188 \
|
||||
--force-fp32 \
|
||||
--lowvram \
|
||||
"$@"
|
||||
```
|
||||
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
# Foreground (see logs):
|
||||
bash ~/start-comfyui.sh
|
||||
|
||||
# Background with logging:
|
||||
nohup bash ~/start-comfyui.sh > /tmp/comfyui.log 2>&1 &
|
||||
|
||||
# Check if running:
|
||||
curl -s http://localhost:8188/system_stats | python3 -m json.tool
|
||||
```
|
||||
|
||||
### CLI Flags Explained
|
||||
|
||||
| Flag | Why |
|
||||
|------|-----|
|
||||
| `--listen 0.0.0.0` | Accept connections from any interface (access from other machines) |
|
||||
| `--port 8188` | Default ComfyUI port |
|
||||
| `--force-fp32` | BC-250 gfx1010 has limited FP16 support in PyTorch ROCm; FP32 prevents crashes |
|
||||
| `--lowvram` | Enables aggressive model offloading — essential for 14.7 GB shared RAM |
|
||||
|
||||
---
|
||||
|
||||
## 7. ComfyUI Workflow — Z-Image Turbo
|
||||
|
||||
### Access the WebUI
|
||||
|
||||
Open in browser: **http://localhost:8188** (or `http://<machine-ip>:8188` from another machine)
|
||||
|
||||
### Pre-made Workflow
|
||||
|
||||
A ready-to-use workflow is saved at:
|
||||
```
|
||||
~/ComfyUI/workflows/z_image_turbo_bc250.json
|
||||
```
|
||||
|
||||
Load it via: **Menu → Load → select `z_image_turbo_bc250.json`**
|
||||
|
||||
### Manual Node Setup
|
||||
|
||||
If building the workflow from scratch, create these nodes:
|
||||
|
||||
#### Node 1: UnetLoaderGGUF
|
||||
- **Type**: `UnetLoaderGGUF` (from ComfyUI-GGUF custom node, category: bootleg)
|
||||
- **unet_name**: `z_image_turbo-Q5_K_S.gguf`
|
||||
- **Output**: MODEL → connect to KSampler's "model" input
|
||||
|
||||
#### Node 2: CLIPLoader
|
||||
- **Type**: `CLIPLoader` (built-in, category: advanced/loaders)
|
||||
- **clip_name**: `gemma2_2b_lumina2.safetensors`
|
||||
- **type**: `lumina2` ← **CRITICAL: must be set to lumina2**
|
||||
- **Output**: CLIP → connect to both CLIP Text Encode nodes
|
||||
|
||||
#### Node 3: CLIP Text Encode (Positive)
|
||||
- **Type**: `CLIPTextEncode`
|
||||
- **text**: Your prompt (e.g., "a beautiful sunset over the ocean")
|
||||
- **Input**: clip ← from CLIPLoader
|
||||
- **Output**: CONDITIONING → connect to KSampler's "positive" input
|
||||
|
||||
#### Node 4: CLIP Text Encode (Negative)
|
||||
- **Type**: `CLIPTextEncode`
|
||||
- **text**: Empty string `""` (turbo models work best with empty negative)
|
||||
- **Input**: clip ← from CLIPLoader
|
||||
- **Output**: CONDITIONING → connect to KSampler's "negative" input
|
||||
|
||||
#### Node 5: Empty Latent Image
|
||||
- **Type**: `EmptyLatentImage`
|
||||
- **width**: `512`
|
||||
- **height**: `512`
|
||||
- **batch_size**: `1`
|
||||
- **Output**: LATENT → connect to KSampler's "latent_image" input
|
||||
|
||||
#### Node 6: KSampler
|
||||
- **Type**: `KSampler`
|
||||
- **seed**: Any number (42)
|
||||
- **control_after_generate**: `fixed` (or `randomize` for variety)
|
||||
- **steps**: `8` (turbo — more steps won't improve quality)
|
||||
- **cfg**: `3.0` (turbo models use low CFG guidance)
|
||||
- **sampler_name**: `euler`
|
||||
- **scheduler**: `sgm_uniform`
|
||||
- **denoise**: `1.0`
|
||||
- **Inputs**: model, positive, negative, latent_image
|
||||
- **Output**: LATENT → connect to VAEDecode
|
||||
|
||||
#### Node 7: VAELoader
|
||||
- **Type**: `VAELoader`
|
||||
- **vae_name**: `ae.safetensors`
|
||||
- **Output**: VAE → connect to VAEDecode's "vae" input
|
||||
|
||||
#### Node 8: VAE Decode
|
||||
- **Type**: `VAEDecode`
|
||||
- **Inputs**: samples (from KSampler), vae (from VAELoader)
|
||||
- **Output**: IMAGE → connect to SaveImage
|
||||
|
||||
#### Node 9: Save Image
|
||||
- **Type**: `SaveImage`
|
||||
- **filename_prefix**: `ComfyUI`
|
||||
- **Input**: images ← from VAEDecode
|
||||
- Output images saved to: `~/ComfyUI/output/`
|
||||
|
||||
### Wiring Summary
|
||||
|
||||
```
|
||||
UnetLoaderGGUF ───MODEL──→ KSampler
|
||||
CLIPLoader ───CLIP──→ CLIPTextEncode (positive) ──CONDITIONING──→ KSampler
|
||||
CLIPLoader ───CLIP──→ CLIPTextEncode (negative) ──CONDITIONING──→ KSampler
|
||||
EmptyLatentImage ──LATENT──→ KSampler
|
||||
KSampler ──LATENT──→ VAEDecode
|
||||
VAELoader ──VAE──→ VAEDecode
|
||||
VAEDecode ──IMAGE──→ SaveImage
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 8. BC-250 Specific Tuning
|
||||
|
||||
### Environment Variables (set in launch script)
|
||||
|
||||
| Variable | Value | Why |
|
||||
|----------|-------|-----|
|
||||
| `HSA_OVERRIDE_GFX_VERSION` | `10.1.0` | BC-250 (gfx1013) needs gfx1010 spoof for ROCm |
|
||||
| `HSA_ENABLE_SDMA` | `0` | SDMA engine has hardware bugs on gfx1013 |
|
||||
| `HIP_VISIBLE_DEVICES` | `0` | Select the BC-250 GPU |
|
||||
| `ROCM_PATH` | `/opt/rocm` | ROCm installation path |
|
||||
| `HSA_TOOLS_LIB` | `""` | Disable profiling tools (stability) |
|
||||
| `HSA_TOOLS_REPORT_LOAD_FAILURE` | `0` | Suppress tool warnings |
|
||||
| `PYTORCH_HIP_ALLOC_CONF` | `expandable_segments:False` | Prevent memory fragmentation |
|
||||
|
||||
### Variables to NEVER Set
|
||||
|
||||
These old workarounds **destroy performance** and must NOT be set:
|
||||
|
||||
| Variable | Why it's bad |
|
||||
|----------|-------------|
|
||||
| `GPU_MAX_HW_QUEUES=1` | Serializes all GPU ops to 1 queue |
|
||||
| `HIP_LAUNCH_BLOCKING=1` | Forces synchronous kernel launches |
|
||||
| `GGML_CUDA_ENABLE_UNIFIED_MEMORY=1` | Page fault overhead |
|
||||
| `GGML_HIP_HOST_ALLOC=1` | Zero-copy over PCIe is slow |
|
||||
|
||||
### Memory Considerations
|
||||
|
||||
- Total available: ~14.7 GB shared system RAM
|
||||
- Diffusion model (GGUF Q5_K_S): ~5.2 GB
|
||||
- Text encoder (Gemma 2 2B f32): ~9.8 GB
|
||||
- VAE: ~335 MB
|
||||
- Total model footprint: ~15.3 GB — exceeds available RAM
|
||||
- **`--lowvram` is essential**: it offloads models to CPU when not in active use
|
||||
- Only one component is on GPU at a time during inference
|
||||
|
||||
### Resolution Recommendations
|
||||
|
||||
| Resolution | Latent Size | Notes |
|
||||
|-----------|-------------|-------|
|
||||
| 512×512 | 64×64 | Fastest, recommended for testing |
|
||||
| 768×768 | 96×96 | Good quality, slower |
|
||||
| 1024×1024 | 128×128 | May OOM on BC-250 |
|
||||
|
||||
---
|
||||
|
||||
## 9. Troubleshooting
|
||||
|
||||
### "KIQ fence timeout" in dmesg → Reboot
|
||||
|
||||
```bash
|
||||
sudo dmesg | grep -i "KIQ fence timeout"
|
||||
```
|
||||
If this appears, the GPU is in a bad state. **Reboot the machine.**
|
||||
|
||||
### ComfyUI won't start — "No module named torch"
|
||||
|
||||
Make sure you activated the venv:
|
||||
```bash
|
||||
source ~/ComfyUI/venv/bin/activate
|
||||
python -c "import torch; print(torch.__version__)"
|
||||
```
|
||||
|
||||
### "CLIP type not found" or wrong model type
|
||||
|
||||
Make sure the CLIPLoader node type is set to **`lumina2`** — NOT `stable_diffusion`.
|
||||
|
||||
### OOM (Out of Memory) during generation
|
||||
|
||||
1. Reduce resolution to 512×512
|
||||
2. Ensure `--lowvram` is set
|
||||
3. Close other programs using RAM
|
||||
4. Try `--use-split-cross-attention` flag
|
||||
|
||||
### Model not showing in dropdown
|
||||
|
||||
Verify symlinks are not broken:
|
||||
```bash
|
||||
ls -la ~/ComfyUI/models/unet/z_image_turbo-Q5_K_S.gguf
|
||||
ls -la ~/ComfyUI/models/vae/ae.safetensors
|
||||
ls -la ~/ComfyUI/models/text_encoders/gemma2_2b_lumina2.safetensors
|
||||
```
|
||||
|
||||
### "UnetLoaderGGUF" node not found
|
||||
|
||||
Ensure ComfyUI-GGUF is installed:
|
||||
```bash
|
||||
ls ~/ComfyUI/custom_nodes/ComfyUI-GGUF/
|
||||
pip list | grep gguf
|
||||
```
|
||||
|
||||
### PyTorch ROCm version mismatch
|
||||
|
||||
```bash
|
||||
python -c "import torch; print(torch.version.cuda)"
|
||||
# Should print: 6.2
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 10. File Inventory
|
||||
|
||||
### Installation Files
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `~/ComfyUI/` | ComfyUI installation directory |
|
||||
| `~/ComfyUI/venv/` | Python 3.11 virtual environment |
|
||||
| `~/ComfyUI/custom_nodes/ComfyUI-GGUF/` | GGUF model loader custom node |
|
||||
| `~/ComfyUI/workflows/z_image_turbo_bc250.json` | Pre-made Z-Image Turbo workflow |
|
||||
| `~/start-comfyui.sh` | Launch script with ROCm env vars |
|
||||
|
||||
### Model Files
|
||||
|
||||
| File | Size | Format |
|
||||
|------|------|--------|
|
||||
| `~/ComfyUI/models/unet/z_image_turbo-Q5_K_S.gguf` | 5.2 GB | Symlink → `~/sd-models/diffusion_models/` |
|
||||
| `~/ComfyUI/models/text_encoders/gemma2_2b_lumina2.safetensors` | 9.8 GB | Merged from Alpha-VLLM/Lumina-Image-2.0 |
|
||||
| `~/ComfyUI/models/vae/ae.safetensors` | 335 MB | Symlink → `~/sd-models/vae/` |
|
||||
|
||||
### Output
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `~/ComfyUI/output/` | Generated images saved here |
|
||||
|
||||
### Python Packages (key ones)
|
||||
|
||||
| Package | Version |
|
||||
|---------|---------|
|
||||
| torch | 2.5.1+rocm6.2 |
|
||||
| torchvision | 0.20.1+rocm6.2 |
|
||||
| torchaudio | 2.5.1+rocm6.2 |
|
||||
| pytorch-triton-rocm | 3.1.0 |
|
||||
| transformers | 5.2.0 |
|
||||
| safetensors | 0.7.0 |
|
||||
| gguf | 0.18.0 |
|
||||
| comfyui-frontend-package | 1.39.19 |
|
||||
|
||||
---
|
||||
|
||||
## 11. Performance Notes
|
||||
|
||||
### Startup Output (successful launch)
|
||||
|
||||
```
|
||||
Total VRAM 14750 MB, total RAM 15205 MB
|
||||
pytorch version: 2.5.1+rocm6.2
|
||||
AMD arch: gfx1010
|
||||
ROCm version: (6, 2)
|
||||
Forcing FP32
|
||||
Set vram state to: LOW_VRAM
|
||||
Device: cuda:0 AMD Radeon Graphics : native
|
||||
ComfyUI version: 0.15.1
|
||||
ComfyUI-GGUF: Partial torch compile only, consider updating pytorch
|
||||
```
|
||||
|
||||
### Expected Timing (BC-250, 512×512, 8 steps)
|
||||
|
||||
| Phase | Estimated Time |
|
||||
|-------|---------------|
|
||||
| Model Loading (first run) | 30-60s |
|
||||
| Text Encoding (Gemma 2 2B) | ~2-5s |
|
||||
| Sampling (8 steps, Euler) | ~60-90s |
|
||||
| VAE Decode | ~10-15s |
|
||||
| **Total (first image)** | **~2-3 min** |
|
||||
| **Total (subsequent)** | **~1-2 min** |
|
||||
|
||||
### Comparison with sdcpp-restapi
|
||||
|
||||
| | ComfyUI + PyTorch | sdcpp-restapi |
|
||||
|---|---|---|
|
||||
| Frontend | Full node-based GUI | REST API + simple WebUI |
|
||||
| Model format | GGUF + safetensors | GGUF only |
|
||||
| Memory management | PyTorch (--lowvram) | ggml manual |
|
||||
| Flexibility | Full workflow customization | Fixed pipeline |
|
||||
| Turbo steps | Configurable per-run | Config-based |
|
||||
|
||||
---
|
||||
|
||||
## Appendix: Quick Start Cheatsheet
|
||||
|
||||
```bash
|
||||
# 1. Launch ComfyUI
|
||||
bash ~/start-comfyui.sh
|
||||
|
||||
# 2. Open browser
|
||||
# http://localhost:8188
|
||||
|
||||
# 3. Load workflow
|
||||
# Menu → Load → z_image_turbo_bc250.json
|
||||
|
||||
# 4. Click "Queue Prompt" to generate
|
||||
|
||||
# 5. Images saved in ~/ComfyUI/output/
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
*Document generated: 2026-03-02 | System: CachyOS + AMD BC-250 + ROCm 7.2.0*
|
||||
@@ -0,0 +1,443 @@
|
||||
# Qwen3-4B on llama.cpp with ROCm (AMD BC-250)
|
||||
|
||||
> Running **Qwen3-4B-Q8_0** via **llama.cpp** with **ROCm HIP** GPU acceleration on the **AMD BC-250 (gfx1010)**.
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
|
||||
- [System Overview](#system-overview)
|
||||
- [Prerequisites](#prerequisites)
|
||||
- [Step 1: Clone llama.cpp](#step-1-clone-llamacpp)
|
||||
- [Step 2: Build llama.cpp with ROCm HIP](#step-2-build-llamacpp-with-rocm-hip)
|
||||
- [Step 3: Download the Model](#step-3-download-the-model)
|
||||
- [Step 4: Run Inference (Interactive Chat)](#step-4-run-inference-interactive-chat)
|
||||
- [Step 5: Run Inference (One-Shot / Batch)](#step-5-run-inference-one-shot--batch)
|
||||
- [Step 6: Run as API Server](#step-6-run-as-api-server)
|
||||
- [Performance Results](#performance-results)
|
||||
- [VRAM / Memory Breakdown](#vram--memory-breakdown)
|
||||
- [Useful Parameters Reference](#useful-parameters-reference)
|
||||
- [Troubleshooting](#troubleshooting)
|
||||
- [Notes & Tips](#notes--tips)
|
||||
|
||||
---
|
||||
|
||||
## System Overview
|
||||
|
||||
| Component | Value |
|
||||
|------------------|-----------------------------------------------|
|
||||
| **GPU** | AMD BC-250 (Navi 10, gfx1010) |
|
||||
| **VRAM** | ~14.4 GiB (14750 MiB) |
|
||||
| **Wave Size** | 32 |
|
||||
| **ROCm Version** | 7.2.0 |
|
||||
| **HIP Version** | 7.2.26043-9999 |
|
||||
| **HIP Compiler** | AMD clang 22.0.0git (ROCm LLVM) |
|
||||
| **OS** | CachyOS (Arch-based), Kernel 6.18.8-3-cachyos |
|
||||
| **CPU** | 12 threads |
|
||||
| **RAM** | 14 GiB system + 14 GiB swap |
|
||||
| **llama.cpp** | Build b8184 (commit `3191462`) |
|
||||
| **CMake** | 4.2.3 |
|
||||
|
||||
---
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Before starting, ensure you have:
|
||||
|
||||
1. **ROCm installed and working** — verify with:
|
||||
```bash
|
||||
rocm-smi
|
||||
rocminfo | grep -E "Name:|gfx"
|
||||
```
|
||||
|
||||
2. **Required packages**:
|
||||
```bash
|
||||
# Arch/CachyOS
|
||||
sudo pacman -S git cmake base-devel aria2
|
||||
|
||||
# Ubuntu/Debian
|
||||
sudo apt install git cmake build-essential aria2
|
||||
```
|
||||
|
||||
3. **ROCm development libraries** (hipblas, rocblas):
|
||||
```bash
|
||||
# Verify they exist
|
||||
ls /opt/rocm/lib/libhipblas.so
|
||||
ls /opt/rocm/lib/librocblas.so
|
||||
ls /opt/rocm/lib/llvm/bin/clang++
|
||||
```
|
||||
|
||||
4. **Know your GPU architecture**:
|
||||
```bash
|
||||
rocminfo | grep "Name:" | grep gfx
|
||||
# Output: gfx1010 (for BC-250)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 1: Clone llama.cpp
|
||||
|
||||
```bash
|
||||
cd ~
|
||||
git clone https://github.com/ggml-org/llama.cpp.git
|
||||
cd llama.cpp
|
||||
```
|
||||
|
||||
If already cloned, update:
|
||||
```bash
|
||||
cd ~/llama.cpp
|
||||
git stash # if you have local changes
|
||||
git pull
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 2: Build llama.cpp with ROCm HIP
|
||||
|
||||
### Configure
|
||||
|
||||
```bash
|
||||
cd ~/llama.cpp
|
||||
rm -rf build
|
||||
mkdir build && cd build
|
||||
|
||||
cmake .. \
|
||||
-DGGML_HIP=ON \
|
||||
-DAMDGPU_TARGETS="gfx1010" \
|
||||
-DCMAKE_BUILD_TYPE=Release \
|
||||
-DCMAKE_HIP_COMPILER=/opt/rocm/lib/llvm/bin/clang++ \
|
||||
-G "Unix Makefiles"
|
||||
```
|
||||
|
||||
**Key flags explained:**
|
||||
|
||||
| Flag | Purpose |
|
||||
|------|---------|
|
||||
| `-DGGML_HIP=ON` | Enable HIP/ROCm GPU backend |
|
||||
| `-DAMDGPU_TARGETS="gfx1010"` | Target GPU architecture (BC-250 = gfx1010) |
|
||||
| `-DCMAKE_HIP_COMPILER=/opt/rocm/lib/llvm/bin/clang++` | Use ROCm's clang directly (required for CMake ≥ 4.x, `hipcc` wrapper is rejected) |
|
||||
| `-G "Unix Makefiles"` | Use Make instead of Ninja |
|
||||
|
||||
> **Important (CMake 4.x):** Do NOT use `-DCMAKE_HIP_COMPILER=/opt/rocm/bin/hipcc` — CMake 4.x explicitly rejects the hipcc wrapper. You must point to the clang++ binary inside ROCm's LLVM directory.
|
||||
|
||||
### Verify Configuration
|
||||
|
||||
```bash
|
||||
grep "GGML_HIP" CMakeCache.txt
|
||||
# Should show: GGML_HIP:BOOL=ON
|
||||
```
|
||||
|
||||
### Build
|
||||
|
||||
```bash
|
||||
make -j$(nproc)
|
||||
```
|
||||
|
||||
> **Build time:** HIP compilation is slow (~15-25 minutes on 12 threads). Each `.cu` template gets compiled to AMDGPU ISA for gfx1010. Be patient.
|
||||
|
||||
### Verify Build Output
|
||||
|
||||
```bash
|
||||
ls -lh build/bin/llama-cli build/bin/llama-server
|
||||
|
||||
# Verify HIP linkage
|
||||
ldd build/bin/llama-cli | grep -i "hip\|rocm"
|
||||
```
|
||||
|
||||
Expected output:
|
||||
```
|
||||
libggml-hip.so.0 => .../libggml-hip.so.0
|
||||
libhipblas.so.3 => /opt/rocm/lib/libhipblas.so.3
|
||||
librocblas.so.5 => /opt/rocm/lib/librocblas.so.5
|
||||
libamdhip64.so.7 => /opt/rocm/lib/libamdhip64.so.7
|
||||
librocsolver.so.0 => /opt/rocm/lib/librocsolver.so.0
|
||||
libhsa-runtime64.so.1 => /opt/rocm/lib/libhsa-runtime64.so.1
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 3: Download the Model
|
||||
|
||||
### Using aria2 (Recommended — Maximum Speed)
|
||||
|
||||
```bash
|
||||
mkdir -p ~/models
|
||||
|
||||
aria2c \
|
||||
-x 16 \
|
||||
-s 16 \
|
||||
-k 1M \
|
||||
-d ~/models \
|
||||
-o Qwen3-4B-Q8_0.gguf \
|
||||
"https://huggingface.co/Qwen/Qwen3-4B-GGUF/resolve/main/Qwen3-4B-Q8_0.gguf"
|
||||
```
|
||||
|
||||
| aria2 Flag | Purpose |
|
||||
|------------|---------|
|
||||
| `-x 16` | 16 connections per server |
|
||||
| `-s 16` | Split into 16 segments |
|
||||
| `-k 1M` | Minimum split size 1MB |
|
||||
|
||||
### Using wget (Fallback)
|
||||
|
||||
```bash
|
||||
wget -c \
|
||||
"https://huggingface.co/Qwen/Qwen3-4B-GGUF/resolve/main/Qwen3-4B-Q8_0.gguf" \
|
||||
-O ~/models/Qwen3-4B-Q8_0.gguf
|
||||
```
|
||||
|
||||
### Verify Download
|
||||
|
||||
```bash
|
||||
ls -lh ~/models/Qwen3-4B-Q8_0.gguf
|
||||
# Expected: ~4.0 GiB (4,280,404,704 bytes)
|
||||
```
|
||||
|
||||
> **Model Source:** [Qwen/Qwen3-4B-GGUF](https://huggingface.co/Qwen/Qwen3-4B-GGUF) on Hugging Face (official Qwen repo).
|
||||
|
||||
---
|
||||
|
||||
## Step 4: Run Inference (Interactive Chat)
|
||||
|
||||
```bash
|
||||
cd ~/llama.cpp/build/bin
|
||||
|
||||
./llama-cli \
|
||||
-m ~/models/Qwen3-4B-Q8_0.gguf \
|
||||
-ngl 99 \
|
||||
-c 4096 \
|
||||
--temp 0.6 \
|
||||
--top-k 20 \
|
||||
--top-p 0.95
|
||||
```
|
||||
|
||||
**Expected startup output:**
|
||||
```
|
||||
ggml_cuda_init: found 1 ROCm devices:
|
||||
Device 0: AMD BC-250, gfx1010:xnack- (0x1010), VMM: no, Wave Size: 32
|
||||
|
||||
build : b213-3191462
|
||||
model : Qwen3-4B-Q8_0.gguf
|
||||
modalities : text
|
||||
```
|
||||
|
||||
You'll get an interactive `>` prompt. Type your question and press Enter.
|
||||
|
||||
**In-chat commands:**
|
||||
| Command | Action |
|
||||
|-------------|---------------------------------|
|
||||
| `/exit` | Exit the chat |
|
||||
| `/clear` | Clear chat history |
|
||||
| `/regen` | Regenerate last response |
|
||||
| `/read` | Load a text file into context |
|
||||
| `Ctrl+C` | Force exit |
|
||||
|
||||
### Disable Thinking Mode
|
||||
|
||||
Qwen3-4B has a "thinking" mode enabled by default (responses start with `[Start thinking]`). To disable it and get direct answers:
|
||||
|
||||
```bash
|
||||
./llama-cli \
|
||||
-m ~/models/Qwen3-4B-Q8_0.gguf \
|
||||
-ngl 99 \
|
||||
-c 4096 \
|
||||
--temp 0.7 \
|
||||
--top-k 20 \
|
||||
--top-p 0.8 \
|
||||
--jinja \
|
||||
--chat-template-file ~/llama.cpp/models/templates/qwen3.jinja \
|
||||
-e
|
||||
```
|
||||
|
||||
Or append `/no_think` to your prompt for per-message control.
|
||||
|
||||
---
|
||||
|
||||
## Step 5: Run Inference (One-Shot / Batch)
|
||||
|
||||
For scripting or single-prompt usage without interactive mode:
|
||||
|
||||
```bash
|
||||
cd ~/llama.cpp/build/bin
|
||||
|
||||
./llama-cli \
|
||||
-m ~/models/Qwen3-4B-Q8_0.gguf \
|
||||
-ngl 99 \
|
||||
-p "Explain what ROCm is in 2 sentences." \
|
||||
-n 200 \
|
||||
--no-display-prompt \
|
||||
--no-conversation
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 6: Run as API Server
|
||||
|
||||
llama.cpp includes an OpenAI-compatible HTTP API server:
|
||||
|
||||
```bash
|
||||
cd ~/llama.cpp/build/bin
|
||||
|
||||
./llama-server \
|
||||
-m ~/models/Qwen3-4B-Q8_0.gguf \
|
||||
-ngl 99 \
|
||||
-c 4096 \
|
||||
--host 0.0.0.0 \
|
||||
--port 8080
|
||||
```
|
||||
|
||||
### Test with curl
|
||||
|
||||
```bash
|
||||
curl http://localhost:8080/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "qwen3-4b",
|
||||
"messages": [
|
||||
{"role": "user", "content": "What is ROCm?"}
|
||||
],
|
||||
"max_tokens": 200,
|
||||
"temperature": 0.7
|
||||
}'
|
||||
```
|
||||
|
||||
### Web UI
|
||||
|
||||
Open `http://localhost:8080` in a browser for the built-in chat UI.
|
||||
|
||||
---
|
||||
|
||||
## Performance Results
|
||||
|
||||
Benchmarked on AMD BC-250 with full GPU offload (`-ngl 99`):
|
||||
|
||||
| Metric | Value |
|
||||
|----------------------|----------------|
|
||||
| **Prompt Processing** | ~84–267 t/s |
|
||||
| **Generation Speed** | ~55–57 t/s |
|
||||
| **Context Size** | 2048–4096 |
|
||||
| **GPU Offload** | All 36 layers |
|
||||
|
||||
> Prompt processing speed varies by prompt length (shorter prompts = higher t/s due to overhead ratio). Generation speed is consistently **~56-57 tokens/second**.
|
||||
|
||||
---
|
||||
|
||||
## VRAM / Memory Breakdown
|
||||
|
||||
From `llama_memory_breakdown_print` at exit (context size 2048):
|
||||
|
||||
| Location | Total | Free | Model | Context | Compute |
|
||||
|------------------|---------|--------|--------|---------|---------|
|
||||
| **ROCm0 (BC-250)** | 14750 MiB | 8382 MiB | 4076 MiB | 288 MiB | 301 MiB |
|
||||
| **Host (CPU)** | — | — | 394 MiB | 0 MiB | 14 MiB |
|
||||
|
||||
- **Model weights**: ~4.0 GiB VRAM (matches the Q8_0 file size)
|
||||
- **Remaining free VRAM**: ~8.4 GiB (plenty of room for larger context windows)
|
||||
- **Host RAM**: ~394 MiB for metadata
|
||||
|
||||
With context size 4096, VRAM usage for context doubles to ~576 MiB, still well within the 14.4 GiB available.
|
||||
|
||||
---
|
||||
|
||||
## Useful Parameters Reference
|
||||
|
||||
| Parameter | Default | Description |
|
||||
|-----------|---------|-------------|
|
||||
| `-m` | — | Path to GGUF model file |
|
||||
| `-ngl 99` | 0 | Number of layers to offload to GPU (99 = all) |
|
||||
| `-c` | 4096 | Context window size (in tokens) |
|
||||
| `-n` | -1 | Max tokens to generate (-1 = unlimited) |
|
||||
| `-p` | — | Initial prompt text |
|
||||
| `--temp` | 0.6 | Sampling temperature (lower = more deterministic) |
|
||||
| `--top-k` | 20 | Top-K sampling |
|
||||
| `--top-p` | 0.95 | Top-P (nucleus) sampling |
|
||||
| `--no-display-prompt` | off | Don't echo the prompt in output |
|
||||
| `--no-conversation` | off | Exit after first response (no interactive loop) |
|
||||
| `-t` | auto | Number of CPU threads |
|
||||
| `--host` | 127.0.0.1 | Server bind address |
|
||||
| `--port` | 8080 | Server port |
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### CMake Error: "CMAKE_HIP_COMPILER is set to the hipcc wrapper"
|
||||
|
||||
**Cause:** CMake ≥ 4.x rejects the `hipcc` wrapper script.
|
||||
**Fix:** Point to the ROCm clang directly:
|
||||
```bash
|
||||
-DCMAKE_HIP_COMPILER=/opt/rocm/lib/llvm/bin/clang++
|
||||
```
|
||||
|
||||
### Build Error: "No rule to make target 'libserver-context.a'"
|
||||
|
||||
**Cause:** Race condition from running multiple `make` processes in the same build directory simultaneously.
|
||||
**Fix:** Kill all builds, `rm -rf build`, and rebuild from scratch with a single `make -j$(nproc)`.
|
||||
|
||||
### "GGML_HIP:BOOL=OFF" in CMakeCache
|
||||
|
||||
**Cause:** ROCm dev libraries not found during cmake configuration.
|
||||
**Fix:** Ensure `/opt/rocm/lib/libhipblas.so` and `/opt/rocm/lib/llvm/bin/clang++` exist. Re-run cmake.
|
||||
|
||||
### Model file is 0 bytes after download
|
||||
|
||||
**Cause:** Incorrect URL (case-sensitive) — Hugging Face returns 404.
|
||||
**Fix:** The correct filename is `Qwen3-4B-Q8_0.gguf` (capital Q, capital B). Full URL:
|
||||
```
|
||||
https://huggingface.co/Qwen/Qwen3-4B-GGUF/resolve/main/Qwen3-4B-Q8_0.gguf
|
||||
```
|
||||
|
||||
### Slow GPU performance / "low-power state" warning
|
||||
|
||||
```
|
||||
WARNING: AMD GPU device(s) is/are in a low-power state
|
||||
```
|
||||
The BC-250 may throttle. Force performance mode:
|
||||
```bash
|
||||
sudo sh -c 'echo high > /sys/class/drm/card1/device/power_dpm_force_performance_level'
|
||||
```
|
||||
|
||||
### "Exception caught: map::at" in rocm-smi
|
||||
|
||||
Known BC-250 issue with rocm-smi power monitoring. Does not affect inference. Ignore safely.
|
||||
|
||||
---
|
||||
|
||||
## Notes & Tips
|
||||
|
||||
- **Qwen3 Thinking Mode**: By default, Qwen3 wraps responses in `[Start thinking]...[End thinking]` blocks showing its reasoning chain. This is a feature, not a bug. Use `--jinja` with the official template or `/no_think` to disable it.
|
||||
|
||||
- **Q8_0 Quantization**: This is the highest quality GGUF quantization (8-bit). The 4B parameter model at Q8_0 uses ~4 GiB VRAM, leaving plenty of headroom on the BC-250's ~14.4 GiB.
|
||||
|
||||
- **Full GPU Offload**: With `-ngl 99`, all 36 transformer layers are offloaded to the GPU. No CPU fallback needed for this model size.
|
||||
|
||||
- **Other Quant Options**: Qwen also provides Q4_K_M (~2.5 GiB) and Q4_0 (~2.3 GiB) variants on the same Hugging Face repo if you want to save VRAM for larger context windows.
|
||||
|
||||
- **Multiple Models**: The BC-250 has enough VRAM to potentially run larger models like Qwen3-8B at Q4_K_M quantization (~5 GiB).
|
||||
|
||||
---
|
||||
|
||||
## Quick Reference
|
||||
|
||||
```bash
|
||||
# Build (one-time)
|
||||
cd ~/llama.cpp && rm -rf build && mkdir build && cd build
|
||||
cmake .. -DGGML_HIP=ON -DAMDGPU_TARGETS="gfx1010" -DCMAKE_BUILD_TYPE=Release \
|
||||
-DCMAKE_HIP_COMPILER=/opt/rocm/lib/llvm/bin/clang++ -G "Unix Makefiles"
|
||||
make -j$(nproc)
|
||||
|
||||
# Download model (one-time)
|
||||
mkdir -p ~/models
|
||||
aria2c -x 16 -s 16 -k 1M -d ~/models -o Qwen3-4B-Q8_0.gguf \
|
||||
"https://huggingface.co/Qwen/Qwen3-4B-GGUF/resolve/main/Qwen3-4B-Q8_0.gguf"
|
||||
|
||||
# Run interactive chat
|
||||
~/llama.cpp/build/bin/llama-cli -m ~/models/Qwen3-4B-Q8_0.gguf -ngl 99 -c 4096
|
||||
|
||||
# Run API server
|
||||
~/llama.cpp/build/bin/llama-server -m ~/models/Qwen3-4B-Q8_0.gguf -ngl 99 -c 4096 --host 0.0.0.0 --port 8080
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
*Documentation generated on March 1, 2026. Based on llama.cpp build b8184, ROCm 7.2.0, AMD BC-250 (gfx1010).*
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,572 @@
|
||||
# Z-Image-Turbo on AMD BC-250 — Complete Guide
|
||||
|
||||
> **Hardware**: AMD BC-250 (Cyan Skillfish, gfx1013→gfx1010, 24 CUs, shared RAM)
|
||||
> **Backend**: ROCm 7.2.0 / HIP (hipBLAS)
|
||||
> **OS**: CachyOS, kernel 6.18.8-3-cachyos with v3 kernel patches
|
||||
> **Server**: [stable-diffusion.cpp-restapi](https://github.com/leejet/stable-diffusion.cpp) (sdcpp-restapi)
|
||||
|
||||
---
|
||||
|
||||
## 1. Overview
|
||||
|
||||
Z-Image-Turbo is a Stable Diffusion model optimized for fast image generation. On the BC-250, it runs via the ROCm/HIP backend using `sdcpp-restapi` as an HTTP server. The server provides a REST API and a WebUI for image generation.
|
||||
|
||||
### Architecture
|
||||
|
||||
```
|
||||
[User / WebUI] → HTTP :8080 → [sdcpp-restapi] → [stable-diffusion.cpp] → [HIP/hipBLAS] → [AMD BC-250 GPU]
|
||||
```
|
||||
|
||||
### Pipeline Phases
|
||||
|
||||
Each image generation goes through three phases:
|
||||
|
||||
| Phase | What it does | Time (BC-250) |
|
||||
|-------|-------------|---------------|
|
||||
| **Text-Encoding** (CLIP) | Encodes the text prompt into embeddings | ~0.5–0.6s |
|
||||
| **Sampling** (Diffusion) | Iterative denoising (8 steps with Euler sampler) | ~73–78s |
|
||||
| **VAE Decode** | Decodes latent space back to pixel image | ~14s |
|
||||
| **Total** | End-to-end generation | **~82–92s** |
|
||||
|
||||
---
|
||||
|
||||
## 2. Directory Layout
|
||||
|
||||
```
|
||||
~/stable-diffusion.cpp-restapi/
|
||||
├── build/
|
||||
│ ├── bin/sdcpp-restapi # Server binary
|
||||
│ └── config.json # Server configuration
|
||||
├── src/ # Server source code
|
||||
└── build/_deps/stable-diffusion-src/ # SD library (fetched via CMake)
|
||||
|
||||
~/sd-models/
|
||||
├── diffusion_models/
|
||||
│ └── z_image_turbo-Q5_K_S.gguf # Main diffusion model (5.2 GB, Q5_K_S quantized)
|
||||
├── vae/
|
||||
│ └── ae.safetensors # VAE decoder (335 MB, f32)
|
||||
├── llm/
|
||||
│ ├── Qwen3-4B-Instruct-2507-Q5_K_S.gguf # LLM for prompt enhancement (2.8 GB)
|
||||
│ ├── Qwen3-8B-Q4_K_M.gguf # Alternative LLM (5.0 GB)
|
||||
│ └── gpt-oss-20b-Q4_K_M.gguf # Large LLM (11.6 GB) — too big for BC-250
|
||||
├── lora/
|
||||
├── clip/
|
||||
├── controlnet/
|
||||
├── esrgan/
|
||||
└── taesd/
|
||||
|
||||
~/sd-outputs/ # Generated images output directory
|
||||
|
||||
~/start-zimage.sh # Launch script
|
||||
~/VibeROCm/ZImage_Documentation.md # This document
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. Building from Source
|
||||
|
||||
```bash
|
||||
cd ~/stable-diffusion.cpp-restapi
|
||||
mkdir -p build && cd build
|
||||
cmake .. -G Ninja \
|
||||
-DCMAKE_BUILD_TYPE=Release \
|
||||
-DSD_HIP=ON \
|
||||
-DSDCPP_WEBUI=ON \
|
||||
-DCMAKE_PREFIX_PATH=/opt/rocm
|
||||
ninja -j$(nproc --all)
|
||||
```
|
||||
|
||||
**Requirements**: ROCm 7.2.0, CMake, Ninja, hipBLAS/rocBLAS
|
||||
|
||||
---
|
||||
|
||||
## 4. Configuration
|
||||
|
||||
### Server Config (`~/stable-diffusion.cpp-restapi/build/config.json`)
|
||||
|
||||
```json
|
||||
{
|
||||
"server": {
|
||||
"host": "0.0.0.0",
|
||||
"port": 8080,
|
||||
"threads": 8
|
||||
},
|
||||
"paths": {
|
||||
"diffusion_models": "/home/dars/sd-models/diffusion_models",
|
||||
"vae": "/home/dars/sd-models/vae",
|
||||
"llm": "/home/dars/sd-models/llm",
|
||||
"lora": "/home/dars/sd-models/lora",
|
||||
"clip": "/home/dars/sd-models/clip",
|
||||
"controlnet": "/home/dars/sd-models/controlnet",
|
||||
"esrgan": "/home/dars/sd-models/esrgan",
|
||||
"taesd": "/home/dars/sd-models/taesd",
|
||||
"output": "/home/dars/sd-outputs"
|
||||
},
|
||||
"sd_defaults": {
|
||||
"n_threads": 10,
|
||||
"keep_clip_on_cpu": true,
|
||||
"keep_vae_on_cpu": true,
|
||||
"flash_attn": false,
|
||||
"offload_to_cpu": true,
|
||||
"free_params_immediately": true
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Environment Variables (set in `start-zimage.sh`)
|
||||
|
||||
#### Required
|
||||
|
||||
| Variable | Value | Why |
|
||||
|----------|-------|-----|
|
||||
| `HSA_OVERRIDE_GFX_VERSION` | `10.1.0` | BC-250 (gfx1013) needs gfx1010 spoof for ROCm |
|
||||
| `HSA_ENABLE_SDMA` | `0` | SDMA engine has hardware bugs on gfx1013 |
|
||||
| `HIP_VISIBLE_DEVICES` | `0` | Select the BC-250 GPU |
|
||||
| `ROCM_PATH` | `/opt/rocm` | ROCm installation path |
|
||||
| `HSA_TOOLS_LIB` | `""` | Disable profiling tools (stability) |
|
||||
| `HSA_TOOLS_REPORT_LOAD_FAILURE` | `0` | Suppress tool warnings |
|
||||
|
||||
#### Explicitly Unset (performance-critical!)
|
||||
|
||||
These old workaround variables were used during development and **destroy performance** if set. The start script explicitly unsets them:
|
||||
|
||||
| Variable | Effect if set | Impact |
|
||||
|----------|--------------|--------|
|
||||
| `GPU_MAX_HW_QUEUES=1` | Serializes all GPU operations to 1 queue | Severe slowdown |
|
||||
| `HIP_LAUNCH_BLOCKING=1` | Forces synchronous kernel launches | Prevents pipelining |
|
||||
| `GGML_CUDA_ENABLE_UNIFIED_MEMORY=1` | Uses hipMallocManaged (page fault overhead) | +20% slower |
|
||||
| `GGML_HIP_HOST_ALLOC=1` | Uses hipHostMalloc (zero-copy over PCIe) | +40% slower |
|
||||
| `GGML_CUDA_NO_PINNED=1` | Disables pinned memory pools | Not needed with v3 patches |
|
||||
| `GGML_HIP_NO_COARSE_GRAIN=1` | Disables coarse-grain memory | Not needed with v3 patches |
|
||||
| `HSA_DISABLE_FRAGMENT_ALLOCATOR=1` | Disables memory fragment allocator | Not needed with v3 patches |
|
||||
|
||||
---
|
||||
|
||||
## 5. Starting the Server
|
||||
|
||||
### Using the Launch Script (recommended)
|
||||
|
||||
```bash
|
||||
bash ~/start-zimage.sh
|
||||
```
|
||||
|
||||
Or in background with logging:
|
||||
|
||||
```bash
|
||||
nohup bash ~/start-zimage.sh > /tmp/zimage.log 2>&1 &
|
||||
```
|
||||
|
||||
The script handles:
|
||||
- GPU health check (aborts if KIQ fence timeout detected — reboot needed)
|
||||
- Unsetting old workaround variables
|
||||
- Setting correct ROCm environment
|
||||
- Starting the server
|
||||
|
||||
### Manual Start (if needed)
|
||||
|
||||
```bash
|
||||
export HSA_OVERRIDE_GFX_VERSION=10.1.0
|
||||
export HSA_ENABLE_SDMA=0
|
||||
export HSA_TOOLS_LIB=""
|
||||
export HIP_VISIBLE_DEVICES=0
|
||||
export ROCM_PATH=/opt/rocm
|
||||
export PATH="/opt/rocm/bin:$PATH"
|
||||
export LD_LIBRARY_PATH="/opt/rocm/lib"
|
||||
|
||||
# CRITICAL: unset old workarounds
|
||||
unset GPU_MAX_HW_QUEUES HIP_LAUNCH_BLOCKING
|
||||
unset GGML_CUDA_ENABLE_UNIFIED_MEMORY GGML_HIP_HOST_ALLOC
|
||||
unset GGML_CUDA_NO_PINNED GGML_HIP_NO_COARSE_GRAIN
|
||||
unset HSA_DISABLE_FRAGMENT_ALLOCATOR
|
||||
|
||||
cd ~/stable-diffusion.cpp-restapi/build
|
||||
./bin/sdcpp-restapi --config config.json
|
||||
```
|
||||
|
||||
### Endpoints
|
||||
|
||||
| Endpoint | Description |
|
||||
|----------|-------------|
|
||||
| `http://localhost:8080/ui` | Web UI |
|
||||
| `http://localhost:8080` | API root |
|
||||
| `ws://localhost:8081` | WebSocket (live progress) |
|
||||
|
||||
---
|
||||
|
||||
## 6. Loading Models
|
||||
|
||||
After the server starts, you must load a model before generating images.
|
||||
|
||||
### Load Z-Image-Turbo (recommended command)
|
||||
|
||||
```bash
|
||||
curl -s -X POST http://localhost:8080/models/load \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model_name": "z_image_turbo-Q5_K_S.gguf",
|
||||
"model_type": "diffusion",
|
||||
"vae": "ae.safetensors",
|
||||
"llm": "Qwen3-4B-Instruct-2507-Q5_K_S.gguf",
|
||||
"options": {
|
||||
"flash_attn": false
|
||||
}
|
||||
}'
|
||||
```
|
||||
|
||||
**Expected response** (takes ~25 seconds):
|
||||
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"message": "Model loaded successfully",
|
||||
"model_name": "z_image_turbo-Q5_K_S.gguf",
|
||||
"model_type": "diffusion",
|
||||
"loaded_components": {
|
||||
"vae": "ae.safetensors",
|
||||
"llm": "Qwen3-4B-Instruct-2507-Q5_K_S.gguf"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Model Load API Reference
|
||||
|
||||
**POST** `/models/load`
|
||||
|
||||
| Field | Type | Required | Description |
|
||||
|-------|------|----------|-------------|
|
||||
| `model_name` | string | **Yes** | Filename of the diffusion model (must be in `diffusion_models/` dir) |
|
||||
| `model_type` | string | **Yes** | `"diffusion"` for SD models |
|
||||
| `vae` | string | Yes | VAE filename (from `vae/` dir) |
|
||||
| `llm` | string | Optional | LLM for prompt enhancement (from `llm/` dir) |
|
||||
| `options` | object | Optional | Model options (see below) |
|
||||
|
||||
### Model Load Options
|
||||
|
||||
| Option | Type | Default | Recommended | Description |
|
||||
|--------|------|---------|-------------|-------------|
|
||||
| `flash_attn` | bool | `false` | **`false`** | Flash attention — gfx1010 does NOT support it |
|
||||
| `keep_clip_on_cpu` | bool | `true` | `true` | CLIP runs fine on CPU, saves GPU memory |
|
||||
| `keep_vae_on_cpu` | bool | `false` | `false` | VAE on GPU is faster |
|
||||
| `offload_to_cpu` | bool | `false` | `false` | CPU offload hurts performance |
|
||||
| `vae_decode_only` | bool | `true` | `true` | Only need decode for txt2img |
|
||||
| `free_params_immediately` | bool | `false` | `false` | Frees memory faster but slower load |
|
||||
| `vae_conv_direct` | bool | `false` | **`false`** | Direct convolutions — model load never finishes! |
|
||||
| `diffusion_conv_direct` | bool | `false` | **`false`** | Direct convolutions — model load never finishes! |
|
||||
| `n_threads` | int | `-1` (auto) | `-1` | CPU threads for compute |
|
||||
| `enable_mmap` | bool | `true` | `true` | Memory-mapped file loading |
|
||||
|
||||
### Unload Model
|
||||
|
||||
```bash
|
||||
curl -s -X POST http://localhost:8080/models/unload
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 7. Generating Images
|
||||
|
||||
### Via API (curl)
|
||||
|
||||
**Step 1: Submit a generation job**
|
||||
|
||||
```bash
|
||||
curl -s -X POST http://localhost:8080/txt2img \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"prompt": "blonde woman",
|
||||
"width": 512,
|
||||
"height": 1024,
|
||||
"steps": 8,
|
||||
"cfg_scale": 1,
|
||||
"sampler": "euler",
|
||||
"scheduler": "smoothstep",
|
||||
"seed": 42
|
||||
}'
|
||||
```
|
||||
|
||||
Response:
|
||||
|
||||
```json
|
||||
{
|
||||
"job_id": "e95872f8-15cb-48ce-a10f-19f2de430d9e",
|
||||
"position": 1,
|
||||
"status": "pending"
|
||||
}
|
||||
```
|
||||
|
||||
**Step 2: Poll for completion**
|
||||
|
||||
```bash
|
||||
curl -s http://localhost:8080/queue/<job_id>
|
||||
```
|
||||
|
||||
Response when done:
|
||||
|
||||
```json
|
||||
{
|
||||
"status": "completed",
|
||||
"duration": "92.2s",
|
||||
"outputs": ["image_filename.png"]
|
||||
}
|
||||
```
|
||||
|
||||
### Via WebUI
|
||||
|
||||
Open `http://localhost:8080/ui` in a browser.
|
||||
|
||||
### Generation Parameters
|
||||
|
||||
| Parameter | Type | Default | Description |
|
||||
|-----------|------|---------|-------------|
|
||||
| `prompt` | string | Required | Text description of the image |
|
||||
| `negative_prompt` | string | `""` | What to avoid |
|
||||
| `width` | int | 512 | Image width (pixels) |
|
||||
| `height` | int | 512 | Image height (pixels) |
|
||||
| `steps` | int | 20 | Sampling steps (8 is optimal for Z-Image-Turbo) |
|
||||
| `cfg_scale` | float | 7.0 | Classifier-free guidance scale (1.0 for turbo models) |
|
||||
| `sampler` | string | `"euler"` | Sampling method |
|
||||
| `scheduler` | string | `"smoothstep"` | Noise schedule |
|
||||
| `seed` | int | random | Seed for reproducibility (-1 = random) |
|
||||
|
||||
### Recommended Settings for Z-Image-Turbo
|
||||
|
||||
```json
|
||||
{
|
||||
"steps": 8,
|
||||
"cfg_scale": 1,
|
||||
"sampler": "euler",
|
||||
"scheduler": "smoothstep"
|
||||
}
|
||||
```
|
||||
|
||||
Z-Image-Turbo is a distilled model — 8 steps is the sweet spot. More steps don't improve quality.
|
||||
|
||||
---
|
||||
|
||||
## 8. Performance Benchmarks
|
||||
|
||||
All benchmarks: 512×1024 image, 8 steps, Euler sampler, smoothstep scheduler, seed 42, prompt "blonde woman".
|
||||
|
||||
### Best Result: ~82s (measured warm run, 3rd consecutive generation)
|
||||
|
||||
### Systematic Benchmark Matrix
|
||||
|
||||
| Config | Text-Enc | Sampling | VAE Decode | Total | vs Baseline |
|
||||
|--------|----------|----------|------------|-------|-------------|
|
||||
| **Clean (cold, 1st gen)** | 0.54s | 77.4s | 13.9s | **92s** | Baseline |
|
||||
| **Clean (warm, 2nd gen)** | — | — | — | **85s** | -8% |
|
||||
| **Clean (warm, 3rd gen)** | — | — | — | **82s** | -11% |
|
||||
| `GGML_CUDA_ENABLE_UNIFIED_MEMORY=1` | 0.52s | 82.7s | 26.1s | **109s** | +18% slower |
|
||||
| `GGML_HIP_HOST_ALLOC=1` | 0.55s | 98.3s | 29.8s | **129s** | +40% slower |
|
||||
| `conv_direct=true` (both) | — | — | — | **N/A** | Model load stalls (>13 min, aborted) |
|
||||
| Vulkan backend (RADV) | — | — | — | **~79s** | Reference |
|
||||
|
||||
### Key Findings
|
||||
|
||||
1. **Clean config is best**: No GGML environment variables. The v3 kernel patches make all old workarounds unnecessary.
|
||||
|
||||
2. **Warm vs cold run**: First generation after model load takes ~92s. Subsequent runs improve to ~85s (2nd) → ~82s (3rd+) as GPU caches warm up.
|
||||
|
||||
3. **GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 hurts**: `hipMallocManaged` causes page fault overhead → +18% slower sampling, +88% slower VAE.
|
||||
|
||||
4. **GGML_HIP_HOST_ALLOC=1 worst**: Zero-copy over PCIe is terrible → +27% slower sampling, +114% slower VAE.
|
||||
|
||||
5. **conv_direct is broken**: Setting `vae_conv_direct=true` and/or `diffusion_conv_direct=true` causes the model load to hang indefinitely (13+ minutes, never completes). These bypass rocBLAS im2col+GEMM path for direct convolutions, but the compute graph build is too expensive for this model size (244 VAE f32 tensors + 180 diffusion q5_K tensors).
|
||||
|
||||
6. **ROCm vs Vulkan gap**: ROCm (82–92s) vs Vulkan (79s). ROCm warm runs are very close to Vulkan (~82s vs ~79s, only ~4% gap). The remaining overhead is ggml compute graph scheduling, not GPU compute itself (~10s actual GPU time vs ~67s framework overhead in sampling).
|
||||
|
||||
### Timing Breakdown
|
||||
|
||||
The 92s total breaks down as:
|
||||
|
||||
```
|
||||
Text-Encoding (CLIP on CPU): 0.5s ( 0.5%) — NOT a bottleneck
|
||||
Sampling (8 steps diffusion): 77.4s ( 84.1%) — main bottleneck
|
||||
├── Actual GPU compute: ~10s ( 10.9%)
|
||||
└── ggml framework overhead: ~67s ( 73.2%) — graph scheduling, kernel launches
|
||||
VAE Decode: 13.9s ( 15.1%)
|
||||
```
|
||||
|
||||
The 67s ggml overhead is an upstream limitation of the stable-diffusion.cpp framework, not something fixable via configuration.
|
||||
|
||||
---
|
||||
|
||||
## 9. Tested Configurations (Full History)
|
||||
|
||||
All the environment variable combinations tested during optimization:
|
||||
|
||||
### Config 1: "All Workarounds" (pre-v3 era)
|
||||
|
||||
```bash
|
||||
GPU_MAX_HW_QUEUES=1
|
||||
HIP_LAUNCH_BLOCKING=1
|
||||
GGML_CUDA_ENABLE_UNIFIED_MEMORY=1
|
||||
GGML_HIP_HOST_ALLOC=1
|
||||
GGML_CUDA_NO_PINNED=1
|
||||
GGML_HIP_NO_COARSE_GRAIN=1
|
||||
HSA_DISABLE_FRAGMENT_ALLOCATOR=1
|
||||
```
|
||||
|
||||
**Result**: ~155s initially, ~82s with warm cache. These variables were needed before v3 kernel patches but are harmful now.
|
||||
|
||||
### Config 2: Clean (current optimal)
|
||||
|
||||
```bash
|
||||
# NO GGML variables set at all
|
||||
# Only ROCm basics:
|
||||
HSA_OVERRIDE_GFX_VERSION=10.1.0
|
||||
HSA_ENABLE_SDMA=0
|
||||
HIP_VISIBLE_DEVICES=0
|
||||
```
|
||||
|
||||
**Result**: 92s cold / ~82s warm (3rd gen). **This is the recommended config.**
|
||||
|
||||
### Config 3: UNIFIED_MEMORY only
|
||||
|
||||
```bash
|
||||
GGML_CUDA_ENABLE_UNIFIED_MEMORY=1
|
||||
```
|
||||
|
||||
**Result**: 109s (+18%). hipMallocManaged page faults add overhead.
|
||||
|
||||
### Config 4: HIP_HOST_ALLOC only
|
||||
|
||||
```bash
|
||||
GGML_HIP_HOST_ALLOC=1
|
||||
```
|
||||
|
||||
**Result**: 129s (+40%). Zero-copy over PCIe kills performance.
|
||||
|
||||
### Config 5: Conv Direct
|
||||
|
||||
```bash
|
||||
# Model load options:
|
||||
vae_conv_direct=true
|
||||
diffusion_conv_direct=true
|
||||
```
|
||||
|
||||
**Result**: Model load never completes (>13 minutes at 100% CPU, aborted). Not viable.
|
||||
|
||||
---
|
||||
|
||||
## 10. Troubleshooting
|
||||
|
||||
### Server won't start
|
||||
|
||||
```
|
||||
[ERROR] GPU is in a broken state (KIQ fence timeout detected)
|
||||
```
|
||||
|
||||
**Fix**: Reboot the system. The BC-250 GPU cannot recover from KIQ timeouts.
|
||||
|
||||
```bash
|
||||
sudo reboot
|
||||
```
|
||||
|
||||
### Model load fails
|
||||
|
||||
```json
|
||||
{"error": "Main model not found: ''"}
|
||||
```
|
||||
|
||||
**Fix**: Use `model_name` field (not `model` or `model_path`). Value must be the **filename only** (not full path). The server looks in the configured `diffusion_models/` directory.
|
||||
|
||||
### Generation hangs or crashes
|
||||
|
||||
- **Do NOT run** `rocm-smi`, `clinfo`, or any other GPU query tool while the server is running
|
||||
- **Do NOT start** multiple GPU processes simultaneously
|
||||
- If the server crashes, **reboot before restarting** — the GPU state may be corrupted
|
||||
|
||||
### Old environment variables leaking from .bashrc
|
||||
|
||||
If generation is unexpectedly slow (>100s), check for leftover variables:
|
||||
|
||||
```bash
|
||||
env | grep -E "GGML_|GPU_MAX|HIP_LAUNCH"
|
||||
```
|
||||
|
||||
If any of the old workaround vars are set, unset them:
|
||||
|
||||
```bash
|
||||
unset GPU_MAX_HW_QUEUES HIP_LAUNCH_BLOCKING
|
||||
unset GGML_CUDA_ENABLE_UNIFIED_MEMORY GGML_HIP_HOST_ALLOC
|
||||
unset GGML_CUDA_NO_PINNED GGML_HIP_NO_COARSE_GRAIN
|
||||
unset HSA_DISABLE_FRAGMENT_ALLOCATOR
|
||||
```
|
||||
|
||||
### flash_attn errors
|
||||
|
||||
```
|
||||
gfx1010 does not support flash attention
|
||||
```
|
||||
|
||||
**Fix**: Always load with `"flash_attn": false`. The gfx1010 architecture does not have the required hardware. This is already the default in the patched code.
|
||||
|
||||
---
|
||||
|
||||
## 11. Quick Reference
|
||||
|
||||
### Start Server + Load Model + Generate Image (one-liner workflow)
|
||||
|
||||
```bash
|
||||
# 1. Start server
|
||||
nohup bash ~/start-zimage.sh > /tmp/zimage.log 2>&1 &
|
||||
sleep 5
|
||||
|
||||
# 2. Load model
|
||||
curl -s -X POST http://localhost:8080/models/load \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model_name": "z_image_turbo-Q5_K_S.gguf",
|
||||
"model_type": "diffusion",
|
||||
"vae": "ae.safetensors",
|
||||
"llm": "Qwen3-4B-Instruct-2507-Q5_K_S.gguf",
|
||||
"options": {"flash_attn": false}
|
||||
}'
|
||||
|
||||
# 3. Generate image
|
||||
JOB=$(curl -s -X POST http://localhost:8080/txt2img \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"prompt": "blonde woman",
|
||||
"width": 512, "height": 1024,
|
||||
"steps": 8, "cfg_scale": 1,
|
||||
"sampler": "euler", "scheduler": "smoothstep"
|
||||
}' | python3 -c "import sys,json; print(json.load(sys.stdin)['job_id'])")
|
||||
|
||||
echo "Job: $JOB"
|
||||
|
||||
# 4. Poll until done
|
||||
while true; do
|
||||
STATUS=$(curl -s http://localhost:8080/queue/$JOB | python3 -c "import sys,json; d=json.load(sys.stdin); print(d.get('status','?'))")
|
||||
echo "Status: $STATUS"
|
||||
[ "$STATUS" = "completed" ] && break
|
||||
sleep 10
|
||||
done
|
||||
|
||||
# 5. Check result
|
||||
curl -s http://localhost:8080/queue/$JOB | python3 -m json.tool
|
||||
```
|
||||
|
||||
### Check Server Logs
|
||||
|
||||
```bash
|
||||
tail -f /tmp/zimage.log
|
||||
```
|
||||
|
||||
### Check Timing Breakdown
|
||||
|
||||
```bash
|
||||
grep -a "get_learned\|sampling completed\|decode_first\|generate_image completed" /tmp/zimage.log | tail -4
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 12. Important Warnings
|
||||
|
||||
1. **Do NOT run `rocm-smi`** or any GPU monitoring tool while the server is running — it can crash the GPU
|
||||
2. **Reboot after any crash** — the BC-250 GPU state cannot be recovered without a full reboot
|
||||
3. **Do NOT set old GGML environment variables** — they are not needed with v3 kernel patches and severely hurt performance
|
||||
4. **Do NOT use `conv_direct`** — model load hangs indefinitely
|
||||
5. **Do NOT use `flash_attn: true`** — gfx1010 does not support flash attention
|
||||
6. **One GPU process at a time** — the BC-250 cannot handle concurrent GPU workloads
|
||||
@@ -0,0 +1,12 @@
|
||||
# AMD BC-250 (Cyan Skillfish / gfx1013) — ROCm Stability Parameters
|
||||
# This GPU is APU-like with shared system memory (no dedicated VRAM).
|
||||
#
|
||||
# noretry=0 — Allow page fault retry (critical for shared memory / APU)
|
||||
# gpu_recovery=1 — Enable GPU recovery on timeout
|
||||
# sched_hw_submission=2 — Limit concurrent HW submissions (prevent queue overload)
|
||||
# ppfeaturemask=0xfff73ef7 — Disable GFXOFF (bit 15), SCLK_DEEP_SLEEP (bit 3),
|
||||
# and ULV (bit 8) to prevent GPU from entering
|
||||
# unrecoverable power-saving states.
|
||||
# Clock management is handled by cyan-skillfish-governor.
|
||||
# Default is 0xfff7bfff.
|
||||
options amdgpu noretry=0 gpu_recovery=1 sched_hw_submission=2 ppfeaturemask=0xfff73ef7
|
||||
@@ -0,0 +1,106 @@
|
||||
#!/bin/bash
|
||||
# =============================================================================
|
||||
# GPU Watchdog for AMD BC-250 (Cyan Skillfish)
|
||||
# =============================================================================
|
||||
# Monitors kernel logs for KIQ fence timeouts and immediately kills the
|
||||
# offending GPU process before the timeout cascade crashes the system.
|
||||
#
|
||||
# BC-250 Crash Pattern:
|
||||
# 1st KIQ timeout → 12s → 2nd KIQ timeout → 12s → cascade → hard crash
|
||||
# Window to act: ~10 seconds after first timeout
|
||||
#
|
||||
# Usage:
|
||||
# ./gpu_watchdog.sh # Monitor and auto-kill
|
||||
# ./gpu_watchdog.sh --dry-run # Monitor only, don't kill
|
||||
# ./gpu_watchdog.sh --max-kiq 2 # Kill after 2 KIQ timeouts
|
||||
# =============================================================================
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
MAX_KIQ_TIMEOUTS=1
|
||||
CHECK_INTERVAL=2
|
||||
DRY_RUN=false
|
||||
LOG_FILE="/tmp/gpu_watchdog.log"
|
||||
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--dry-run) DRY_RUN=true; shift ;;
|
||||
--max-kiq) MAX_KIQ_TIMEOUTS="$2"; shift 2 ;;
|
||||
--interval) CHECK_INTERVAL="$2"; shift 2 ;;
|
||||
*) echo "Unknown: $1"; exit 1 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
log() {
|
||||
local msg="[$(date '+%H:%M:%S')] $1"
|
||||
echo "$msg"
|
||||
echo "$msg" >> "$LOG_FILE"
|
||||
}
|
||||
|
||||
get_kiq_count() {
|
||||
local count
|
||||
count=$(journalctl -k -b --no-pager 2>/dev/null | grep -c "timeout waiting for kiq fence" | head -1 | tr -d '[:space:]')
|
||||
echo "${count:-0}"
|
||||
}
|
||||
|
||||
kill_gpu_processes() {
|
||||
local pids
|
||||
pids=$(fuser /dev/kfd 2>/dev/null || pgrep -f "sdcpp-restapi" 2>/dev/null || true)
|
||||
pids=$(echo "$pids" | xargs)
|
||||
|
||||
if [[ -z "$pids" ]]; then
|
||||
log "WARN: No GPU processes found"
|
||||
return 1
|
||||
fi
|
||||
|
||||
for pid in $pids; do
|
||||
local name
|
||||
name=$(ps -p "$pid" -o comm= 2>/dev/null || echo "?")
|
||||
if [[ "$DRY_RUN" == "true" ]]; then
|
||||
log "DRY-RUN: Would SIGKILL PID $pid ($name)"
|
||||
else
|
||||
log "SIGKILL PID $pid ($name)"
|
||||
kill -9 "$pid" 2>/dev/null || true
|
||||
fi
|
||||
done
|
||||
|
||||
if [[ "$DRY_RUN" == "false" ]]; then
|
||||
systemctl --user stop zimage 2>/dev/null || true
|
||||
log "Stopped zimage service"
|
||||
fi
|
||||
}
|
||||
|
||||
echo "=========================================="
|
||||
echo " GPU Watchdog — AMD BC-250"
|
||||
echo " Kill after: $MAX_KIQ_TIMEOUTS KIQ timeout(s)"
|
||||
echo " Interval: ${CHECK_INTERVAL}s"
|
||||
echo " Dry run: $DRY_RUN"
|
||||
echo "=========================================="
|
||||
|
||||
log "Watchdog started"
|
||||
|
||||
INITIAL_KIQ=$(get_kiq_count)
|
||||
PREV_KIQ=$INITIAL_KIQ
|
||||
log "Baseline KIQ count: $INITIAL_KIQ"
|
||||
|
||||
[[ "$INITIAL_KIQ" -gt 0 ]] && log "WARNING: GPU already has $INITIAL_KIQ pre-existing KIQ timeouts"
|
||||
|
||||
while true; do
|
||||
CUR=$(get_kiq_count)
|
||||
NEW=$((CUR - INITIAL_KIQ))
|
||||
|
||||
if [[ "$CUR" -ne "$PREV_KIQ" ]]; then
|
||||
log "!! KIQ timeout #$CUR (new=$NEW)"
|
||||
|
||||
if [[ "$NEW" -ge "$MAX_KIQ_TIMEOUTS" ]]; then
|
||||
log "!!! THRESHOLD HIT — EMERGENCY KILL !!!"
|
||||
kill_gpu_processes
|
||||
log "GPU processes killed. Reboot needed for clean GPU."
|
||||
INITIAL_KIQ=$CUR
|
||||
log "Watchdog reset, continuing..."
|
||||
fi
|
||||
fi
|
||||
|
||||
PREV_KIQ=$CUR
|
||||
sleep "$CHECK_INTERVAL"
|
||||
done
|
||||
@@ -0,0 +1,180 @@
|
||||
# Z-Image-Turbo on AMD BC-250 — Setup Documentation
|
||||
|
||||
## Overview
|
||||
|
||||
**Z-Image-Turbo** image generation server running on **AMD BC-250** (GFX1013 RDNA2, 24 CUs) via **Vulkan** (RADV/Mesa) on **CachyOS**.
|
||||
|
||||
| Component | Detail |
|
||||
|-----------|--------|
|
||||
| **GPU** | AMD BC-250 — 24 CUs, GFX1013 RDNA2, 16 GB shared GDDR6 |
|
||||
| **Driver** | RADV (Mesa 25.3.4) via Vulkan 1.4.335 |
|
||||
| **Backend** | Vulkan only (no ROCm) |
|
||||
| **Server** | [stable-diffusion.cpp-restapi](https://github.com/fszontagh/stable-diffusion.cpp-restapi) |
|
||||
| **OS** | CachyOS (Arch-based) |
|
||||
|
||||
## Models
|
||||
|
||||
| Model | File | Size | Location |
|
||||
|-------|------|------|----------|
|
||||
| Z-Image-Turbo | `z_image_turbo-Q5_K_S.gguf` | 4.9 GB | `~/sd-models/diffusion_models/` |
|
||||
| Qwen3-4B Instruct | `Qwen3-4B-Instruct-2507-Q5_K_S.gguf` | 2.8 GB | `~/sd-models/llm/` |
|
||||
| FLUX VAE | `ae.safetensors` | 320 MB | `~/sd-models/vae/` |
|
||||
|
||||
## Directory Structure
|
||||
|
||||
```
|
||||
~/
|
||||
├── stable-diffusion.cpp-restapi/
|
||||
│ └── build/
|
||||
│ ├── bin/sdcpp-restapi # Server binary (80 MB)
|
||||
│ ├── config.json # Server configuration
|
||||
│ └── webui/ # Vue.js WebUI
|
||||
├── sd-models/
|
||||
│ ├── diffusion_models/ # Main diffusion model GGUFs (Z-Image)
|
||||
│ ├── checkpoints/ # Legacy checkpoint models
|
||||
│ ├── vae/ # VAE models
|
||||
│ ├── llm/ # LLM models (prompt enhancement)
|
||||
│ ├── lora/ # LoRA adapters
|
||||
│ ├── clip/ # CLIP models
|
||||
│ ├── t5/ # T5 text encoders
|
||||
│ ├── embeddings/ # Textual inversions
|
||||
│ ├── controlnet/ # ControlNet models
|
||||
│ ├── esrgan/ # Upscaler models
|
||||
│ └── taesd/ # Tiny AutoEncoder models
|
||||
├── sd-outputs/ # Generated images
|
||||
├── start-zimage.sh # Quick start script
|
||||
└── .config/systemd/user/
|
||||
└── zimage.service # Systemd user service
|
||||
```
|
||||
|
||||
## Quick Start
|
||||
|
||||
```bash
|
||||
# Option 1: Manual start
|
||||
~/start-zimage.sh
|
||||
|
||||
# Option 2: Systemd service
|
||||
systemctl --user start zimage
|
||||
```
|
||||
|
||||
Then open: **http://localhost:8080/ui**
|
||||
|
||||
## Usage
|
||||
|
||||
### WebUI
|
||||
|
||||
1. Open http://localhost:8080/ui
|
||||
2. Go to **Models** → select `z_image_turbo-Q5_K_S.gguf`
|
||||
3. Set VAE to `ae.safetensors`, LLM to `Qwen3-4B-Instruct-2507-Q5_K_S.gguf`
|
||||
4. First load takes 20–40 seconds
|
||||
5. Settings: **Steps 8**, **CFG 1.0**, **Euler** sampler
|
||||
6. Start with 512×512 (~37 s), then try 1024×1024 (~80 s)
|
||||
|
||||
### API — Load Model
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:8080/models/load \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model_name": "z_image_turbo-Q5_K_S.gguf",
|
||||
"model_type": "diffusion",
|
||||
"vae": "ae.safetensors",
|
||||
"llm": "Qwen3-4B-Instruct-2507-Q5_K_S.gguf"
|
||||
}'
|
||||
```
|
||||
|
||||
### API — Generate Image (512×512)
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:8080/txt2img \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"prompt": "astronaut on mars, cinematic",
|
||||
"width": 512,
|
||||
"height": 512,
|
||||
"steps": 8,
|
||||
"cfg_scale": 1.0,
|
||||
"sampler_name": "euler",
|
||||
"scheduler": "smoothstep",
|
||||
"seed": -1
|
||||
}' --output image.png
|
||||
```
|
||||
|
||||
## Systemd Service
|
||||
|
||||
```bash
|
||||
# Enable auto-start on login
|
||||
systemctl --user enable zimage
|
||||
|
||||
# Start / stop / restart
|
||||
systemctl --user start zimage
|
||||
systemctl --user stop zimage
|
||||
systemctl --user restart zimage
|
||||
|
||||
# View logs
|
||||
journalctl --user -u zimage -f
|
||||
```
|
||||
|
||||
## Vulkan Environment Variables
|
||||
|
||||
| Variable | Value | Purpose |
|
||||
|----------|-------|---------|
|
||||
| `AMD_VULKAN_ICD` | `RADV` | Use Mesa RADV driver |
|
||||
| `GGML_VK_FORCE_MAX_ALLOCATION_SIZE` | `536870912` | 512 MB max alloc (OOM prevention) |
|
||||
| `RADV_PERFTEST` | `nggc` | NGG culling compute boost |
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Slow generation (>20s at 1024×1024)
|
||||
```bash
|
||||
# Check logs for allocation failures
|
||||
journalctl --user -u zimage -f
|
||||
|
||||
# Try reducing max allocation to 256 MB
|
||||
export GGML_VK_FORCE_MAX_ALLOCATION_SIZE=268435456
|
||||
|
||||
# Try AMDVLK instead of RADV
|
||||
sudo pacman -S amdvlk
|
||||
export AMD_VULKAN_ICD=AMDVLK
|
||||
```
|
||||
|
||||
### Vulkan not detecting GPU
|
||||
```bash
|
||||
# Verify Vulkan
|
||||
vulkaninfo --summary | grep BC-250
|
||||
|
||||
# Force ICD file path
|
||||
VK_ICD_FILENAMES=/usr/share/vulkan/icd.d/radeon_icd.x86_64.json vulkaninfo
|
||||
```
|
||||
|
||||
### Out of Memory (OOM)
|
||||
- Use Q3_K_S quantization (smaller model)
|
||||
- Set `batch_count: 1` in generation requests
|
||||
- Reduce resolution to 512×512
|
||||
|
||||
### Monitor GPU
|
||||
```bash
|
||||
# GPU utilization
|
||||
watch radeontop
|
||||
|
||||
# VRAM usage
|
||||
cat /sys/class/drm/card0/device/mem_info_vram_used
|
||||
```
|
||||
|
||||
## Build from Source (Reference)
|
||||
|
||||
```bash
|
||||
cd ~/stable-diffusion.cpp-restapi
|
||||
mkdir -p build && cd build
|
||||
cmake .. -G Ninja -DCMAKE_BUILD_TYPE=Release -DSD_VULKAN=ON -DSDCPP_WEBUI=ON
|
||||
ninja -j$(nproc --all)
|
||||
```
|
||||
|
||||
## Performance Expectations
|
||||
|
||||
| Resolution | Expected Time | Notes |
|
||||
|------------|--------------|-------|
|
||||
| 512×512 | ~37 seconds | Q5_K_S, 8 steps, Euler, Vulkan |
|
||||
| 512×1024 | ~80 seconds | Q5_K_S, 8 steps, Euler, Vulkan |
|
||||
| 1024×1024 | ~150 seconds | Q5_K_S, 8 steps, Euler, Vulkan |
|
||||
| First load | 20–40 seconds | One-time on startup |
|
||||
Binary file not shown.
@@ -0,0 +1,80 @@
|
||||
/**
|
||||
* Minimal HIP diagnostic — step-by-step GPU compute validation
|
||||
* Tests each operation individually with timeout awareness
|
||||
*/
|
||||
#include <hip/hip_runtime.h>
|
||||
#include <cstdio>
|
||||
#include <cstdlib>
|
||||
#include <cstring>
|
||||
|
||||
#define HIP_CHECK(call) do { \
|
||||
hipError_t err = call; \
|
||||
if (err != hipSuccess) { \
|
||||
fprintf(stderr, "HIP ERROR [%d]: %s at %s:%d\n", \
|
||||
(int)err, hipGetErrorString(err), __FILE__, __LINE__); \
|
||||
exit(1); \
|
||||
} \
|
||||
} while(0)
|
||||
|
||||
__global__ void simpleKernel(int* out) {
|
||||
out[threadIdx.x] = threadIdx.x * 2;
|
||||
}
|
||||
|
||||
int main() {
|
||||
setbuf(stdout, NULL); // Force unbuffered output
|
||||
setbuf(stderr, NULL);
|
||||
|
||||
printf("STEP 0: HIP init\n");
|
||||
int count;
|
||||
HIP_CHECK(hipGetDeviceCount(&count));
|
||||
printf(" Devices: %d\n", count);
|
||||
|
||||
hipDeviceProp_t p;
|
||||
HIP_CHECK(hipGetDeviceProperties(&p, 0));
|
||||
printf(" Device: %s, CU=%d, Mem=%zuMB, isIntegrated=%d\n",
|
||||
p.name, p.multiProcessorCount, p.totalGlobalMem/(1024*1024), p.integrated);
|
||||
|
||||
printf("STEP 1: hipMalloc (small: 256 bytes)\n");
|
||||
int *d_buf = nullptr;
|
||||
HIP_CHECK(hipMalloc(&d_buf, 256));
|
||||
printf(" OK: d_buf=%p\n", (void*)d_buf);
|
||||
|
||||
printf("STEP 2: hipMemset\n");
|
||||
HIP_CHECK(hipMemset(d_buf, 0, 256));
|
||||
printf(" OK\n");
|
||||
|
||||
printf("STEP 3: hipMemcpy H2D (64 ints)\n");
|
||||
int h_buf[64];
|
||||
for (int i = 0; i < 64; i++) h_buf[i] = i + 100;
|
||||
HIP_CHECK(hipMemcpy(d_buf, h_buf, 256, hipMemcpyHostToDevice));
|
||||
printf(" OK\n");
|
||||
|
||||
printf("STEP 4: Launch kernel (1 block, 32 threads)\n");
|
||||
simpleKernel<<<1, 32>>>(d_buf);
|
||||
HIP_CHECK(hipGetLastError());
|
||||
printf(" Launched\n");
|
||||
|
||||
printf("STEP 5: hipDeviceSynchronize\n");
|
||||
HIP_CHECK(hipDeviceSynchronize());
|
||||
printf(" OK\n");
|
||||
|
||||
printf("STEP 6: hipMemcpy D2H\n");
|
||||
int h_out[64];
|
||||
memset(h_out, 0, 256);
|
||||
HIP_CHECK(hipMemcpy(h_out, d_buf, 128, hipMemcpyDeviceToHost));
|
||||
printf(" OK\n");
|
||||
|
||||
printf("STEP 7: Verify results\n");
|
||||
int pass = 1;
|
||||
for (int i = 0; i < 32; i++) {
|
||||
if (h_out[i] != i * 2) {
|
||||
printf(" FAIL: h_out[%d]=%d expected %d\n", i, h_out[i], i*2);
|
||||
pass = 0;
|
||||
}
|
||||
}
|
||||
if (pass) printf(" ALL 32 VALUES CORRECT\n");
|
||||
|
||||
HIP_CHECK(hipFree(d_buf));
|
||||
printf("\nRESULT: %s\n", pass ? "PASS — ROCm HIP COMPUTE WORKS" : "FAIL");
|
||||
return pass ? 0 : 1;
|
||||
}
|
||||
Binary file not shown.
@@ -0,0 +1,108 @@
|
||||
/**
|
||||
* HIP Minimal Probe — Safe diagnostic for AMD BC-250
|
||||
* Step-by-step: each step prints BEFORE attempting, so we know where it hangs/crashes
|
||||
*/
|
||||
#include <hip/hip_runtime.h>
|
||||
#include <cstdio>
|
||||
#include <unistd.h> // _exit() — bypasses C++ destructors
|
||||
|
||||
int main() {
|
||||
printf("PROBE: Starting HIP minimal probe...\n");
|
||||
fflush(stdout);
|
||||
|
||||
// Step 1: hipGetDeviceCount
|
||||
printf("PROBE [1/6]: hipGetDeviceCount... ");
|
||||
fflush(stdout);
|
||||
int count = 0;
|
||||
hipError_t err = hipGetDeviceCount(&count);
|
||||
if (err != hipSuccess) {
|
||||
printf("FAILED: %s\n", hipGetErrorString(err));
|
||||
return 1;
|
||||
}
|
||||
printf("OK (%d devices)\n", count);
|
||||
fflush(stdout);
|
||||
|
||||
if (count == 0) {
|
||||
printf("PROBE: No devices. Exiting.\n");
|
||||
return 1;
|
||||
}
|
||||
|
||||
// Step 2: hipGetDeviceProperties
|
||||
printf("PROBE [2/6]: hipGetDeviceProperties... ");
|
||||
fflush(stdout);
|
||||
hipDeviceProp_t props;
|
||||
err = hipGetDeviceProperties(&props, 0);
|
||||
if (err != hipSuccess) {
|
||||
printf("FAILED: %s\n", hipGetErrorString(err));
|
||||
return 1;
|
||||
}
|
||||
printf("OK\n");
|
||||
printf(" Name: %s\n", props.name);
|
||||
printf(" GCN Arch: %s\n", props.gcnArchName);
|
||||
printf(" CUs: %d\n", props.multiProcessorCount);
|
||||
printf(" Total Mem: %zu MB\n", props.totalGlobalMem / (1024*1024));
|
||||
printf(" Managed Memory: %s\n", props.managedMemory ? "YES" : "NO");
|
||||
printf(" Concurrent Managed: %s\n", props.concurrentManagedAccess ? "YES" : "NO");
|
||||
printf(" Integrated (APU): %s\n", props.integrated ? "YES" : "NO");
|
||||
printf(" pageableMemoryAccess: %s\n", props.pageableMemoryAccess ? "YES" : "NO");
|
||||
fflush(stdout);
|
||||
|
||||
// Step 3: hipSetDevice
|
||||
printf("PROBE [3/6]: hipSetDevice(0)... ");
|
||||
fflush(stdout);
|
||||
err = hipSetDevice(0);
|
||||
if (err != hipSuccess) {
|
||||
printf("FAILED: %s\n", hipGetErrorString(err));
|
||||
return 1;
|
||||
}
|
||||
printf("OK\n");
|
||||
fflush(stdout);
|
||||
|
||||
// Step 4: Try hipHostMalloc (pinned host memory — safest for APU)
|
||||
printf("PROBE [4/6]: hipHostMalloc (64 KB, coherent)... ");
|
||||
fflush(stdout);
|
||||
float* hostPtr = nullptr;
|
||||
err = hipHostMalloc(&hostPtr, 64 * 1024, hipHostMallocCoherent);
|
||||
if (err != hipSuccess) {
|
||||
printf("FAILED: %s\n", hipGetErrorString(err));
|
||||
printf(" Trying hipHostMallocDefault...\n");
|
||||
err = hipHostMalloc(&hostPtr, 64 * 1024, hipHostMallocDefault);
|
||||
if (err != hipSuccess) {
|
||||
printf(" Also FAILED: %s\n", hipGetErrorString(err));
|
||||
return 1;
|
||||
}
|
||||
}
|
||||
printf("OK (ptr=%p)\n", (void*)hostPtr);
|
||||
fflush(stdout);
|
||||
|
||||
// Step 5: Try hipMallocManaged (unified memory)
|
||||
printf("PROBE [5/6]: hipMallocManaged (64 KB)... ");
|
||||
fflush(stdout);
|
||||
float* managedPtr = nullptr;
|
||||
err = hipMallocManaged(&managedPtr, 64 * 1024);
|
||||
if (err != hipSuccess) {
|
||||
printf("FAILED: %s (this may be expected without XNACK)\n", hipGetErrorString(err));
|
||||
fflush(stdout);
|
||||
} else {
|
||||
printf("OK (ptr=%p)\n", (void*)managedPtr);
|
||||
fflush(stdout);
|
||||
// Write test
|
||||
managedPtr[0] = 42.0f;
|
||||
printf(" Write test: managedPtr[0] = %f\n", managedPtr[0]);
|
||||
fflush(stdout);
|
||||
hipFree(managedPtr);
|
||||
}
|
||||
|
||||
// Step 6: Free host memory — do NOT call hipDeviceReset()!
|
||||
// hipDeviceReset() causes KIQ fence timeout → system hang on BC-250
|
||||
printf("PROBE [6/6]: Cleanup (no device reset)... ");
|
||||
fflush(stdout);
|
||||
hipHostFree(hostPtr);
|
||||
// hipDeviceReset() intentionally omitted — crashes BC-250
|
||||
printf("OK\n");
|
||||
fflush(stdout);
|
||||
|
||||
printf("\nPROBE: ALL STEPS COMPLETED SUCCESSFULLY\n");
|
||||
fflush(stdout);
|
||||
_exit(0); // HARD EXIT — bypasses HIP runtime destructors that crash BC-250
|
||||
}
|
||||
Binary file not shown.
@@ -0,0 +1,151 @@
|
||||
/**
|
||||
* HIP Vector Addition Test — AMD BC-250 ROCm Validation
|
||||
*
|
||||
* Uses hipMallocManaged (unified memory) — REQUIRED for BC-250 which is an
|
||||
* APU-like device with shared system memory (no dedicated VRAM).
|
||||
* Standard hipMalloc + hipMemcpy will crash the system!
|
||||
*
|
||||
* Verifies GPU compute works end-to-end:
|
||||
* 1. HIP runtime initializes & device query
|
||||
* 2. Managed memory allocation (unified address space)
|
||||
* 3. GPU kernel execution
|
||||
* 4. Results are numerically correct
|
||||
*/
|
||||
#include <hip/hip_runtime.h>
|
||||
#include <cstdio>
|
||||
#include <cstdlib>
|
||||
#include <cmath>
|
||||
#include <unistd.h> // _exit() — bypasses C++ destructors
|
||||
|
||||
#define HIP_CHECK(call) do { \
|
||||
hipError_t err = call; \
|
||||
if (err != hipSuccess) { \
|
||||
fprintf(stderr, "HIP Error: %s at %s:%d\n", \
|
||||
hipGetErrorString(err), __FILE__, __LINE__); \
|
||||
fflush(stderr); \
|
||||
exit(1); \
|
||||
} \
|
||||
} while(0)
|
||||
|
||||
__global__ void vectorAdd(const float* A, const float* B, float* C, int N) {
|
||||
int i = blockDim.x * blockIdx.x + threadIdx.x;
|
||||
if (i < N) {
|
||||
C[i] = A[i] + B[i];
|
||||
}
|
||||
}
|
||||
|
||||
int main() {
|
||||
printf("=== HIP Vector Addition Test (Managed Memory) ===\n");
|
||||
printf(" Target: AMD BC-250 (gfx1013, APU shared memory)\n\n");
|
||||
fflush(stdout);
|
||||
|
||||
// Step 1: Query device
|
||||
printf("[1/4] Querying HIP device... ");
|
||||
fflush(stdout);
|
||||
int deviceCount = 0;
|
||||
HIP_CHECK(hipGetDeviceCount(&deviceCount));
|
||||
|
||||
if (deviceCount == 0) {
|
||||
printf("FAIL: No HIP devices found!\n");
|
||||
return 1;
|
||||
}
|
||||
|
||||
hipDeviceProp_t props;
|
||||
HIP_CHECK(hipGetDeviceProperties(&props, 0));
|
||||
printf("OK\n");
|
||||
printf(" Device: %s\n", props.name);
|
||||
printf(" GCN Arch: %s\n", props.gcnArchName);
|
||||
printf(" Compute Units: %d\n", props.multiProcessorCount);
|
||||
printf(" Total Memory: %zu MB (shared system RAM)\n", props.totalGlobalMem / (1024*1024));
|
||||
printf(" Integrated: %s\n", props.integrated ? "YES (APU)" : "NO");
|
||||
printf(" Managed Memory: %s\n", props.managedMemory ? "YES" : "NO");
|
||||
fflush(stdout);
|
||||
|
||||
// Step 2: Allocate MANAGED memory (unified — safe for APU/shared memory)
|
||||
const int N = 1 << 16; // 65536 elements
|
||||
size_t bytes = N * sizeof(float);
|
||||
|
||||
printf("[2/4] Allocating managed memory (%.1f KB x3)... ", bytes/1024.0);
|
||||
fflush(stdout);
|
||||
|
||||
float *A = nullptr, *B = nullptr, *C = nullptr;
|
||||
HIP_CHECK(hipMallocManaged(&A, bytes));
|
||||
HIP_CHECK(hipMallocManaged(&B, bytes));
|
||||
HIP_CHECK(hipMallocManaged(&C, bytes));
|
||||
printf("OK\n");
|
||||
fflush(stdout);
|
||||
|
||||
// Initialize on host (managed memory is accessible from both CPU and GPU)
|
||||
for (int i = 0; i < N; i++) {
|
||||
A[i] = sinf(i) * sinf(i);
|
||||
B[i] = cosf(i) * cosf(i);
|
||||
C[i] = 0.0f;
|
||||
}
|
||||
|
||||
// Step 3: Launch kernel
|
||||
int threadsPerBlock = 256;
|
||||
int blocksPerGrid = (N + threadsPerBlock - 1) / threadsPerBlock;
|
||||
|
||||
printf("[3/4] Launching kernel (%d blocks x %d threads)... ", blocksPerGrid, threadsPerBlock);
|
||||
fflush(stdout);
|
||||
|
||||
hipEvent_t start, stop;
|
||||
HIP_CHECK(hipEventCreate(&start));
|
||||
HIP_CHECK(hipEventCreate(&stop));
|
||||
|
||||
HIP_CHECK(hipEventRecord(start));
|
||||
hipLaunchKernelGGL(vectorAdd, dim3(blocksPerGrid), dim3(threadsPerBlock), 0, 0, A, B, C, N);
|
||||
HIP_CHECK(hipGetLastError());
|
||||
HIP_CHECK(hipEventRecord(stop));
|
||||
HIP_CHECK(hipEventSynchronize(stop));
|
||||
|
||||
float ms = 0;
|
||||
HIP_CHECK(hipEventElapsedTime(&ms, start, stop));
|
||||
printf("OK (%.3f ms)\n", ms);
|
||||
fflush(stdout);
|
||||
|
||||
// Step 4: Verify on host (managed memory — no memcpy needed!)
|
||||
printf("[4/4] Verifying results... ");
|
||||
fflush(stdout);
|
||||
|
||||
// Ensure GPU is done
|
||||
HIP_CHECK(hipDeviceSynchronize());
|
||||
|
||||
int errors = 0;
|
||||
for (int i = 0; i < N; i++) {
|
||||
float expected = A[i] + B[i]; // sin²(x) + cos²(x) = 1.0
|
||||
if (fabsf(C[i] - expected) > 1e-5) {
|
||||
if (errors < 5) {
|
||||
fprintf(stderr, "\n Mismatch at [%d]: got %f, expected %f",
|
||||
i, C[i], expected);
|
||||
}
|
||||
errors++;
|
||||
}
|
||||
}
|
||||
|
||||
if (errors == 0) {
|
||||
printf("PASSED — all %d elements correct (sin²+cos²=1.0)\n", N);
|
||||
} else {
|
||||
printf("FAILED — %d/%d mismatches\n", errors, N);
|
||||
}
|
||||
fflush(stdout);
|
||||
|
||||
// CRITICAL BC-250 WORKAROUND:
|
||||
// The HIP runtime's static destructors trigger KIQ queue teardown on the
|
||||
// BC-250's shared-memory architecture, causing "timeout waiting for kiq fence"
|
||||
// and a full system hang (no GPU reset possible on shared RAM APU).
|
||||
//
|
||||
// We MUST use _exit() to terminate immediately, bypassing:
|
||||
// - C++ static destructors (HIP runtime cleanup)
|
||||
// - atexit() handlers
|
||||
// - HIP's internal queue teardown via KIQ
|
||||
//
|
||||
// Memory is reclaimed by the OS. The GPU queues are released by KFD
|
||||
// when the process file descriptors are closed, which is safer than
|
||||
// the HIP runtime's explicit teardown path.
|
||||
|
||||
printf("\n=== ROCm HIP Compute: %s ===\n", errors == 0 ? "FULLY OPERATIONAL" : "FAILED");
|
||||
fflush(stdout);
|
||||
fflush(stderr);
|
||||
_exit(errors > 0 ? 1 : 0); // HARD EXIT — bypasses HIP destructors
|
||||
}
|
||||
@@ -0,0 +1,130 @@
|
||||
> The system will freeze and I have to force reboot it.
|
||||
You need an old kernel (~5.10.0) and proprietary drivers (It won't freeze completely and the amount of VRAM will be correctly determined in pytorch).
|
||||
Try:
|
||||
hiveos-0.6-217-stable (5.10.0-hiveos #110.hiveos.220411) or old ubuntu (focal or maybe jammy)
|
||||
22.20.5 AMDGPU-PRO driver
|
||||
https://repo.radeon.com/rocm/manylinux/
|
||||
https://repo.radeon.com/rocm/apt/
|
||||
|
||||
https://www.reddit.com/r/Hiveon_Official/comments/tzttyu/hiveon_os_v06215220409_whats_new/
|
||||
the closest version I could find hiveos-0.6-217-stable@220423
|
||||
https://web.archive.org/web/20220511060106/https://download.hiveos.farm/history/
|
||||
https://web.archive.org/web/20220514095728/https://download.hiveos.farm/history/hiveos-0.6-217-stable@220423.img.xz
|
||||
|
||||
https://github.com/minershive/hiveos-pxe-diskless/issues/26 ->
|
||||
https://github.com/panaceya/hiveos-pxe-diskless/compare/master...TheJames5:hiveos-pxe-diskless:master#diff-b335630551682c19a781afebcf4d07bf978fb1f8ac04c6bf87428ed5106870f5R7
|
||||
|
||||
also https://github.com/Gddrig/Qubic-AMD/releases see old releases
|
||||
|
||||
23
|
||||
You can try running it under the 1010 architecture:
|
||||
|
||||
export HSA_OVERRIDE_GFX_VERSION=10.1.0
|
||||
export AMDGPU_TARGETS=gfx1010
|
||||
export HCC_AMDGPU_TARGET=gfx1010
|
||||
export PYTORCH_ROCM_ARCH=gfx1010
|
||||
export HSA_ENABLE_SDMA=0
|
||||
export HSA_ENABLE_PEER_SDMA=0
|
||||
|
||||
|
||||
ROCM 5.2, which afaik is the "last known good" version for RDNA1 cards
|
||||
Something else to maybe try: IIRC from ROCM 5.3-6.0 they broke support for gfx101* (and fixed it again in 6.1). I remember reading that post-6.1+ there were some pretty significant performance regressions in 6.x for this GPU target, which weren't present in 5.2
|
||||
|
||||
Yeah when I was trying this, I wasn't able to get ROCm to build compute kernels/tensor libraries specifically for the gfx1013... had to build everything targetting gfx1010 and then run programs with HSA_OVERRIDE_GFX_VERSION=10.1.0 to force it to detect the gfx1013 as a gfx1010 (since its ISA is a superset of gfx1010, it should work in theory)
|
||||
afaik none of the rocm libraries have build configs for gfx1013 in any of the versions I checked, but they have gfx1010-1012
|
||||
Also be careful not to try to run gfx1030 code on it, since that's a different isa I believe (in some cases I had it default to this)
|
||||
|
||||
|
||||
|
||||
But you will probably have to compile rocm and pytorch kernels for the bc250 architecture (gfx1013).
|
||||
Examples of projects where they built for unsupported architectures:
|
||||
https://github.com/ulyssesrr/docker-rocm-xtra
|
||||
https://github.com/xuhuisheng/rocm-build/tree/master/navi10
|
||||
https://github.com/woodrex83/ROCm-For-RX580
|
||||
https://github.com/robertrosenbusch/gfx803_rocm
|
||||
|
||||
|
||||
more about regression https://github.com/ROCm/ROCm/discussions/4030
|
||||
GitHub
|
||||
Regression in rocm 5.3 and newer for gfx1010 · ROCm ROCm · Discus...
|
||||
Since when pytorch 2 was officially released, i wasn't able to run it on my 5700XT, while i was previously able to use it just fine on pytorch 1.13.1 by setting "export HSA_OVERRIDE_GFX_VE...
|
||||
Regression in rocm 5.3 and newer for gfx1010 · ROCm ROCm · Discus...
|
||||
|
||||
Starting glibc 2.41, the precompiled PyTorch wheels with ROCm 5.2 support no longer work due to changes in the stack execution policy for shared libraries.
|
||||
If you run a rolling release Linux distribution and have this version of glibc or later, compiling ROCm from source is the only way to stay current on RDNA1 hardware.
|
||||
|
||||
|
||||
NightFox
|
||||
—
|
||||
24.02.26, 22:37
|
||||
Weitergeleitet
|
||||
Item Version Description
|
||||
Operating System Ubuntu Ubuntu20.04.3
|
||||
Ubuntu18.04.6
|
||||
Ubuntu GFX amdgpu-pro-21.50-1347991-ubuntu-20.04.tar.xz \scbufs01\SCBUSW\SWQA\SCBU_SW_Programs\Robin\Drivers\EXT_release\3Dec
|
||||
amdgpu-pro-21.50-1347991-ubuntu-18.04.tar.xz
|
||||
TRM teamredminer-v0.8.6.6-linux.tgz \scbufs01\SCBUSW\SWQA\SCBU_SW_Programs\Robin\Tool\Offline_TRM\offline_benchmark
|
||||
BC250 Community • Dienstag
|
||||
Weitergeleitet
|
||||
Was what the p3.00 bios was certified on
|
||||
BC250 Community • Dienstag
|
||||
n00bos
|
||||
—
|
||||
24.02.26, 23:22
|
||||
i remember this is very true , had the same thing on my rx470
|
||||
probably we should use the same ubuntu version that works with the mi50 "hack"
|
||||
NightFox
|
||||
—
|
||||
25.02.26, 00:44
|
||||
AMD removed old versions from the repositories 🙁
|
||||
The only thing we have left is: https://download.hiveos.farm/repo/binary/amd-ocl/
|
||||
There is still 23.80 but it does not have Ubuntu package for ROCm
|
||||
n00bos
|
||||
—
|
||||
25.02.26, 00:58
|
||||
yes! i remember this when i was trying to get rocm working on my rx470 i was running in to amd removing old versions from repositories and i found it very sus
|
||||
maybe some data hoarder has them ?
|
||||
n00bos
|
||||
—
|
||||
25.02.26, 00:59
|
||||
can be compiled from source
|
||||
NightFox
|
||||
—
|
||||
25.02.26, 09:24
|
||||
Weitergeleitet
|
||||
oh hell yeah. 22.20 installs on ubuntu 20.04, BUT you have to edit the apt sources that the .deb file installs. since AMD has archived the repos now, and invoking the amdgpu-install script will fail for the missing repo.
|
||||
|
||||
cd /etc/apt/sources.list.d
|
||||
|
||||
|
||||
sudo nano amdgpu.list
|
||||
|
||||
|
||||
|
||||
in there you will see:
|
||||
deb https://repo.radeon.com/amdgpu/22.20/ubuntu focal main
|
||||
|
||||
change this to (add '.' before the 22.20):
|
||||
deb https://repo.radeon.com/amdgpu/.22.20/ubuntu focal main
|
||||
|
||||
make the same kind of change inside amdgpu-proprietary.list
|
||||
|
||||
then you can properly invoke the amdgpu-install script (i have a sever CLI install, no GUI so i did not install graphics)
|
||||
|
||||
amdgpu-install --usecase=opencl --opencl=rocr --accept-eula
|
||||
|
||||
BC250 Community • Mittwoch
|
||||
🅳🅳🅻
|
||||
—
|
||||
26.02.26, 04:06
|
||||
https://web.archive.org/ ???
|
||||
NightFox
|
||||
—
|
||||
26.02.26, 10:15
|
||||
There are no driver archives, but you can use a script to try archiving hidden dirs while they are still accessible.
|
||||
The installation packages are located here:
|
||||
https://repo.radeon.com/amdgpu/.22.20/ubuntu/pool/proprietary/
|
||||
https://repo.radeon.com/amdgpu/.21.15/ubuntu/pool/proprietary/
|
||||
The full list of versions is still available here:
|
||||
https://repo.radeon.com/amdgpu-install/
|
||||
I was thinking of downloading everything there and posting a mirror on GitHub, but I don't have time yet.
|
||||
@@ -0,0 +1,167 @@
|
||||
#!/bin/bash
|
||||
# BC-250 KIQ Fix - Post-Reboot Validation Script
|
||||
# Run this after rebooting with the patched amdgpu module
|
||||
#
|
||||
# Tests:
|
||||
# 1. Module loaded check
|
||||
# 2. dmesg for KIQ fence timeouts (should be ZERO)
|
||||
# 3. rocminfo
|
||||
# 4. hip_probe (with normal exit, no _exit hack)
|
||||
# 5. hip_vector_add
|
||||
# 6. Multiple sequential GPU runs (stress test)
|
||||
# 7. Process exit cleanup (the main fix target)
|
||||
|
||||
set +e # Don't exit on error - we want to see all results
|
||||
|
||||
RED='\033[0;31m'
|
||||
GREEN='\033[0;32m'
|
||||
YELLOW='\033[1;33m'
|
||||
NC='\033[0m'
|
||||
|
||||
pass() { echo -e "${GREEN}[PASS]${NC} $1"; }
|
||||
fail() { echo -e "${RED}[FAIL]${NC} $1"; }
|
||||
warn() { echo -e "${YELLOW}[WARN]${NC} $1"; }
|
||||
info() { echo -e " $1"; }
|
||||
|
||||
echo "=========================================="
|
||||
echo "BC-250 KIQ Fix - Post-Reboot Validation"
|
||||
echo "=========================================="
|
||||
echo "Date: $(date)"
|
||||
echo "Kernel: $(uname -r)"
|
||||
echo ""
|
||||
|
||||
# Test 1: Module loaded
|
||||
echo "--- Test 1: Module Status ---"
|
||||
if lsmod | grep -q amdgpu; then
|
||||
pass "amdgpu module is loaded"
|
||||
MODULE_SIZE=$(lsmod | grep "^amdgpu " | awk '{print $2}')
|
||||
info "Module size: ${MODULE_SIZE}K"
|
||||
else
|
||||
fail "amdgpu module not loaded!"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Test 2: Check for KIQ fence timeouts at boot
|
||||
echo ""
|
||||
echo "--- Test 2: KIQ Fence Timeouts (boot) ---"
|
||||
KIQ_ERRORS=$(dmesg | grep -ci "kiq.*fence\|fence.*kiq\|KIQ.*timeout" 2>/dev/null || echo "0")
|
||||
if [ "$KIQ_ERRORS" = "0" ]; then
|
||||
pass "No KIQ fence timeout errors in dmesg"
|
||||
else
|
||||
warn "Found $KIQ_ERRORS KIQ-related messages (checking if they're errors...)"
|
||||
dmesg | grep -i "kiq.*fence\|fence.*kiq\|KIQ.*timeout" | head -5
|
||||
fi
|
||||
|
||||
# Test 3: SDMA check (known issue, cosmetic)
|
||||
echo ""
|
||||
echo "--- Test 3: SDMA Status ---"
|
||||
SDMA_ERRORS=$(dmesg | grep -ci "sdma.*fence\|sdma.*timeout" 2>/dev/null || echo "0")
|
||||
if [ "$SDMA_ERRORS" = "0" ]; then
|
||||
pass "No SDMA fence errors"
|
||||
else
|
||||
warn "SDMA errors present (cosmetic, mitigated by HSA_ENABLE_SDMA=0)"
|
||||
fi
|
||||
|
||||
# Test 4: rocminfo
|
||||
echo ""
|
||||
echo "--- Test 4: rocminfo ---"
|
||||
if timeout 30 rocminfo 2>/dev/null | grep -q "gfx10"; then
|
||||
pass "rocminfo detects GPU"
|
||||
rocminfo 2>/dev/null | grep "Name:\|Marketing Name:\|Compute Unit:" | head -5
|
||||
else
|
||||
fail "rocminfo failed or timed out"
|
||||
fi
|
||||
|
||||
# Test 5: hip_probe
|
||||
echo ""
|
||||
echo "--- Test 5: hip_probe ---"
|
||||
HIP_PROBE="/home/dars/VibeROCm/hip_probe"
|
||||
if [ -x "$HIP_PROBE" ]; then
|
||||
if timeout 30 "$HIP_PROBE" 2>&1; then
|
||||
pass "hip_probe completed successfully"
|
||||
else
|
||||
fail "hip_probe failed (exit code: $?)"
|
||||
fi
|
||||
|
||||
# CRITICAL: Check for KIQ errors AFTER process exit
|
||||
sleep 3
|
||||
POST_KIQ=$(dmesg | tail -20 | grep -ci "kiq.*fence\|fence.*kiq\|KIQ.*timeout" 2>/dev/null || echo "0")
|
||||
if [ "$POST_KIQ" = "0" ]; then
|
||||
pass "No KIQ errors after hip_probe exit!"
|
||||
else
|
||||
fail "KIQ errors appeared after hip_probe exit"
|
||||
dmesg | tail -10
|
||||
fi
|
||||
else
|
||||
warn "hip_probe not found at $HIP_PROBE"
|
||||
fi
|
||||
|
||||
# Test 6: hip_vector_add
|
||||
echo ""
|
||||
echo "--- Test 6: hip_vector_add ---"
|
||||
HIP_VECTOR="/home/dars/VibeROCm/hip_vector_add"
|
||||
if [ -x "$HIP_VECTOR" ]; then
|
||||
if timeout 30 "$HIP_VECTOR" 2>&1; then
|
||||
pass "hip_vector_add completed successfully"
|
||||
else
|
||||
fail "hip_vector_add failed (exit code: $?)"
|
||||
fi
|
||||
|
||||
sleep 3
|
||||
POST_KIQ=$(dmesg | tail -20 | grep -ci "kiq.*fence\|fence.*kiq\|KIQ.*timeout" 2>/dev/null || echo "0")
|
||||
if [ "$POST_KIQ" = "0" ]; then
|
||||
pass "No KIQ errors after hip_vector_add exit!"
|
||||
else
|
||||
fail "KIQ errors appeared after hip_vector_add exit"
|
||||
fi
|
||||
else
|
||||
warn "hip_vector_add not found at $HIP_VECTOR"
|
||||
fi
|
||||
|
||||
# Test 7: Multiple sequential runs (stress test for process lifecycle)
|
||||
echo ""
|
||||
echo "--- Test 7: Sequential GPU Stress (5 rounds) ---"
|
||||
if [ -x "$HIP_VECTOR" ]; then
|
||||
STRESS_PASS=0
|
||||
for i in 1 2 3 4 5; do
|
||||
if timeout 30 "$HIP_VECTOR" >/dev/null 2>&1; then
|
||||
STRESS_PASS=$((STRESS_PASS + 1))
|
||||
else
|
||||
fail "Round $i failed"
|
||||
break
|
||||
fi
|
||||
sleep 1
|
||||
done
|
||||
|
||||
if [ "$STRESS_PASS" -eq 5 ]; then
|
||||
pass "All 5 sequential GPU runs completed successfully!"
|
||||
else
|
||||
fail "Only $STRESS_PASS/5 rounds passed"
|
||||
fi
|
||||
|
||||
# Final KIQ check
|
||||
sleep 5
|
||||
FINAL_KIQ=$(dmesg | tail -50 | grep -ci "kiq.*fence\|fence.*kiq\|KIQ.*timeout" 2>/dev/null || echo "0")
|
||||
if [ "$FINAL_KIQ" = "0" ]; then
|
||||
pass "No KIQ errors after stress test!"
|
||||
else
|
||||
fail "KIQ errors after stress test: $FINAL_KIQ"
|
||||
dmesg | tail -20 | grep -i "kiq\|fence"
|
||||
fi
|
||||
else
|
||||
warn "Skipping stress test"
|
||||
fi
|
||||
|
||||
# Test 8: rocminfo AFTER all GPU tests (this is the killer - previously hung)
|
||||
echo ""
|
||||
echo "--- Test 8: rocminfo After GPU Tests ---"
|
||||
if timeout 30 rocminfo 2>/dev/null | grep -q "gfx10"; then
|
||||
pass "rocminfo still works after GPU tests! (Previously this HUNG)"
|
||||
else
|
||||
fail "rocminfo hung or failed after GPU tests"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "=========================================="
|
||||
echo "Validation complete."
|
||||
echo "=========================================="
|
||||
@@ -0,0 +1,186 @@
|
||||
#!/bin/bash
|
||||
# BC-250 v3 Post-Reboot Verification Script
|
||||
# Run this IMMEDIATELY after booting with v3 module
|
||||
#
|
||||
# Tests in order of escalation:
|
||||
# 1. Module verification (no GPU access)
|
||||
# 2. Boot parameter verification
|
||||
# 3. sysfs GPU state check
|
||||
# 4. rocminfo (light GPU access)
|
||||
# 5. Single HIP test (compute)
|
||||
# 6. Sequential HIP test (the crash scenario from v2)
|
||||
#
|
||||
# Usage: bash post_reboot_v3_test.sh [--full]
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
RED='\033[0;31m'
|
||||
GREEN='\033[0;32m'
|
||||
YELLOW='\033[1;33m'
|
||||
NC='\033[0m'
|
||||
|
||||
pass() { echo -e "${GREEN}[PASS]${NC} $1"; }
|
||||
fail() { echo -e "${RED}[FAIL]${NC} $1"; }
|
||||
warn() { echo -e "${YELLOW}[WARN]${NC} $1"; }
|
||||
info() { echo -e " $1"; }
|
||||
|
||||
FULL_TEST=false
|
||||
[[ "${1:-}" == "--full" ]] && FULL_TEST=true
|
||||
|
||||
echo "============================================"
|
||||
echo " BC-250 v3 Post-Reboot Verification"
|
||||
echo " $(date)"
|
||||
echo "============================================"
|
||||
echo ""
|
||||
|
||||
# Test 1: Kernel version
|
||||
echo "--- Test 1: Kernel & Module ---"
|
||||
KVER=$(uname -r)
|
||||
if [[ "$KVER" == "6.18.8-3-cachyos" ]]; then
|
||||
pass "Kernel: $KVER"
|
||||
else
|
||||
warn "Unexpected kernel: $KVER"
|
||||
fi
|
||||
|
||||
# Test 2: v3 strings in loaded module
|
||||
V3_STRINGS=$(sudo dmesg | grep -c "BC-250" 2>/dev/null || echo "0")
|
||||
if [[ "$V3_STRINGS" -ge 1 ]]; then
|
||||
pass "v3 BC-250 messages in dmesg ($V3_STRINGS occurrences)"
|
||||
else
|
||||
warn "No BC-250 messages in dmesg yet (may appear on first GPU use)"
|
||||
fi
|
||||
|
||||
# Check GFXOFF disable message
|
||||
if sudo dmesg | grep -q "GFXOFF disabled"; then
|
||||
pass "GFXOFF disabled message confirmed in dmesg"
|
||||
else
|
||||
fail "GFXOFF disable message NOT found — v3 patch may not be loaded"
|
||||
fi
|
||||
|
||||
# Test 3: ppfeaturemask
|
||||
echo ""
|
||||
echo "--- Test 2: Boot Parameters ---"
|
||||
CMDLINE=$(cat /proc/cmdline)
|
||||
if echo "$CMDLINE" | grep -q "ppfeaturemask=0xfff73ef7"; then
|
||||
pass "ppfeaturemask=0xfff73ef7 in boot cmdline"
|
||||
else
|
||||
warn "ppfeaturemask not in boot cmdline (may be set via modprobe)"
|
||||
fi
|
||||
|
||||
# Check actual ppfeaturemask from module
|
||||
ACTUAL_PP=$(cat /sys/module/amdgpu/parameters/ppfeaturemask 2>/dev/null || echo "unknown")
|
||||
if [[ "$ACTUAL_PP" == "0xfff73ef7" ]] || [[ "$ACTUAL_PP" == "4293869303" ]]; then
|
||||
pass "Active ppfeaturemask: $ACTUAL_PP (GFXOFF+DeepSleep+ULV disabled)"
|
||||
else
|
||||
warn "Active ppfeaturemask: $ACTUAL_PP (expected 0xfff73ef7 / 4293869303)"
|
||||
fi
|
||||
|
||||
# Test 4: KIQ errors check
|
||||
echo ""
|
||||
echo "--- Test 3: KIQ Error Check ---"
|
||||
KIQ_ERRORS=$(sudo dmesg | grep -c "timeout waiting for kiq fence" 2>/dev/null || echo "0")
|
||||
if [[ "$KIQ_ERRORS" -eq 0 ]]; then
|
||||
pass "Zero KIQ timeout errors"
|
||||
else
|
||||
fail "Found $KIQ_ERRORS KIQ timeout errors!"
|
||||
fi
|
||||
|
||||
# Check for GPU unreachable messages
|
||||
GPU_DEAD=$(sudo dmesg | grep -c "GPU unreachable\|GPU died" 2>/dev/null || echo "0")
|
||||
if [[ "$GPU_DEAD" -eq 0 ]]; then
|
||||
pass "Zero GPU-unreachable events"
|
||||
else
|
||||
fail "Found $GPU_DEAD GPU health-check failures!"
|
||||
fi
|
||||
|
||||
# Test 5: sysfs GPU state
|
||||
echo ""
|
||||
echo "--- Test 4: GPU State ---"
|
||||
if [[ -f /sys/class/drm/card0/device/pp_dpm_sclk ]]; then
|
||||
SCLK=$(cat /sys/class/drm/card0/device/pp_dpm_sclk)
|
||||
pass "GPU clock levels accessible:"
|
||||
echo "$SCLK" | sed 's/^/ /'
|
||||
else
|
||||
fail "Cannot read GPU clock levels"
|
||||
fi
|
||||
|
||||
if [[ -f /sys/class/drm/card0/device/pp_od_clk_voltage ]]; then
|
||||
OD=$(cat /sys/class/drm/card0/device/pp_od_clk_voltage)
|
||||
pass "OD voltage table accessible:"
|
||||
echo "$OD" | sed 's/^/ /'
|
||||
else
|
||||
warn "Cannot read OD voltage table"
|
||||
fi
|
||||
|
||||
# Test 6: cyan-skillfish-governor
|
||||
echo ""
|
||||
echo "--- Test 5: Governor Status ---"
|
||||
if systemctl is-active --quiet cyan-skillfish-governor.service 2>/dev/null; then
|
||||
pass "cyan-skillfish-governor is running"
|
||||
else
|
||||
warn "cyan-skillfish-governor is NOT running"
|
||||
fi
|
||||
|
||||
# Test 7: rocminfo
|
||||
echo ""
|
||||
echo "--- Test 6: ROCm Runtime ---"
|
||||
if command -v rocminfo &>/dev/null; then
|
||||
ROCM_OUT=$(timeout 30 rocminfo 2>&1)
|
||||
if echo "$ROCM_OUT" | grep -q "gfx1013\|gfx10"; then
|
||||
pass "rocminfo detects GPU (gfx1013)"
|
||||
elif echo "$ROCM_OUT" | grep -q "Agent"; then
|
||||
pass "rocminfo detects agents"
|
||||
info "$(echo "$ROCM_OUT" | grep -i "name" | head -3)"
|
||||
else
|
||||
warn "rocminfo ran but output unexpected"
|
||||
fi
|
||||
else
|
||||
warn "rocminfo not found"
|
||||
fi
|
||||
|
||||
# Test 8: HIP test (only with --full)
|
||||
echo ""
|
||||
if $FULL_TEST; then
|
||||
echo "--- Test 7: HIP Compute (FULL MODE) ---"
|
||||
|
||||
HIP_TEST="/home/dars/VibeROCm/hip_vector_add/hip_vector_add"
|
||||
if [[ -x "$HIP_TEST" ]]; then
|
||||
echo " Running first HIP test..."
|
||||
if timeout 60 "$HIP_TEST" 2>&1; then
|
||||
pass "First HIP vector_add completed"
|
||||
sleep 3
|
||||
|
||||
echo ""
|
||||
echo " Running SECOND HIP test (this is the crash scenario)..."
|
||||
echo " Monitoring dmesg for GPU errors during test..."
|
||||
sudo dmesg -C # Clear dmesg
|
||||
if timeout 60 "$HIP_TEST" 2>&1; then
|
||||
pass "Second HIP vector_add completed — CRASH BUG RESOLVED!"
|
||||
# Check if any GPU errors occurred during the test
|
||||
POST_ERRORS=$(sudo dmesg | grep -c "BC-250.*GPU\|timeout\|error" 2>/dev/null || echo "0")
|
||||
if [[ "$POST_ERRORS" -eq 0 ]]; then
|
||||
pass "No GPU errors during sequential HIP tests"
|
||||
else
|
||||
warn "GPU events during test ($POST_ERRORS), check dmesg"
|
||||
fi
|
||||
else
|
||||
fail "Second HIP test failed or timed out"
|
||||
sudo dmesg | grep -i "BC-250\|error\|timeout\|GPU" | tail -10
|
||||
fi
|
||||
else
|
||||
fail "First HIP test failed"
|
||||
sudo dmesg | grep -i "BC-250\|error\|timeout\|GPU" | tail -10
|
||||
fi
|
||||
else
|
||||
warn "HIP test binary not found: $HIP_TEST"
|
||||
info "Build with: cd /home/dars/VibeROCm/hip_vector_add && hipcc hip_vector_add.cpp -o hip_vector_add"
|
||||
fi
|
||||
else
|
||||
echo "--- Test 7: HIP Compute (SKIPPED — use --full to enable) ---"
|
||||
info "Run: bash post_reboot_v3_test.sh --full"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "============================================"
|
||||
echo " Verification Complete"
|
||||
echo "============================================"
|
||||
@@ -0,0 +1,173 @@
|
||||
#!/bin/bash
|
||||
# BC-250 Safe Post-Reboot Test Script (v2)
|
||||
# Tests GPU functionality after KIQ bypass patch installation
|
||||
# CRITICAL: This script performs INCREMENTAL testing with safety checks
|
||||
set -e
|
||||
|
||||
RED='\033[0;31m'
|
||||
GREEN='\033[0;32m'
|
||||
YELLOW='\033[1;33m'
|
||||
NC='\033[0m'
|
||||
|
||||
pass() { echo -e "${GREEN}[PASS]${NC} $1"; }
|
||||
fail() { echo -e "${RED}[FAIL]${NC} $1"; }
|
||||
warn() { echo -e "${YELLOW}[WARN]${NC} $1"; }
|
||||
info() { echo -e "[INFO] $1"; }
|
||||
|
||||
ERRORS=0
|
||||
TESTS=0
|
||||
|
||||
echo "==========================================="
|
||||
echo " BC-250 KIQ Bypass Patch - Verification"
|
||||
echo " Date: $(date)"
|
||||
echo "==========================================="
|
||||
echo ""
|
||||
|
||||
# === STEP 0: Verify patch markers in dmesg ===
|
||||
((TESTS++))
|
||||
info "Step 0: Checking dmesg for patch markers..."
|
||||
if dmesg 2>/dev/null | grep -q "BC-250 KIQ bypass active"; then
|
||||
pass "BC-250 KIQ bypass markers found in dmesg"
|
||||
dmesg | grep "BC-250" | while read line; do echo " → $line"; done
|
||||
else
|
||||
warn "No BC-250 bypass markers in dmesg yet (may appear after first GPU use)"
|
||||
fi
|
||||
|
||||
# === STEP 1: Check for KIQ errors at boot ===
|
||||
((TESTS++))
|
||||
info "Step 1: Checking for KIQ errors in boot log..."
|
||||
if dmesg 2>/dev/null | grep -q "timeout waiting for kiq fence"; then
|
||||
fail "KIQ fence timeout detected at boot!"
|
||||
((ERRORS++))
|
||||
echo " → GPU is likely in a bad state. Do NOT proceed."
|
||||
exit 1
|
||||
else
|
||||
pass "No KIQ fence timeouts at boot"
|
||||
fi
|
||||
|
||||
# === STEP 2: Check GPU is alive ===
|
||||
((TESTS++))
|
||||
info "Step 2: Checking GPU hardware..."
|
||||
if ls /sys/class/drm/card0/device/ &>/dev/null; then
|
||||
pass "GPU DRM device present"
|
||||
else
|
||||
fail "No GPU DRM device"
|
||||
((ERRORS++))
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# === STEP 3: Check Z-Image-Turbo is disabled ===
|
||||
((TESTS++))
|
||||
info "Step 3: Checking Z-Image-Turbo service..."
|
||||
if systemctl --user is-active zimage.service 2>/dev/null | grep -q "^active"; then
|
||||
fail "Z-Image-Turbo is RUNNING! Stop it first: systemctl --user stop zimage"
|
||||
((ERRORS++))
|
||||
exit 1
|
||||
else
|
||||
pass "Z-Image-Turbo is not running"
|
||||
fi
|
||||
|
||||
# === STEP 4: rocminfo (safe, no TLB flush) ===
|
||||
((TESTS++))
|
||||
info "Step 4: Running rocminfo..."
|
||||
if rocminfo 2>&1 | grep -q "gfx10"; then
|
||||
pass "rocminfo detects GPU"
|
||||
else
|
||||
fail "rocminfo failed"
|
||||
((ERRORS++))
|
||||
fi
|
||||
|
||||
# Check dmesg AGAIN after rocminfo
|
||||
if dmesg 2>/dev/null | grep -q "timeout waiting for kiq fence"; then
|
||||
fail "KIQ error appeared after rocminfo!"
|
||||
((ERRORS++))
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo ""
|
||||
info "Step 4 complete. Checking dmesg for any BC-250 markers..."
|
||||
dmesg 2>/dev/null | grep "BC-250" 2>/dev/null || info "(no BC-250 markers yet)"
|
||||
echo ""
|
||||
|
||||
# === STEP 5: hip_probe (uses _exit(0), minimal cleanup) ===
|
||||
((TESTS++))
|
||||
info "Step 5: Running hip_probe (minimal GPU test)..."
|
||||
cd /home/dars/VibeROCm
|
||||
|
||||
if [ -f hip_probe ]; then
|
||||
timeout 30 ./hip_probe 2>&1
|
||||
RESULT=$?
|
||||
if [ $RESULT -eq 0 ]; then
|
||||
pass "hip_probe passed"
|
||||
else
|
||||
fail "hip_probe failed (exit code: $RESULT)"
|
||||
((ERRORS++))
|
||||
fi
|
||||
else
|
||||
warn "hip_probe binary not found, skipping"
|
||||
fi
|
||||
|
||||
# Check dmesg after hip_probe
|
||||
sleep 2
|
||||
if dmesg 2>/dev/null | grep -q "timeout waiting for kiq fence"; then
|
||||
fail "KIQ error appeared after hip_probe!"
|
||||
((ERRORS++))
|
||||
echo " → The KIQ bypass patch may not be working. STOP HERE."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
info "Checking BC-250 bypass markers after hip_probe..."
|
||||
dmesg 2>/dev/null | grep "BC-250" 2>/dev/null || info "(no markers)"
|
||||
echo ""
|
||||
|
||||
# === STEP 6: Wait and verify GPU is still alive ===
|
||||
info "Step 6: Waiting 10 seconds to check GPU stability..."
|
||||
sleep 10
|
||||
if dmesg 2>/dev/null | grep -q "timeout waiting for kiq fence\|GPU fault\|GPU hang"; then
|
||||
fail "GPU error detected during wait period!"
|
||||
((ERRORS++))
|
||||
exit 1
|
||||
else
|
||||
pass "GPU stable after 10-second wait"
|
||||
fi
|
||||
|
||||
# === STEP 7: hip_vector_add (full GPU compute test) ===
|
||||
((TESTS++))
|
||||
info "Step 7: Running hip_vector_add (full compute test)..."
|
||||
if [ -f hip_vector_add ]; then
|
||||
timeout 60 ./hip_vector_add 2>&1
|
||||
RESULT=$?
|
||||
if [ $RESULT -eq 0 ]; then
|
||||
pass "hip_vector_add passed"
|
||||
else
|
||||
fail "hip_vector_add failed (exit code: $RESULT)"
|
||||
((ERRORS++))
|
||||
fi
|
||||
else
|
||||
warn "hip_vector_add binary not found, skipping"
|
||||
fi
|
||||
|
||||
# Final dmesg check
|
||||
sleep 5
|
||||
if dmesg 2>/dev/null | grep -q "timeout waiting for kiq fence"; then
|
||||
fail "KIQ error appeared after hip_vector_add!"
|
||||
((ERRORS++))
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "==========================================="
|
||||
info "Final dmesg BC-250 markers:"
|
||||
dmesg 2>/dev/null | grep "BC-250" 2>/dev/null || info "(none)"
|
||||
echo ""
|
||||
|
||||
if [ $ERRORS -eq 0 ]; then
|
||||
echo -e "${GREEN}ALL TESTS PASSED${NC} ($TESTS tests)"
|
||||
echo ""
|
||||
echo "The KIQ bypass patch is working. The GPU survived compute workloads"
|
||||
echo "without hanging. You can now safely re-enable Z-Image-Turbo:"
|
||||
echo " systemctl --user enable --now zimage.service"
|
||||
else
|
||||
echo -e "${RED}$ERRORS ERRORS${NC} out of $TESTS tests"
|
||||
echo ""
|
||||
echo "The patch may need additional work. Check dmesg for details."
|
||||
fi
|
||||
Reference in New Issue
Block a user