Uploaded sanitized BC250/ROCm Repository.

2026-08-20 00:45:43 +02:00
parent 7d2184f1e8
commit d7d22e93b3
678 changed files with 65963 additions and 1 deletions
@@ -0,0 +1,443 @@
+# Qwen3-4B on llama.cpp with ROCm (AMD BC-250)
+
+> Running **Qwen3-4B-Q8_0** via **llama.cpp** with **ROCm HIP** GPU acceleration on the **AMD BC-250 (gfx1010)**.
+
+---
+
+## Table of Contents
+
+- [System Overview](#system-overview)
+- [Prerequisites](#prerequisites)
+- [Step 1: Clone llama.cpp](#step-1-clone-llamacpp)
+- [Step 2: Build llama.cpp with ROCm HIP](#step-2-build-llamacpp-with-rocm-hip)
+- [Step 3: Download the Model](#step-3-download-the-model)
+- [Step 4: Run Inference (Interactive Chat)](#step-4-run-inference-interactive-chat)
+- [Step 5: Run Inference (One-Shot / Batch)](#step-5-run-inference-one-shot--batch)
+- [Step 6: Run as API Server](#step-6-run-as-api-server)
+- [Performance Results](#performance-results)
+- [VRAM / Memory Breakdown](#vram--memory-breakdown)
+- [Useful Parameters Reference](#useful-parameters-reference)
+- [Troubleshooting](#troubleshooting)
+- [Notes & Tips](#notes--tips)
+
+---
+
+## System Overview
+
+| Component        | Value                                         |
+|------------------|-----------------------------------------------|
+| **GPU**          | AMD BC-250 (Navi 10, gfx1010)                 |
+| **VRAM**         | ~14.4 GiB (14750 MiB)                         |
+| **Wave Size**    | 32                                            |
+| **ROCm Version** | 7.2.0                                         |
+| **HIP Version**  | 7.2.26043-9999                                |
+| **HIP Compiler** | AMD clang 22.0.0git (ROCm LLVM)               |
+| **OS**           | CachyOS (Arch-based), Kernel 6.18.8-3-cachyos |
+| **CPU**          | 12 threads                                    |
+| **RAM**          | 14 GiB system + 14 GiB swap                   |
+| **llama.cpp**    | Build b8184 (commit `3191462`)                 |
+| **CMake**        | 4.2.3                                         |
+
+---
+
+## Prerequisites
+
+Before starting, ensure you have:
+
+1. **ROCm installed and working** — verify with:
+   ```bash
+   rocm-smi
+   rocminfo | grep -E "Name:|gfx"
+   ```
+
+2. **Required packages**:
+   ```bash
+   # Arch/CachyOS
+   sudo pacman -S git cmake base-devel aria2
+
+   # Ubuntu/Debian
+   sudo apt install git cmake build-essential aria2
+   ```
+
+3. **ROCm development libraries** (hipblas, rocblas):
+   ```bash
+   # Verify they exist
+   ls /opt/rocm/lib/libhipblas.so
+   ls /opt/rocm/lib/librocblas.so
+   ls /opt/rocm/lib/llvm/bin/clang++
+   ```
+
+4. **Know your GPU architecture**:
+   ```bash
+   rocminfo | grep "Name:" | grep gfx
+   # Output: gfx1010 (for BC-250)
+   ```
+
+---
+
+## Step 1: Clone llama.cpp
+
+```bash
+cd ~
+git clone https://github.com/ggml-org/llama.cpp.git
+cd llama.cpp
+```
+
+If already cloned, update:
+```bash
+cd ~/llama.cpp
+git stash    # if you have local changes
+git pull
+```
+
+---
+
+## Step 2: Build llama.cpp with ROCm HIP
+
+### Configure
+
+```bash
+cd ~/llama.cpp
+rm -rf build
+mkdir build && cd build
+
+cmake .. \
+  -DGGML_HIP=ON \
+  -DAMDGPU_TARGETS="gfx1010" \
+  -DCMAKE_BUILD_TYPE=Release \
+  -DCMAKE_HIP_COMPILER=/opt/rocm/lib/llvm/bin/clang++ \
+  -G "Unix Makefiles"
+```
+
+**Key flags explained:**
+
+| Flag | Purpose |
+|------|---------|
+| `-DGGML_HIP=ON` | Enable HIP/ROCm GPU backend |
+| `-DAMDGPU_TARGETS="gfx1010"` | Target GPU architecture (BC-250 = gfx1010) |
+| `-DCMAKE_HIP_COMPILER=/opt/rocm/lib/llvm/bin/clang++` | Use ROCm's clang directly (required for CMake ≥ 4.x, `hipcc` wrapper is rejected) |
+| `-G "Unix Makefiles"` | Use Make instead of Ninja |
+
+> **Important (CMake 4.x):** Do NOT use `-DCMAKE_HIP_COMPILER=/opt/rocm/bin/hipcc` — CMake 4.x explicitly rejects the hipcc wrapper. You must point to the clang++ binary inside ROCm's LLVM directory.
+
+### Verify Configuration
+
+```bash
+grep "GGML_HIP" CMakeCache.txt
+# Should show: GGML_HIP:BOOL=ON
+```
+
+### Build
+
+```bash
+make -j$(nproc)
+```
+
+> **Build time:** HIP compilation is slow (~15-25 minutes on 12 threads). Each `.cu` template gets compiled to AMDGPU ISA for gfx1010. Be patient.
+
+### Verify Build Output
+
+```bash
+ls -lh build/bin/llama-cli build/bin/llama-server
+
+# Verify HIP linkage
+ldd build/bin/llama-cli | grep -i "hip\|rocm"
+```
+
+Expected output:
+```
+libggml-hip.so.0 => .../libggml-hip.so.0
+libhipblas.so.3 => /opt/rocm/lib/libhipblas.so.3
+librocblas.so.5 => /opt/rocm/lib/librocblas.so.5
+libamdhip64.so.7 => /opt/rocm/lib/libamdhip64.so.7
+librocsolver.so.0 => /opt/rocm/lib/librocsolver.so.0
+libhsa-runtime64.so.1 => /opt/rocm/lib/libhsa-runtime64.so.1
+```
+
+---
+
+## Step 3: Download the Model
+
+### Using aria2 (Recommended — Maximum Speed)
+
+```bash
+mkdir -p ~/models
+
+aria2c \
+  -x 16 \
+  -s 16 \
+  -k 1M \
+  -d ~/models \
+  -o Qwen3-4B-Q8_0.gguf \
+  "https://huggingface.co/Qwen/Qwen3-4B-GGUF/resolve/main/Qwen3-4B-Q8_0.gguf"
+```
+
+| aria2 Flag | Purpose |
+|------------|---------|
+| `-x 16`    | 16 connections per server |
+| `-s 16`    | Split into 16 segments |
+| `-k 1M`   | Minimum split size 1MB |
+
+### Using wget (Fallback)
+
+```bash
+wget -c \
+  "https://huggingface.co/Qwen/Qwen3-4B-GGUF/resolve/main/Qwen3-4B-Q8_0.gguf" \
+  -O ~/models/Qwen3-4B-Q8_0.gguf
+```
+
+### Verify Download
+
+```bash
+ls -lh ~/models/Qwen3-4B-Q8_0.gguf
+# Expected: ~4.0 GiB (4,280,404,704 bytes)
+```
+
+> **Model Source:** [Qwen/Qwen3-4B-GGUF](https://huggingface.co/Qwen/Qwen3-4B-GGUF) on Hugging Face (official Qwen repo).
+
+---
+
+## Step 4: Run Inference (Interactive Chat)
+
+```bash
+cd ~/llama.cpp/build/bin
+
+./llama-cli \
+  -m ~/models/Qwen3-4B-Q8_0.gguf \
+  -ngl 99 \
+  -c 4096 \
+  --temp 0.6 \
+  --top-k 20 \
+  --top-p 0.95
+```
+
+**Expected startup output:**
+```
+ggml_cuda_init: found 1 ROCm devices:
+  Device 0: AMD BC-250, gfx1010:xnack- (0x1010), VMM: no, Wave Size: 32
+
+build      : b213-3191462
+model      : Qwen3-4B-Q8_0.gguf
+modalities : text
+```
+
+You'll get an interactive `>` prompt. Type your question and press Enter.
+
+**In-chat commands:**
+| Command     | Action                          |
+|-------------|---------------------------------|
+| `/exit`     | Exit the chat                   |
+| `/clear`    | Clear chat history              |
+| `/regen`    | Regenerate last response        |
+| `/read`     | Load a text file into context   |
+| `Ctrl+C`    | Force exit                      |
+
+### Disable Thinking Mode
+
+Qwen3-4B has a "thinking" mode enabled by default (responses start with `[Start thinking]`). To disable it and get direct answers:
+
+```bash
+./llama-cli \
+  -m ~/models/Qwen3-4B-Q8_0.gguf \
+  -ngl 99 \
+  -c 4096 \
+  --temp 0.7 \
+  --top-k 20 \
+  --top-p 0.8 \
+  --jinja \
+  --chat-template-file ~/llama.cpp/models/templates/qwen3.jinja \
+  -e
+```
+
+Or append `/no_think` to your prompt for per-message control.
+
+---
+
+## Step 5: Run Inference (One-Shot / Batch)
+
+For scripting or single-prompt usage without interactive mode:
+
+```bash
+cd ~/llama.cpp/build/bin
+
+./llama-cli \
+  -m ~/models/Qwen3-4B-Q8_0.gguf \
+  -ngl 99 \
+  -p "Explain what ROCm is in 2 sentences." \
+  -n 200 \
+  --no-display-prompt \
+  --no-conversation
+```
+
+---
+
+## Step 6: Run as API Server
+
+llama.cpp includes an OpenAI-compatible HTTP API server:
+
+```bash
+cd ~/llama.cpp/build/bin
+
+./llama-server \
+  -m ~/models/Qwen3-4B-Q8_0.gguf \
+  -ngl 99 \
+  -c 4096 \
+  --host 0.0.0.0 \
+  --port 8080
+```
+
+### Test with curl
+
+```bash
+curl http://localhost:8080/v1/chat/completions \
+  -H "Content-Type: application/json" \
+  -d '{
+    "model": "qwen3-4b",
+    "messages": [
+      {"role": "user", "content": "What is ROCm?"}
+    ],
+    "max_tokens": 200,
+    "temperature": 0.7
+  }'
+```
+
+### Web UI
+
+Open `http://localhost:8080` in a browser for the built-in chat UI.
+
+---
+
+## Performance Results
+
+Benchmarked on AMD BC-250 with full GPU offload (`-ngl 99`):
+
+| Metric               | Value          |
+|----------------------|----------------|
+| **Prompt Processing** | ~84–267 t/s   |
+| **Generation Speed**  | ~55–57 t/s    |
+| **Context Size**      | 2048–4096      |
+| **GPU Offload**       | All 36 layers  |
+
+> Prompt processing speed varies by prompt length (shorter prompts = higher t/s due to overhead ratio). Generation speed is consistently **~56-57 tokens/second**.
+
+---
+
+## VRAM / Memory Breakdown
+
+From `llama_memory_breakdown_print` at exit (context size 2048):
+
+| Location         | Total   | Free   | Model  | Context | Compute |
+|------------------|---------|--------|--------|---------|---------|
+| **ROCm0 (BC-250)** | 14750 MiB | 8382 MiB | 4076 MiB | 288 MiB | 301 MiB |
+| **Host (CPU)**     | —       | —      | 394 MiB | 0 MiB   | 14 MiB  |
+
+- **Model weights**: ~4.0 GiB VRAM (matches the Q8_0 file size)
+- **Remaining free VRAM**: ~8.4 GiB (plenty of room for larger context windows)
+- **Host RAM**: ~394 MiB for metadata
+
+With context size 4096, VRAM usage for context doubles to ~576 MiB, still well within the 14.4 GiB available.
+
+---
+
+## Useful Parameters Reference
+
+| Parameter | Default | Description |
+|-----------|---------|-------------|
+| `-m` | — | Path to GGUF model file |
+| `-ngl 99` | 0 | Number of layers to offload to GPU (99 = all) |
+| `-c` | 4096 | Context window size (in tokens) |
+| `-n` | -1 | Max tokens to generate (-1 = unlimited) |
+| `-p` | — | Initial prompt text |
+| `--temp` | 0.6 | Sampling temperature (lower = more deterministic) |
+| `--top-k` | 20 | Top-K sampling |
+| `--top-p` | 0.95 | Top-P (nucleus) sampling |
+| `--no-display-prompt` | off | Don't echo the prompt in output |
+| `--no-conversation` | off | Exit after first response (no interactive loop) |
+| `-t` | auto | Number of CPU threads |
+| `--host` | 127.0.0.1 | Server bind address |
+| `--port` | 8080 | Server port |
+
+---
+
+## Troubleshooting
+
+### CMake Error: "CMAKE_HIP_COMPILER is set to the hipcc wrapper"
+
+**Cause:** CMake ≥ 4.x rejects the `hipcc` wrapper script.  
+**Fix:** Point to the ROCm clang directly:
+```bash
+-DCMAKE_HIP_COMPILER=/opt/rocm/lib/llvm/bin/clang++
+```
+
+### Build Error: "No rule to make target 'libserver-context.a'"
+
+**Cause:** Race condition from running multiple `make` processes in the same build directory simultaneously.  
+**Fix:** Kill all builds, `rm -rf build`, and rebuild from scratch with a single `make -j$(nproc)`.
+
+### "GGML_HIP:BOOL=OFF" in CMakeCache
+
+**Cause:** ROCm dev libraries not found during cmake configuration.  
+**Fix:** Ensure `/opt/rocm/lib/libhipblas.so` and `/opt/rocm/lib/llvm/bin/clang++` exist. Re-run cmake.
+
+### Model file is 0 bytes after download
+
+**Cause:** Incorrect URL (case-sensitive) — Hugging Face returns 404.  
+**Fix:** The correct filename is `Qwen3-4B-Q8_0.gguf` (capital Q, capital B). Full URL:
+```
+https://huggingface.co/Qwen/Qwen3-4B-GGUF/resolve/main/Qwen3-4B-Q8_0.gguf
+```
+
+### Slow GPU performance / "low-power state" warning
+
+```
+WARNING: AMD GPU device(s) is/are in a low-power state
+```
+The BC-250 may throttle. Force performance mode:
+```bash
+sudo sh -c 'echo high > /sys/class/drm/card1/device/power_dpm_force_performance_level'
+```
+
+### "Exception caught: map::at" in rocm-smi
+
+Known BC-250 issue with rocm-smi power monitoring. Does not affect inference. Ignore safely.
+
+---
+
+## Notes & Tips
+
+- **Qwen3 Thinking Mode**: By default, Qwen3 wraps responses in `[Start thinking]...[End thinking]` blocks showing its reasoning chain. This is a feature, not a bug. Use `--jinja` with the official template or `/no_think` to disable it.
+
+- **Q8_0 Quantization**: This is the highest quality GGUF quantization (8-bit). The 4B parameter model at Q8_0 uses ~4 GiB VRAM, leaving plenty of headroom on the BC-250's ~14.4 GiB.
+
+- **Full GPU Offload**: With `-ngl 99`, all 36 transformer layers are offloaded to the GPU. No CPU fallback needed for this model size.
+
+- **Other Quant Options**: Qwen also provides Q4_K_M (~2.5 GiB) and Q4_0 (~2.3 GiB) variants on the same Hugging Face repo if you want to save VRAM for larger context windows.
+
+- **Multiple Models**: The BC-250 has enough VRAM to potentially run larger models like Qwen3-8B at Q4_K_M quantization (~5 GiB).
+
+---
+
+## Quick Reference
+
+```bash
+# Build (one-time)
+cd ~/llama.cpp && rm -rf build && mkdir build && cd build
+cmake .. -DGGML_HIP=ON -DAMDGPU_TARGETS="gfx1010" -DCMAKE_BUILD_TYPE=Release \
+  -DCMAKE_HIP_COMPILER=/opt/rocm/lib/llvm/bin/clang++ -G "Unix Makefiles"
+make -j$(nproc)
+
+# Download model (one-time)
+mkdir -p ~/models
+aria2c -x 16 -s 16 -k 1M -d ~/models -o Qwen3-4B-Q8_0.gguf \
+  "https://huggingface.co/Qwen/Qwen3-4B-GGUF/resolve/main/Qwen3-4B-Q8_0.gguf"
+
+# Run interactive chat
+~/llama.cpp/build/bin/llama-cli -m ~/models/Qwen3-4B-Q8_0.gguf -ngl 99 -c 4096
+
+# Run API server
+~/llama.cpp/build/bin/llama-server -m ~/models/Qwen3-4B-Q8_0.gguf -ngl 99 -c 4096 --host 0.0.0.0 --port 8080
+```
+
+---
+
+*Documentation generated on March 1, 2026. Based on llama.cpp build b8184, ROCm 7.2.0, AMD BC-250 (gfx1010).*