ROCm-Research-Archive/Danis ROCm Kernel Patch Research/Qwen3_4B_LlamaCpp_ROCm_Documentation.md

# Qwen3-4B on llama.cpp with ROCm (AMD BC-250)

> Running **Qwen3-4B-Q8_0** via **llama.cpp** with **ROCm HIP** GPU acceleration on the **AMD BC-250 (gfx1010)**.

---

## Table of Contents

- [System Overview](#system-overview)
- [Prerequisites](#prerequisites)
- [Step 1: Clone llama.cpp](#step-1-clone-llamacpp)
- [Step 2: Build llama.cpp with ROCm HIP](#step-2-build-llamacpp-with-rocm-hip)
- [Step 3: Download the Model](#step-3-download-the-model)
- [Step 4: Run Inference (Interactive Chat)](#step-4-run-inference-interactive-chat)
- [Step 5: Run Inference (One-Shot / Batch)](#step-5-run-inference-one-shot--batch)
- [Step 6: Run as API Server](#step-6-run-as-api-server)
- [Performance Results](#performance-results)
- [VRAM / Memory Breakdown](#vram--memory-breakdown)
- [Useful Parameters Reference](#useful-parameters-reference)
- [Troubleshooting](#troubleshooting)
- [Notes & Tips](#notes--tips)

---

## System Overview

| Component        | Value                                         |
|------------------|-----------------------------------------------|
| **GPU**          | AMD BC-250 (Navi 10, gfx1010)                 |
| **VRAM**         | ~14.4 GiB (14750 MiB)                         |
| **Wave Size**    | 32                                            |
| **ROCm Version** | 7.2.0                                         |
| **HIP Version**  | 7.2.26043-9999                                |
| **HIP Compiler** | AMD clang 22.0.0git (ROCm LLVM)               |
| **OS**           | CachyOS (Arch-based), Kernel 6.18.8-3-cachyos |
| **CPU**          | 12 threads                                    |
| **RAM**          | 14 GiB system + 14 GiB swap                   |
| **llama.cpp**    | Build b8184 (commit `3191462`)                 |
| **CMake**        | 4.2.3                                         |

---

## Prerequisites

Before starting, ensure you have:

1. **ROCm installed and working** — verify with:
   ```bash
   rocm-smi
   rocminfo | grep -E "Name:|gfx"
   ```

2. **Required packages**:
   ```bash
   # Arch/CachyOS
   sudo pacman -S git cmake base-devel aria2

   # Ubuntu/Debian
   sudo apt install git cmake build-essential aria2
   ```

3. **ROCm development libraries** (hipblas, rocblas):
   ```bash
   # Verify they exist
   ls /opt/rocm/lib/libhipblas.so
   ls /opt/rocm/lib/librocblas.so
   ls /opt/rocm/lib/llvm/bin/clang++
   ```

4. **Know your GPU architecture**:
   ```bash
   rocminfo | grep "Name:" | grep gfx
   # Output: gfx1010 (for BC-250)
   ```

---

## Step 1: Clone llama.cpp

```bash
cd ~
git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
```

If already cloned, update:
```bash
cd ~/llama.cpp
git stash    # if you have local changes
git pull
```

---

## Step 2: Build llama.cpp with ROCm HIP

### Configure

```bash
cd ~/llama.cpp
rm -rf build
mkdir build && cd build

cmake .. \
  -DGGML_HIP=ON \
  -DAMDGPU_TARGETS="gfx1010" \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_HIP_COMPILER=/opt/rocm/lib/llvm/bin/clang++ \
  -G "Unix Makefiles"
```

**Key flags explained:**

| Flag | Purpose |
|------|---------|
| `-DGGML_HIP=ON` | Enable HIP/ROCm GPU backend |
| `-DAMDGPU_TARGETS="gfx1010"` | Target GPU architecture (BC-250 = gfx1010) |
| `-DCMAKE_HIP_COMPILER=/opt/rocm/lib/llvm/bin/clang++` | Use ROCm's clang directly (required for CMake ≥ 4.x, `hipcc` wrapper is rejected) |
| `-G "Unix Makefiles"` | Use Make instead of Ninja |

> **Important (CMake 4.x):** Do NOT use `-DCMAKE_HIP_COMPILER=/opt/rocm/bin/hipcc` — CMake 4.x explicitly rejects the hipcc wrapper. You must point to the clang++ binary inside ROCm's LLVM directory.

### Verify Configuration

```bash
grep "GGML_HIP" CMakeCache.txt
# Should show: GGML_HIP:BOOL=ON
```

### Build

```bash
make -j$(nproc)
```

> **Build time:** HIP compilation is slow (~15-25 minutes on 12 threads). Each `.cu` template gets compiled to AMDGPU ISA for gfx1010. Be patient.

### Verify Build Output

```bash
ls -lh build/bin/llama-cli build/bin/llama-server

# Verify HIP linkage
ldd build/bin/llama-cli | grep -i "hip\|rocm"
```

Expected output:
```
libggml-hip.so.0 => .../libggml-hip.so.0
libhipblas.so.3 => /opt/rocm/lib/libhipblas.so.3
librocblas.so.5 => /opt/rocm/lib/librocblas.so.5
libamdhip64.so.7 => /opt/rocm/lib/libamdhip64.so.7
librocsolver.so.0 => /opt/rocm/lib/librocsolver.so.0
libhsa-runtime64.so.1 => /opt/rocm/lib/libhsa-runtime64.so.1
```

---

## Step 3: Download the Model

### Using aria2 (Recommended — Maximum Speed)

```bash
mkdir -p ~/models

aria2c \
  -x 16 \
  -s 16 \
  -k 1M \
  -d ~/models \
  -o Qwen3-4B-Q8_0.gguf \
  "https://huggingface.co/Qwen/Qwen3-4B-GGUF/resolve/main/Qwen3-4B-Q8_0.gguf"
```

| aria2 Flag | Purpose |
|------------|---------|
| `-x 16`    | 16 connections per server |
| `-s 16`    | Split into 16 segments |
| `-k 1M`   | Minimum split size 1MB |

### Using wget (Fallback)

```bash
wget -c \
  "https://huggingface.co/Qwen/Qwen3-4B-GGUF/resolve/main/Qwen3-4B-Q8_0.gguf" \
  -O ~/models/Qwen3-4B-Q8_0.gguf
```

### Verify Download

```bash
ls -lh ~/models/Qwen3-4B-Q8_0.gguf
# Expected: ~4.0 GiB (4,280,404,704 bytes)
```

> **Model Source:** [Qwen/Qwen3-4B-GGUF](https://huggingface.co/Qwen/Qwen3-4B-GGUF) on Hugging Face (official Qwen repo).

---

## Step 4: Run Inference (Interactive Chat)

```bash
cd ~/llama.cpp/build/bin

./llama-cli \
  -m ~/models/Qwen3-4B-Q8_0.gguf \
  -ngl 99 \
  -c 4096 \
  --temp 0.6 \
  --top-k 20 \
  --top-p 0.95
```

**Expected startup output:**
```
ggml_cuda_init: found 1 ROCm devices:
  Device 0: AMD BC-250, gfx1010:xnack- (0x1010), VMM: no, Wave Size: 32

build      : b213-3191462
model      : Qwen3-4B-Q8_0.gguf
modalities : text
```

You'll get an interactive `>` prompt. Type your question and press Enter.

**In-chat commands:**
| Command     | Action                          |
|-------------|---------------------------------|
| `/exit`     | Exit the chat                   |
| `/clear`    | Clear chat history              |
| `/regen`    | Regenerate last response        |
| `/read`     | Load a text file into context   |
| `Ctrl+C`    | Force exit                      |

### Disable Thinking Mode

Qwen3-4B has a "thinking" mode enabled by default (responses start with `[Start thinking]`). To disable it and get direct answers:

```bash
./llama-cli \
  -m ~/models/Qwen3-4B-Q8_0.gguf \
  -ngl 99 \
  -c 4096 \
  --temp 0.7 \
  --top-k 20 \
  --top-p 0.8 \
  --jinja \
  --chat-template-file ~/llama.cpp/models/templates/qwen3.jinja \
  -e
```

Or append `/no_think` to your prompt for per-message control.

---

## Step 5: Run Inference (One-Shot / Batch)

For scripting or single-prompt usage without interactive mode:

```bash
cd ~/llama.cpp/build/bin

./llama-cli \
  -m ~/models/Qwen3-4B-Q8_0.gguf \
  -ngl 99 \
  -p "Explain what ROCm is in 2 sentences." \
  -n 200 \
  --no-display-prompt \
  --no-conversation
```

---

## Step 6: Run as API Server

llama.cpp includes an OpenAI-compatible HTTP API server:

```bash
cd ~/llama.cpp/build/bin

./llama-server \
  -m ~/models/Qwen3-4B-Q8_0.gguf \
  -ngl 99 \
  -c 4096 \
  --host 0.0.0.0 \
  --port 8080
```

### Test with curl

```bash
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3-4b",
    "messages": [
      {"role": "user", "content": "What is ROCm?"}
    ],
    "max_tokens": 200,
    "temperature": 0.7
  }'
```

### Web UI

Open `http://localhost:8080` in a browser for the built-in chat UI.

---

## Performance Results

Benchmarked on AMD BC-250 with full GPU offload (`-ngl 99`):

| Metric               | Value          |
|----------------------|----------------|
| **Prompt Processing** | ~84–267 t/s   |
| **Generation Speed**  | ~55–57 t/s    |
| **Context Size**      | 2048–4096      |
| **GPU Offload**       | All 36 layers  |

> Prompt processing speed varies by prompt length (shorter prompts = higher t/s due to overhead ratio). Generation speed is consistently **~56-57 tokens/second**.

---

## VRAM / Memory Breakdown

From `llama_memory_breakdown_print` at exit (context size 2048):

| Location         | Total   | Free   | Model  | Context | Compute |
|------------------|---------|--------|--------|---------|---------|
| **ROCm0 (BC-250)** | 14750 MiB | 8382 MiB | 4076 MiB | 288 MiB | 301 MiB |
| **Host (CPU)**     | —       | —      | 394 MiB | 0 MiB   | 14 MiB  |

- **Model weights**: ~4.0 GiB VRAM (matches the Q8_0 file size)
- **Remaining free VRAM**: ~8.4 GiB (plenty of room for larger context windows)
- **Host RAM**: ~394 MiB for metadata

With context size 4096, VRAM usage for context doubles to ~576 MiB, still well within the 14.4 GiB available.

---

## Useful Parameters Reference

| Parameter | Default | Description |
|-----------|---------|-------------|
| `-m` | — | Path to GGUF model file |
| `-ngl 99` | 0 | Number of layers to offload to GPU (99 = all) |
| `-c` | 4096 | Context window size (in tokens) |
| `-n` | -1 | Max tokens to generate (-1 = unlimited) |
| `-p` | — | Initial prompt text |
| `--temp` | 0.6 | Sampling temperature (lower = more deterministic) |
| `--top-k` | 20 | Top-K sampling |
| `--top-p` | 0.95 | Top-P (nucleus) sampling |
| `--no-display-prompt` | off | Don't echo the prompt in output |
| `--no-conversation` | off | Exit after first response (no interactive loop) |
| `-t` | auto | Number of CPU threads |
| `--host` | 127.0.0.1 | Server bind address |
| `--port` | 8080 | Server port |

---

## Troubleshooting

### CMake Error: "CMAKE_HIP_COMPILER is set to the hipcc wrapper"

**Cause:** CMake ≥ 4.x rejects the `hipcc` wrapper script.  
**Fix:** Point to the ROCm clang directly:
```bash
-DCMAKE_HIP_COMPILER=/opt/rocm/lib/llvm/bin/clang++
```

### Build Error: "No rule to make target 'libserver-context.a'"

**Cause:** Race condition from running multiple `make` processes in the same build directory simultaneously.  
**Fix:** Kill all builds, `rm -rf build`, and rebuild from scratch with a single `make -j$(nproc)`.

### "GGML_HIP:BOOL=OFF" in CMakeCache

**Cause:** ROCm dev libraries not found during cmake configuration.  
**Fix:** Ensure `/opt/rocm/lib/libhipblas.so` and `/opt/rocm/lib/llvm/bin/clang++` exist. Re-run cmake.

### Model file is 0 bytes after download

**Cause:** Incorrect URL (case-sensitive) — Hugging Face returns 404.  
**Fix:** The correct filename is `Qwen3-4B-Q8_0.gguf` (capital Q, capital B). Full URL:
```
https://huggingface.co/Qwen/Qwen3-4B-GGUF/resolve/main/Qwen3-4B-Q8_0.gguf
```

### Slow GPU performance / "low-power state" warning

```
WARNING: AMD GPU device(s) is/are in a low-power state
```
The BC-250 may throttle. Force performance mode:
```bash
sudo sh -c 'echo high > /sys/class/drm/card1/device/power_dpm_force_performance_level'
```

### "Exception caught: map::at" in rocm-smi

Known BC-250 issue with rocm-smi power monitoring. Does not affect inference. Ignore safely.

---

## Notes & Tips

- **Qwen3 Thinking Mode**: By default, Qwen3 wraps responses in `[Start thinking]...[End thinking]` blocks showing its reasoning chain. This is a feature, not a bug. Use `--jinja` with the official template or `/no_think` to disable it.

- **Q8_0 Quantization**: This is the highest quality GGUF quantization (8-bit). The 4B parameter model at Q8_0 uses ~4 GiB VRAM, leaving plenty of headroom on the BC-250's ~14.4 GiB.

- **Full GPU Offload**: With `-ngl 99`, all 36 transformer layers are offloaded to the GPU. No CPU fallback needed for this model size.

- **Other Quant Options**: Qwen also provides Q4_K_M (~2.5 GiB) and Q4_0 (~2.3 GiB) variants on the same Hugging Face repo if you want to save VRAM for larger context windows.

- **Multiple Models**: The BC-250 has enough VRAM to potentially run larger models like Qwen3-8B at Q4_K_M quantization (~5 GiB).

---

## Quick Reference

```bash
# Build (one-time)
cd ~/llama.cpp && rm -rf build && mkdir build && cd build
cmake .. -DGGML_HIP=ON -DAMDGPU_TARGETS="gfx1010" -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_HIP_COMPILER=/opt/rocm/lib/llvm/bin/clang++ -G "Unix Makefiles"
make -j$(nproc)

# Download model (one-time)
mkdir -p ~/models
aria2c -x 16 -s 16 -k 1M -d ~/models -o Qwen3-4B-Q8_0.gguf \
  "https://huggingface.co/Qwen/Qwen3-4B-GGUF/resolve/main/Qwen3-4B-Q8_0.gguf"

# Run interactive chat
~/llama.cpp/build/bin/llama-cli -m ~/models/Qwen3-4B-Q8_0.gguf -ngl 99 -c 4096

# Run API server
~/llama.cpp/build/bin/llama-server -m ~/models/Qwen3-4B-Q8_0.gguf -ngl 99 -c 4096 --host 0.0.0.0 --port 8080
```

---

*Documentation generated on March 1, 2026. Based on llama.cpp build b8184, ROCm 7.2.0, AMD BC-250 (gfx1010).*