Fabian/ROCm-Research-Archive
Archived
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
This repository has been archived on 2026-08-19. You can view files and clone it. You cannot open issues or pull requests or push a commit.
Files
d7d22e93b38544cd6de21a7bcf9730a824baaa68
ROCm-Research-Archive/_TestScripts/Danis ROCm Kernel Patch Research/Qwen3_4B_LlamaCpp_ROCm_Documentation.md
T
Fabian d7d22e93b3 Uploaded sanitized BC250/ROCm Repository.
2026-08-20 00:45:43 +02:00

12 KiB
Raw Blame History

Qwen3-4B on llama.cpp with ROCm (AMD BC-250)

Running Qwen3-4B-Q8_0 via llama.cpp with ROCm HIP GPU acceleration on the AMD BC-250 (gfx1010).

Table of Contents

System Overview

Component Value
GPU AMD BC-250 (Navi 10, gfx1010)
VRAM ~14.4 GiB (14750 MiB)
Wave Size 32
ROCm Version 7.2.0
HIP Version 7.2.26043-9999
HIP Compiler AMD clang 22.0.0git (ROCm LLVM)
OS CachyOS (Arch-based), Kernel 6.18.8-3-cachyos
CPU 12 threads
RAM 14 GiB system + 14 GiB swap
llama.cpp Build b8184 (commit 3191462)
CMake 4.2.3

Prerequisites

Before starting, ensure you have:

  1. ROCm installed and working — verify with:

    Bash
    rocm-smi
rocminfo | grep -E "Name:|gfx"

  2. Required packages:

    Bash
    # Arch/CachyOS
sudo pacman -S git cmake base-devel aria2

# Ubuntu/Debian
sudo apt install git cmake build-essential aria2

  3. ROCm development libraries (hipblas, rocblas):

    Bash
    # Verify they exist
ls /opt/rocm/lib/libhipblas.so
ls /opt/rocm/lib/librocblas.so
ls /opt/rocm/lib/llvm/bin/clang++

  4. Know your GPU architecture:

    Bash
    rocminfo | grep "Name:" | grep gfx
# Output: gfx1010 (for BC-250)

Step 1: Clone llama.cpp

Bash
cd ~
git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp

If already cloned, update:

Bash
cd ~/llama.cpp
git stash    # if you have local changes
git pull

Step 2: Build llama.cpp with ROCm HIP

Configure

Bash
cd ~/llama.cpp
rm -rf build
mkdir build && cd build

cmake .. \
  -DGGML_HIP=ON \
  -DAMDGPU_TARGETS="gfx1010" \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_HIP_COMPILER=/opt/rocm/lib/llvm/bin/clang++ \
  -G "Unix Makefiles"

Key flags explained:

Flag Purpose
-DGGML_HIP=ON Enable HIP/ROCm GPU backend
-DAMDGPU_TARGETS="gfx1010" Target GPU architecture (BC-250 = gfx1010)
-DCMAKE_HIP_COMPILER=/opt/rocm/lib/llvm/bin/clang++ Use ROCm's clang directly (required for CMake ≥ 4.x, hipcc wrapper is rejected)
-G "Unix Makefiles" Use Make instead of Ninja

Important (CMake 4.x): Do NOT use -DCMAKE_HIP_COMPILER=/opt/rocm/bin/hipcc — CMake 4.x explicitly rejects the hipcc wrapper. You must point to the clang++ binary inside ROCm's LLVM directory.

Verify Configuration

Bash
grep "GGML_HIP" CMakeCache.txt
# Should show: GGML_HIP:BOOL=ON

Build

Bash
make -j$(nproc)

Build time: HIP compilation is slow (~15-25 minutes on 12 threads). Each .cu template gets compiled to AMDGPU ISA for gfx1010. Be patient.

Verify Build Output

Bash
ls -lh build/bin/llama-cli build/bin/llama-server

# Verify HIP linkage
ldd build/bin/llama-cli | grep -i "hip\|rocm"

Expected output:

Text Only
libggml-hip.so.0 => .../libggml-hip.so.0
libhipblas.so.3 => /opt/rocm/lib/libhipblas.so.3
librocblas.so.5 => /opt/rocm/lib/librocblas.so.5
libamdhip64.so.7 => /opt/rocm/lib/libamdhip64.so.7
librocsolver.so.0 => /opt/rocm/lib/librocsolver.so.0
libhsa-runtime64.so.1 => /opt/rocm/lib/libhsa-runtime64.so.1

Step 3: Download the Model

Bash
mkdir -p ~/models

aria2c \
  -x 16 \
  -s 16 \
  -k 1M \
  -d ~/models \
  -o Qwen3-4B-Q8_0.gguf \
  "https://huggingface.co/Qwen/Qwen3-4B-GGUF/resolve/main/Qwen3-4B-Q8_0.gguf"
aria2 Flag Purpose
-x 16 16 connections per server
-s 16 Split into 16 segments
-k 1M Minimum split size 1MB

Using wget (Fallback)

Bash
wget -c \
  "https://huggingface.co/Qwen/Qwen3-4B-GGUF/resolve/main/Qwen3-4B-Q8_0.gguf" \
  -O ~/models/Qwen3-4B-Q8_0.gguf

Verify Download

Bash
ls -lh ~/models/Qwen3-4B-Q8_0.gguf
# Expected: ~4.0 GiB (4,280,404,704 bytes)

Model Source: Qwen/Qwen3-4B-GGUF on Hugging Face (official Qwen repo).

Step 4: Run Inference (Interactive Chat)

Bash
cd ~/llama.cpp/build/bin

./llama-cli \
  -m ~/models/Qwen3-4B-Q8_0.gguf \
  -ngl 99 \
  -c 4096 \
  --temp 0.6 \
  --top-k 20 \
  --top-p 0.95

Expected startup output:

Text Only
ggml_cuda_init: found 1 ROCm devices:
  Device 0: AMD BC-250, gfx1010:xnack- (0x1010), VMM: no, Wave Size: 32

build      : b213-3191462
model      : Qwen3-4B-Q8_0.gguf
modalities : text

You'll get an interactive > prompt. Type your question and press Enter.

In-chat commands: | Command | Action | |-------------|---------------------------------| | /exit | Exit the chat | | /clear | Clear chat history | | /regen | Regenerate last response | | /read | Load a text file into context | | Ctrl+C | Force exit |

Disable Thinking Mode

Qwen3-4B has a "thinking" mode enabled by default (responses start with [Start thinking]). To disable it and get direct answers:

Bash
./llama-cli \
  -m ~/models/Qwen3-4B-Q8_0.gguf \
  -ngl 99 \
  -c 4096 \
  --temp 0.7 \
  --top-k 20 \
  --top-p 0.8 \
  --jinja \
  --chat-template-file ~/llama.cpp/models/templates/qwen3.jinja \
  -e

Or append /no_think to your prompt for per-message control.

Step 5: Run Inference (One-Shot / Batch)

For scripting or single-prompt usage without interactive mode:

Bash
cd ~/llama.cpp/build/bin

./llama-cli \
  -m ~/models/Qwen3-4B-Q8_0.gguf \
  -ngl 99 \
  -p "Explain what ROCm is in 2 sentences." \
  -n 200 \
  --no-display-prompt \
  --no-conversation

Step 6: Run as API Server

llama.cpp includes an OpenAI-compatible HTTP API server:

Bash
cd ~/llama.cpp/build/bin

./llama-server \
  -m ~/models/Qwen3-4B-Q8_0.gguf \
  -ngl 99 \
  -c 4096 \
  --host 0.0.0.0 \
  --port 8080

Test with curl

Bash
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3-4b",
    "messages": [
      {"role": "user", "content": "What is ROCm?"}
    ],
    "max_tokens": 200,
    "temperature": 0.7
  }'

Web UI

Open http://localhost:8080 in a browser for the built-in chat UI.

Performance Results

Benchmarked on AMD BC-250 with full GPU offload (-ngl 99):

Metric Value
Prompt Processing ~84–267 t/s
Generation Speed ~55–57 t/s
Context Size 2048–4096
GPU Offload All 36 layers

Prompt processing speed varies by prompt length (shorter prompts = higher t/s due to overhead ratio). Generation speed is consistently ~56-57 tokens/second.

VRAM / Memory Breakdown

From llama_memory_breakdown_print at exit (context size 2048):

Location Total Free Model Context Compute
ROCm0 (BC-250) 14750 MiB 8382 MiB 4076 MiB 288 MiB 301 MiB
Host (CPU) — — 394 MiB 0 MiB 14 MiB
  • Model weights: ~4.0 GiB VRAM (matches the Q8_0 file size)
  • Remaining free VRAM: ~8.4 GiB (plenty of room for larger context windows)
  • Host RAM: ~394 MiB for metadata

With context size 4096, VRAM usage for context doubles to ~576 MiB, still well within the 14.4 GiB available.

Useful Parameters Reference

Parameter Default Description
-m — Path to GGUF model file
-ngl 99 0 Number of layers to offload to GPU (99 = all)
-c 4096 Context window size (in tokens)
-n -1 Max tokens to generate (-1 = unlimited)
-p — Initial prompt text
--temp 0.6 Sampling temperature (lower = more deterministic)
--top-k 20 Top-K sampling
--top-p 0.95 Top-P (nucleus) sampling
--no-display-prompt off Don't echo the prompt in output
--no-conversation off Exit after first response (no interactive loop)
-t auto Number of CPU threads
--host 127.0.0.1 Server bind address
--port 8080 Server port

Troubleshooting

CMake Error: "CMAKE_HIP_COMPILER is set to the hipcc wrapper"

Cause: CMake ≥ 4.x rejects the hipcc wrapper script.
Fix: Point to the ROCm clang directly:

Bash
-DCMAKE_HIP_COMPILER=/opt/rocm/lib/llvm/bin/clang++

Build Error: "No rule to make target 'libserver-context.a'"

Cause: Race condition from running multiple make processes in the same build directory simultaneously.
Fix: Kill all builds, rm -rf build, and rebuild from scratch with a single make -j$(nproc).

"GGML_HIP:BOOL=OFF" in CMakeCache

Cause: ROCm dev libraries not found during cmake configuration.
Fix: Ensure /opt/rocm/lib/libhipblas.so and /opt/rocm/lib/llvm/bin/clang++ exist. Re-run cmake.

Model file is 0 bytes after download

Cause: Incorrect URL (case-sensitive) — Hugging Face returns 404.
Fix: The correct filename is Qwen3-4B-Q8_0.gguf (capital Q, capital B). Full URL:

Text Only
https://huggingface.co/Qwen/Qwen3-4B-GGUF/resolve/main/Qwen3-4B-Q8_0.gguf

Slow GPU performance / "low-power state" warning

Text Only
WARNING: AMD GPU device(s) is/are in a low-power state
The BC-250 may throttle. Force performance mode:
Bash
sudo sh -c 'echo high > /sys/class/drm/card1/device/power_dpm_force_performance_level'

"Exception caught: map::at" in rocm-smi

Known BC-250 issue with rocm-smi power monitoring. Does not affect inference. Ignore safely.

Notes & Tips

  • Qwen3 Thinking Mode: By default, Qwen3 wraps responses in [Start thinking]...[End thinking] blocks showing its reasoning chain. This is a feature, not a bug. Use --jinja with the official template or /no_think to disable it.

  • Q8_0 Quantization: This is the highest quality GGUF quantization (8-bit). The 4B parameter model at Q8_0 uses ~4 GiB VRAM, leaving plenty of headroom on the BC-250's ~14.4 GiB.

  • Full GPU Offload: With -ngl 99, all 36 transformer layers are offloaded to the GPU. No CPU fallback needed for this model size.

  • Other Quant Options: Qwen also provides Q4_K_M (~2.5 GiB) and Q4_0 (~2.3 GiB) variants on the same Hugging Face repo if you want to save VRAM for larger context windows.

  • Multiple Models: The BC-250 has enough VRAM to potentially run larger models like Qwen3-8B at Q4_K_M quantization (~5 GiB).

Quick Reference

Bash
# Build (one-time)
cd ~/llama.cpp && rm -rf build && mkdir build && cd build
cmake .. -DGGML_HIP=ON -DAMDGPU_TARGETS="gfx1010" -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_HIP_COMPILER=/opt/rocm/lib/llvm/bin/clang++ -G "Unix Makefiles"
make -j$(nproc)

# Download model (one-time)
mkdir -p ~/models
aria2c -x 16 -s 16 -k 1M -d ~/models -o Qwen3-4B-Q8_0.gguf \
  "https://huggingface.co/Qwen/Qwen3-4B-GGUF/resolve/main/Qwen3-4B-Q8_0.gguf"

# Run interactive chat
~/llama.cpp/build/bin/llama-cli -m ~/models/Qwen3-4B-Q8_0.gguf -ngl 99 -c 4096

# Run API server
~/llama.cpp/build/bin/llama-server -m ~/models/Qwen3-4B-Q8_0.gguf -ngl 99 -c 4096 --host 0.0.0.0 --port 8080

Documentation generated on March 1, 2026. Based on llama.cpp build b8184, ROCm 7.2.0, AMD BC-250 (gfx1010).

Reference in New Issue View Git Blame Copy Permalink