ROCm-Research-Archive/README.md

!!!warning ""
    This was my try on implementing ROCm to the BC250. It is discontinued as of March 22th 2026 (Latest Commit).
    This Repository is a sanitized re-upload of my findings based off of [Dani's research](./_TestScripts/Danis%20ROCm%20Kernel%20Patch%20Research/) including the Patches that are now applyed to [project-ariel](https://github.com/cachenetics/project-ariel). This Repository is now a public Archive under the GPL-2.0 License and can be used as such.


# ROCm on AsRock BC-250 — Complete Installation Guide

**Hardware**: AsRock BC-250 (AMD Cyan Skillfish / gfx1013)  

**Target ROCm**: 7.2.0 | **OS**: CachyOS (Arch-based) | **Shell**: fish / bash

**Bootloader**: Limine | **Compiler**: Clang (LLVM)

**Author**: [Fabian](https://git.sudx.de/Fabian) | **Last Updated**: 2026-03-17

**Honorable Mention**: [Dani](https://git.sudx.de/Dani) (Creator of first KIQ Timeout fix)


# This is not working Standalone.

---

## Table of Contents

1. [Hardware Overview](#1-hardware-overview)
2. [Prerequisites](#2-prerequisites)
3. [Step 1 — Install ROCm Packages](#3-step-1--install-rocm-packages)
4. [Step 2 — User Group Configuration](#4-step-2--user-group-configuration)
5. [Step 3 — Environment Variables](#5-step-3--environment-variables)
6. [Step 4 — Kernel Boot Parameters](#6-step-4--kernel-boot-parameters)
7. [Step 5 — Modprobe Configuration](#7-step-5--modprobe-configuration)
8. [Step 6 — Build Patched amdgpu Kernel Module](#8-step-6--build-patched-amdgpu-kernel-module)
9. [Step 7 — Install Module & Reboot](#9-step-7--install-module--reboot)
10. [Step 8 — Post-Reboot Verification](#10-step-8--post-reboot-verification)
11. [Step 9 — HIP Compute Tests](#11-step-9--hip-compute-tests)
12. [Cyan Skillfish Governor (GPU Clock Management)](#12-cyan-skillfish-governor-gpu-clock-management)
13. [GPU Watchdog (Safety Monitor)](#13-gpu-watchdog-safety-monitor)
14. [HIP Programming Guidelines for BC-250](#14-hip-programming-guidelines-for-bc-250)
15. [Kernel Module Rebuild After Updates](#15-kernel-module-rebuild-after-updates)
16. [Technical Reference — Why This GPU Needs Patches](#16-technical-reference--why-this-gpu-needs-patches)
17. [Troubleshooting](#17-troubleshooting)
18. [File Inventory](#18-file-inventory)

---

## 1. Hardware Overview

```
GPU:              AMD BC-250 (Cyan Skillfish)
Device ID:        0x13FE (Vendor: 0x1002 AMD)
Architecture:     RDNA 1.5 (GFX 10.1.3 / gfx1013)
ROCm Target:      gfx10-1-generic (auto-mapped), runs as gfx1010 via override
Compute Units:    24 (reported by some tools as 12 due to SIMD config)
Wavefront Size:   32 (RDNA-style)
Memory:           ~14.75 GB shared system GDDR6 (NO dedicated VRAM)
Memory Type:      APU-style unified memory (system RAM shared with CPU)
PCIe:             Internal SoC fabric (Completion Timeout: Not Supported)
```

### Why This GPU Is Special

The BC-250 is a **cryptocurrency mining board repurposed as a compute accelerator**. It behaves
like an APU — there is **no dedicated VRAM**; all GPU memory operations use system RAM. This has
critical implications:

| Behavior | Explanation |
|----------|-------------|
| **No standard hipMalloc** | Standard device memory allocation targets non-existent VRAM |
| **hipMallocManaged required** | Unified memory works on shared RAM |
| **hipHostMalloc works** | Pinned host memory is safe for APU |
| **GPU reset = system crash** | Resetting the GPU corrupts shared system RAM |
| **SDMA engine unreliable** | Must disable via `HSA_ENABLE_SDMA=0` |
| **GFXOFF is fatal** | GPU enters power-save and can't wake — CPU hangs on MMIO read |
| **KIQ ring is broken** | KIQ commands cause fence timeouts → system hang |

---

## 2. Prerequisites

- CachyOS (or Arch-based distro) with a working internet connection
- `sudo` access for the installation user
- `paru` or another AUR helper (for the GPU governor)
- Sufficient disk space (~2 GB for kernel source + build artifacts)

Install base build dependencies:

```fish
sudo pacman -S --needed --noconfirm base-devel bc python
```

---

## 3. Step 1 — Install ROCm Packages

Install all 16 required ROCm 7.2.0 packages:

```fish
sudo pacman -S --needed --noconfirm \
  rocm-core hsa-rocr rocminfo rocm-smi-lib rocm-device-libs \
  rocm-llvm comgr hip-runtime-amd rocm-hip-runtime \
  rocm-opencl-runtime rocblas hipblas rocrand rocm-cmake \
  rocm-language-runtime hipblas-common
```

You can also run the provided script:
```fish
bash scripts/01_install_rocm_packages.sh
```

### Package List

| Package | Description |
|---------|-------------|
| `rocm-core` | ROCm core (version files) |
| `hsa-rocr` | HSA Runtime API |
| `rocminfo` | ROCm system info tool |
| `rocm-smi-lib` | ROCm SMI library |
| `rocm-device-libs` | ROCm device libraries |
| `rocm-llvm` | ROCm LLVM/Clang compiler (~4.5 GB) |
| `comgr` | AMDGPU Code Object Manager |
| `hip-runtime-amd` | HIP Runtime (AMD backend) |
| `rocm-hip-runtime` | Meta-package for HIP runtime |
| `rocm-opencl-runtime` | ROCm OpenCL runtime |
| `rocblas` | ROCm BLAS library |
| `hipblas` | ROCm BLAS marshalling library |
| `hipblas-common` | hipBLAS common files |
| `rocrand` | ROCm random number generator |
| `rocm-cmake` | ROCm CMake modules |
| `rocm-language-runtime` | ROCm language runtime meta |

---

## 4. Step 2 — User Group Configuration

Add your user to the `render` and `video` groups:

```fish
sudo usermod -aG render,video $USER
```

**You must log out and back in** (or reboot) for group changes to take effect.

Verify:
```fish
groups $USER
# Should include: render video
```

---

## 5. Step 3 — Environment Variables

The BC-250 requires specific environment variables for ROCm to function. With the v3 kernel
patches (Step 6), only a **minimal set** is needed.

### For fish shell

Add to `~/.config/fish/config.fish`:

```fish
# === ROCm Configuration for AMD BC-250 ===
set -gx PATH /opt/rocm/bin $PATH
set -gx LD_LIBRARY_PATH /opt/rocm/lib $LD_LIBRARY_PATH
set -gx ROCM_PATH /opt/rocm
set -gx HSA_OVERRIDE_GFX_VERSION 10.1.0
set -gx HIP_VISIBLE_DEVICES 0
set -gx HSA_ENABLE_SDMA 0
set -gx HSA_TOOLS_LIB ""
set -gx HSA_TOOLS_REPORT_LOAD_FAILURE 0
```

### For bash shell

Add to `~/.bashrc`:

```bash
# === ROCm Configuration for AMD BC-250 ===
export PATH="/opt/rocm/bin:$PATH"
export LD_LIBRARY_PATH="/opt/rocm/lib:$LD_LIBRARY_PATH"
export ROCM_PATH=/opt/rocm
export HSA_OVERRIDE_GFX_VERSION=10.1.0
export HIP_VISIBLE_DEVICES=0
export HSA_ENABLE_SDMA=0
export HSA_TOOLS_LIB=""
export HSA_TOOLS_REPORT_LOAD_FAILURE=0
```

You can also run the provided script:
```fish
bash scripts/02_configure_environment.sh
```

### Variable Reference

| Variable | Value | Why Required |
|----------|-------|--------------|
| `HSA_OVERRIDE_GFX_VERSION` | `10.1.0` | Maps gfx1013 → gfx1010 (closest supported target) |
| `HIP_VISIBLE_DEVICES` | `0` | Explicit GPU device selection |
| `HSA_ENABLE_SDMA` | `0` | SDMA engine is broken on gfx1013, use shader DMA |
| `HSA_TOOLS_LIB` | `""` | Prevent profiling tools from destabilizing GPU |
| `HSA_TOOLS_REPORT_LOAD_FAILURE` | `0` | Suppress tool load warnings |

### Variables You Must NOT Set

These were pre-v3 workarounds that **severely hurt performance**:

| Variable | Why It's Harmful |
|----------|------------------|
| `GPU_MAX_HW_QUEUES=1` | Serializes all GPU ops to 1 queue — severe slowdown |
| `HIP_LAUNCH_BLOCKING=1` | Forces synchronous kernel launches — prevents pipelining |
| `HSA_DISABLE_FRAGMENT_ALLOCATOR=1` | Not needed with v3 patches |

---

## 6. Step 4 — Kernel Boot Parameters

### Limine Bootloader

Edit `/etc/default/limine` and add these parameters to the `KERNEL_CMDLINE`:

```
amdgpu.gpu_recovery=1 amdgpu.noretry=0 amdgpu.dc=0 amdgpu.lockup_timeout=120000 amdgpu.ppfeaturemask=0xfff73ef7
```

Apply changes:
```fish
sudo limine-update
```

### GRUB Bootloader (if applicable)

Edit `/etc/default/grub` and add to `GRUB_CMDLINE_LINUX_DEFAULT`:

```
amdgpu.gpu_recovery=1 amdgpu.noretry=0 amdgpu.dc=0 amdgpu.lockup_timeout=120000 amdgpu.ppfeaturemask=0xfff73ef7
```

Apply changes:
```bash
sudo grub-mkconfig -o /boot/grub/grub.cfg
```

### Parameter Reference

| Parameter | Value | Purpose |
|-----------|-------|---------|
| `amdgpu.gpu_recovery=1` | Enabled | **CRITICAL**: Auto-recover from GPU hangs |
| `amdgpu.noretry=0` | Retry enabled | Allow page fault retry (required for shared memory) |
| `amdgpu.dc=0` | Display disabled | Disable display controller (headless — prevents IRQ errors) |
| `amdgpu.lockup_timeout=120000` | 120 seconds | Time before declaring GPU hung (allows heavy compute) |
| `amdgpu.ppfeaturemask=0xfff73ef7` | Custom mask | Disable GFXOFF + DeepSleep + ULV (see below) |

### ppfeaturemask Calculation

```
Default value:  0xfff7bfff
BC-250 value:   0xfff73ef7

Bits disabled:
  Bit 15 (0x8000) — PP_GFXOFF_MASK:          GPU enters unrecoverable power-save
  Bit  8 (0x0100) — PP_ULV_MASK:             Ultra-low voltage may destabilize GPU
  Bit  3 (0x0008) — PP_SCLK_DEEP_SLEEP_MASK: Deep clock sleep prevents GPU wake
```

---

## 7. Step 5 — Modprobe Configuration

Copy the provided config file:
```fish
sudo cp config/amdgpu.conf /etc/modprobe.d/amdgpu.conf
```

Or create manually — file `/etc/modprobe.d/amdgpu.conf`:

```
# AMD BC-250 (Cyan Skillfish / gfx1013) — ROCm Stability Parameters
# noretry=0              — Allow page fault retry (critical for shared memory / APU)
# gpu_recovery=1         — Enable GPU recovery on timeout
# sched_hw_submission=2  — Limit concurrent HW submissions (prevent queue overload)
# ppfeaturemask=0xfff73ef7 — Disable GFXOFF (bit 15), DeepSleep (bit 3), ULV (bit 8)
options amdgpu noretry=0 gpu_recovery=1 sched_hw_submission=2 ppfeaturemask=0xfff73ef7
```

---

## 8. Step 6 — Build Patched amdgpu Kernel Module

This is the most critical step. The BC-250 has a hardware/firmware bug where the KIQ (Kernel
Interface Queue) ring causes fatal GPU hangs. The v3 patch applies 8 surgical modifications
across 3 kernel source files to bypass KIQ and add dead-GPU detection.

### Why a Custom Module is Needed

The stock amdgpu driver routes TLB (Translation Lookaside Buffer) invalidation commands through
the KIQ ring. On the BC-250, this causes:

1. **KIQ fence timeouts** → GPU becomes unresponsive
2. **GFXOFF power-save** → GPU can't wake, CPU hangs on MMIO read (no PCIe timeout)
3. **System freeze** → No recovery possible, hard power-off required

The v3 patch has three protection layers:
- **Layer 1**: Disable GFXOFF for Cyan Skillfish (`gfx_v10_0.c`)
- **Layer 2**: KIQ bypass + dead-GPU detection (`gmc_v10_0.c`, `amdgpu_gmc.c`)
- **Layer 3**: Boot parameters disable GFXOFF/DeepSleep/ULV (Step 4)

### CRITICAL: Use CachyOS Kernel Source (Not Vanilla!)

**You MUST build from the CachyOS kernel source**, not vanilla kernel.org source. CachyOS patches
their kernel with struct layout changes that are incompatible with vanilla source headers. Building
from vanilla kernel source will produce a module that causes a **NULL pointer dereference** at boot
in `drm_vma_offset_add` during `gmc_v10_0_sw_init`.

### Build Process

#### Option A: Automated (Recommended)

The provided build script handles everything:

```fish
# Edit scripts/04_build_patched_module.sh and verify your kernel version first!
bash scripts/04_build_patched_module.sh
```

#### Option B: Manual Step-by-Step

##### 1. Identify Your Kernel Version

```fish
uname -r
# Example output: 6.19.6-2-cachyos
```

Note both the **kernel version** (e.g., `6.19.6`) and the **full string** (e.g., `6.19.6-2-cachyos`).

##### 2. Find the Matching CachyOS Source

Go to https://github.com/CachyOS/linux/releases and find the release matching your kernel.

For example, kernel `6.19.6-2-cachyos` → release `cachyos-6.19.6-1` or similar.

```fish
mkdir -p ~/kernel-build
cd ~/kernel-build

# Download (replace URL with your matching version)
curl -L -o cachyos-source.tar.gz \
  "https://github.com/CachyOS/linux/archive/refs/tags/YOUR_TAG.tar.gz"

# Extract
tar xf cachyos-source.tar.gz
cd linux-*  # or whatever the extracted directory is named
```

##### 3. Prepare Build Environment

```fish
set KVER (uname -r)

# Copy the running kernel's config and build files
cp /usr/lib/modules/$KVER/build/.config .
cp /usr/lib/modules/$KVER/build/Module.symvers .

# Copy localversion files (these set the kernel version string)
cp /usr/lib/modules/$KVER/build/localversion.* . 2>/dev/null

# Prepare the build system
make LLVM=1 olddefconfig
make LLVM=1 modules_prepare
```

> **Note**: `LLVM=1` is **mandatory** — CachyOS kernels are compiled with Clang, not GCC.
> Using GCC will produce an incompatible module.

##### 4. Apply the Patches

Copy the three patch scripts from this guide's `patches/` directory to the BC-250, then run them:

```fish
set AMDGPU_DIR (pwd)/drivers/gpu/drm/amd/amdgpu

python3 patches/patch1_gfxoff.py $AMDGPU_DIR
python3 patches/patch2_gmc.py $AMDGPU_DIR
python3 patches/patch3_amdgpu_gmc.py $AMDGPU_DIR
```

Verify the patches were applied:
```fish
grep -c "BC-250" $AMDGPU_DIR/gfx_v10_0.c    # Should be ≥ 4
grep -c "BC-250" $AMDGPU_DIR/gmc_v10_0.c     # Should be ≥ 14
grep -c "BC-250" $AMDGPU_DIR/amdgpu_gmc.c    # Should be ≥ 9
```

##### 5. Build the Module

```fish
make LLVM=1 -j(nproc) M=drivers/gpu/drm/amd/amdgpu modules
```

This takes 3-10 minutes depending on CPU. The output should end with:
```
LD [M]  drivers/gpu/drm/amd/amdgpu/amdgpu.ko
```

##### 6. Strip and Compress

```fish
# Strip debug symbols: ~600MB → ~30MB
strip --strip-debug drivers/gpu/drm/amd/amdgpu/amdgpu.ko

# Compress: ~30MB → ~4.5MB
zstd -19 -f drivers/gpu/drm/amd/amdgpu/amdgpu.ko
```

---

## 9. Step 7 — Install Module & Reboot

##### 1. Backup the Original Module

```fish
set KVER (uname -r)
set MODULE_DIR /usr/lib/modules/$KVER/kernel/drivers/gpu/drm/amd/amdgpu

sudo cp $MODULE_DIR/amdgpu.ko.zst $MODULE_DIR/amdgpu.ko.zst.original
```

##### 2. Install the Patched Module

```fish
sudo cp drivers/gpu/drm/amd/amdgpu/amdgpu.ko.zst $MODULE_DIR/amdgpu.ko.zst
sudo depmod -a
```

##### 3. Rebuild Initramfs

For Limine:
```fish
sudo limine-update
```

For GRUB/mkinitcpio:
```fish
sudo mkinitcpio -P
```

##### 4. Verify Before Reboot

```fish
# Check the installed module has all 9 BC-250 strings
zstd -d -c $MODULE_DIR/amdgpu.ko.zst | strings | grep "BC-250"
```

Expected output (9 strings):
```
BC-250: GFXOFF disabled to prevent GPU power-state hangs
BC-250: GPU unreachable (MMIO returned 0xFFFFFFFF)...
BC-250: GPU died during sem acquire (0xFFFFFFFF)
BC-250: GPU died during TLB flush ACK wait (0xFFFFFFFF)
BC-250: GPU unreachable in fw_reg_write_reg_wait...
BC-250: GPU died during reg_write_reg_wait (0xFFFFFFFF)
BC-250 KIQ bypass active
BC-250 KIQ bypass active in fw_reg_write_reg_wait
BC-250: MMIO reg write/wait timeout
```

##### 5. Reboot

```fish
sudo reboot
```

##### Emergency Rollback

If the system doesn't boot or the GPU crashes, restore the original module:

```fish
set KVER (uname -r)
set MODULE_DIR /usr/lib/modules/$KVER/kernel/drivers/gpu/drm/amd/amdgpu

sudo cp $MODULE_DIR/amdgpu.ko.zst.original $MODULE_DIR/amdgpu.ko.zst
sudo depmod -a
sudo limine-update  # or: sudo mkinitcpio -P
sudo reboot
```

---

## 10. Step 8 — Post-Reboot Verification

After reboot, run the verification script:

```fish
bash scripts/06_verify_installation.sh
```

Or check manually:

### Check dmesg for v3 Module

```fish
sudo dmesg | grep -E "BC-250|GFXOFF|out-of-tree"
```

Expected:
```
amdgpu: loading out-of-tree module taints kernel.
amdgpu 0000:01:00.0: amdgpu: BC-250: GFXOFF disabled to prevent GPU power-state hangs
```

### Check for Errors

```fish
# KIQ timeout errors (should be 0)
sudo dmesg | grep -ci "timeout waiting for kiq fence"

# GPU unreachable events (should be 0)
sudo dmesg | grep -ci "GPU unreachable\|GPU died"

# NULL pointer / BUG (should be 0)
sudo dmesg | grep -ci "BUG\|NULL pointer"
```

### Check Device Nodes

```fish
ls /dev/dri/
# Expected: card0  renderD128  (and by-path/)

ls /dev/kfd
# Expected: /dev/kfd (character device)
```

### Check ppfeaturemask

```fish
cat /sys/module/amdgpu/parameters/ppfeaturemask
# Expected: 0xfff73ef7
```

### Check rocminfo

```fish
rocminfo | grep -E "Name:|Marketing|gfx|Done"
```

Expected output includes:
```
  Name:                    gfx1010
  Marketing Name:          AMD BC-250
*** Done ***
```

---

## 11. Step 9 — HIP Compute Tests

Compile and run the provided HIP test programs:

### Compile Tests

```fish
cd tests/
hipcc --offload-arch=gfx1010 -o hip_probe hip_probe.cpp
hipcc --offload-arch=gfx1010 -o hip_minimal_test hip_minimal_test.cpp
hipcc --offload-arch=gfx1010 -o hip_vector_add hip_vector_add.cpp
```

### Run Tests (in order)

```fish
# Test 1: Device probe (no compute)
./hip_probe

# Test 2: Minimal kernel compute (32 values)
./hip_minimal_test

# Test 3: Full vector add with managed memory (65536 elements, sin²+cos²=1.0)
./hip_vector_add
```

### Sequential Stress Test

The critical test — on unpatched kernels, the **second** consecutive HIP run crashes the system:

```fish
for i in (seq 1 5)
    echo "=== Round $i ==="
    ./hip_minimal_test
    sleep 2
end
```

All 5 rounds should pass. If any round hangs or crashes, the module patch may not be applied
correctly.

You can also run the comprehensive test script:

```fish
bash scripts/07_run_hip_tests.sh
```

---

## 12. Cyan Skillfish Governor (GPU Clock Management)

The BC-250's GPU clock/voltage management is handled by the `cyan-skillfish-governor` service.
The kernel patches handle GFXOFF/power-state prevention; the governor handles DPM clock scaling.

### Installation

```fish
paru -S cyan-skillfish-governor
```

### Enable & Start

```fish
sudo systemctl enable --now cyan-skillfish-governor
```

### Verify

```fish
systemctl is-active cyan-skillfish-governor
# Expected: active

cat /sys/class/drm/card0/device/pp_dpm_sclk
# Expected: Shows clock levels with governor managing transitions
```

### Safe Operating Points

| Clock | Voltage | Use Case |
|-------|---------|----------|
| 1000 MHz | 700 mV | Idle |
| 1500 MHz | 900 mV | Light load |
| 2000 MHz | 1000 mV | Compute (max recommended) |

---

## 13. GPU Watchdog (Safety Monitor)

A safety watchdog script monitors kernel logs for KIQ timeouts and auto-kills GPU processes
before the timeout cascade crashes the system. The window to act is ~10 seconds after first timeout.

### Usage

```fish
# Monitor and auto-kill on KIQ timeout
bash scripts/gpu_watchdog.sh

# Monitor only (no kill)
bash scripts/gpu_watchdog.sh --dry-run

# Kill after 2 KIQ timeouts instead of 1
bash scripts/gpu_watchdog.sh --max-kiq 2
```

### Running as a Background Monitor

```fish
nohup bash scripts/gpu_watchdog.sh > /tmp/gpu_watchdog.log 2>&1 &
```

---

## 14. HIP Programming Guidelines for BC-250

### Memory Allocation

| Function | Safe? | Notes |
|----------|-------|-------|
| `hipMallocManaged()` | **YES** | Preferred — unified memory for APU |
| `hipHostMalloc()` | **YES** | Pinned host memory — safe |
| `hipHostMallocCoherent` | **YES** | Cache-coherent host memory |
| `hipMalloc()` | **CAUTION** | Works with v3 patches but may be slower |
| `hipMemcpy()` | **CAUTION** | Works with v3 patches |

### Critical Rules

1. **Never call `hipDeviceReset()`** — triggers KIQ queue teardown, crashes the system
2. **Use `_exit(0)` instead of `return 0`** in HIP programs — bypasses C++ static destructors
   that trigger HIP runtime cleanup, which can cause KIQ fence timeouts
3. **Never call `rocm-smi` repeatedly** — GPU management queries can destabilize the device
4. **Run a single long-lived daemon** for GPU workloads when possible — each process startup/exit
   cycles TLB flush paths

### Example: Safe HIP Program Exit

```cpp
#include <hip/hip_runtime.h>
#include <unistd.h>  // _exit()

int main() {
    // ... your HIP code ...

    // CRITICAL: Use _exit() to avoid HIP runtime destructor crash
    printf("Done\n");
    fflush(stdout);
    _exit(0);  // Bypasses C++ destructors and HIP cleanup
}
```

---

## 15. Kernel Module Rebuild After Updates

**Any CachyOS kernel update will replace your patched module.** After updating the kernel,
you must rebuild and reinstall the patched module.

### Quick Rebuild Steps

```fish
set OLD_KVER "6.19.6-2-cachyos"    # Previous kernel
set NEW_KVER (uname -r)             # New kernel after update

# If kernel version changed, download new CachyOS source and repeat Step 6
# If only pkgrel changed (e.g., -2-cachyos → -3-cachyos), you may be able to
# just copy the module and rebuild initramfs:

set MODULE_DIR /usr/lib/modules/$NEW_KVER/kernel/drivers/gpu/drm/amd/amdgpu
sudo cp /usr/lib/modules/$OLD_KVER/kernel/drivers/gpu/drm/amd/amdgpu/amdgpu.ko.zst $MODULE_DIR/
sudo depmod -a
sudo limine-update
```

> **Note**: This copy approach only works if the kernel ABI hasn't changed between versions.
> If the module fails to load, you must do a full rebuild from the new kernel source.

---

## 16. Technical Reference — Why This GPU Needs Patches

### The KIQ Problem

KIQ (Kernel Interface Queue) is a privileged ring buffer used by the amdgpu driver for TLB
invalidation and GPU management. On the BC-250, KIQ commands cause **fence timeouts** — the GPU
never acknowledges the command, and the CPU waits indefinitely.

There are exactly **4 code paths** that use the KIQ ring at runtime:

| # | Function | File | Patched? |
|---|----------|------|----------|
| 1 | `gmc_v10_0_flush_gpu_tlb()` | gmc_v10_0.c | **YES** — goto use_mmio |
| 2 | `amdgpu_gmc_flush_gpu_tlb_pasid()` | amdgpu_gmc.c | **YES** — direct callout |
| 3 | `amdgpu_gmc_fw_reg_write_reg_wait()` | amdgpu_gmc.c | **YES** — MMIO write+poll |
| 4 | `amdgpu_gfx_enable/disable_kcq()` | amdgpu_gfx.c | No (boot only, works fine) |

### The GFXOFF Problem

After a HIP process exits, the GPU enters GFXOFF (a power-saving state). When the next process
starts and triggers a TLB flush, the GPU is unresponsive. MMIO reads via `readl()` inside a
spinlock-protected loop hang the CPU indefinitely because the BC-250's internal PCIe fabric has
**no completion timeout** (`DevCap2: Completion Timeout: Not Supported`).

### The v3 Three-Layer Solution

| Layer | Mechanism | Purpose |
|-------|-----------|---------|
| 1 — GFXOFF Disable | `gfx_v10_0.c`: `PP_GFXOFF_MASK` cleared for IP 10.1.3 | Prevents GPU from entering unrecoverable power-save |
| 2 — KIQ Bypass + Dead-GPU | `gmc_v10_0.c` + `amdgpu_gmc.c`: MMIO instead of KIQ, 0xFFFFFFFF checks | Avoids KIQ ring entirely; detects dead GPU before hanging |
| 3 — Boot Parameters | `ppfeaturemask=0xfff73ef7` | Belt-and-suspenders: disables GFXOFF+DeepSleep+ULV at power management level |

### Patch Summary (8 modifications across 3 files)

| # | File | Function | Type | Purpose |
|---|------|----------|------|---------|
| 1 | `gfx_v10_0.c` | `gfx_v10_0_check_gfxoff_flag` | v3 | Disable GFXOFF for IP 10.1.3 |
| 2 | `gmc_v10_0.c` | `gmc_v10_0_flush_gpu_tlb` | v2 | KIQ bypass → `goto use_mmio` |
| 3 | `gmc_v10_0.c` | `gmc_v10_0_flush_gpu_tlb` | v3 | Pre-spinlock 0xFFFFFFFF health check |
| 4 | `gmc_v10_0.c` | `gmc_v10_0_flush_gpu_tlb` | v3 | Sem acquire loop dead-GPU bail |
| 5 | `gmc_v10_0.c` | `gmc_v10_0_flush_gpu_tlb` | v3 | ACK-wait loop dead-GPU bail |
| 6 | `gmc_v10_0.c` | `gmc_v10_0_hw_init` | v2 | `flush_pasid_uses_kiq = false` |
| 7 | `amdgpu_gmc.c` | `amdgpu_gmc_flush_gpu_tlb_pasid` | v2 | Direct MMIO TLB flush |
| 8 | `amdgpu_gmc.c` | `amdgpu_gmc_fw_reg_write_reg_wait` | v2+v3 | MMIO write+poll + dead-GPU detection |

### Why Mining OS (Kernel 5.10) Worked Without Patches

In kernel 5.10 + AMDGPU-PRO 22.20, TLB flush was simpler and more localized. The centralized
`amdgpu_gmc_flush_gpu_tlb_pasid()` in `amdgpu_gmc.c` and the deferred work
`amdgpu_vm_tlb_fence_work()` didn't exist yet. KIQ failures returned `-ETIME` gracefully instead
of cascading to GPU death. Modern kernels (6.x) refactored TLB flushing into a shared centralized
path that defaults to KIQ for all hardware — which breaks BC-250.

---

## 17. Troubleshooting

### System freezes after HIP program runs

**Cause**: v3 module not loaded, or initramfs not rebuilt after installation.

**Fix**: Boot from recovery, restore original module, verify initramfs was rebuilt:
```fish
sudo cp /usr/lib/modules/(uname -r)/kernel/drivers/gpu/drm/amd/amdgpu/amdgpu.ko.zst.original \
       /usr/lib/modules/(uname -r)/kernel/drivers/gpu/drm/amd/amdgpu/amdgpu.ko.zst
sudo depmod -a
sudo limine-update  # MUST rebuild initramfs!
sudo reboot
```

### NULL pointer dereference at boot (drm_vma_offset_add)

**Cause**: Module was built from vanilla kernel.org source instead of CachyOS source.

**Fix**: Rebuild from CachyOS-specific kernel source (see Step 6).

### `rocminfo` shows no GPU / "No HSA GPU agents found"

**Cause**: Environment variables not set, or `/dev/kfd` doesn't exist.

**Fix**:
```fish
# Check env vars
echo $HSA_OVERRIDE_GFX_VERSION  # Should be 10.1.0

# Check /dev/kfd exists
ls -la /dev/kfd

# Check user groups
groups  # Should include 'render'
```

### "KIQ timeout" messages in dmesg

**Cause**: Stock (unpatched) module is loaded instead of v3.

**Fix**: Verify the v3 module is installed:
```fish
zstd -d -c /usr/lib/modules/(uname -r)/kernel/drivers/gpu/drm/amd/amdgpu/amdgpu.ko.zst \
  | strings | grep "BC-250"
# Should show 9 BC-250 strings
```

### Module version mismatch after kernel update

**Cause**: CachyOS updated the kernel, replacing the patched module.

**Fix**: Rebuild the module from matching CachyOS source (see Section 15).

---

## 18. File Inventory

```
BC250_ROCm_Guide/
├── README.md                          ← This guide
├── config/
│   └── amdgpu.conf                   ← Modprobe configuration
├── patches/
│   ├── patch1_gfxoff.py              ← Layer 1: Disable GFXOFF
│   ├── patch2_gmc.py                 ← Layer 2: KIQ bypass + dead-GPU (gmc_v10_0.c)
│   └── patch3_amdgpu_gmc.py          ← Layer 2: KIQ bypass + dead-GPU (amdgpu_gmc.c)
├── scripts/
│   ├── 01_install_rocm_packages.sh   ← Package installation
│   ├── 02_configure_environment.sh   ← Environment variables + modprobe + boot params
│   ├── 04_build_patched_module.sh    ← Download source, patch, build module
│   ├── 05_install_module.sh          ← Install module + rebuild initramfs
│   ├── 06_verify_installation.sh     ← Post-reboot verification
│   ├── 07_run_hip_tests.sh           ← Compile and run HIP tests
│   └── gpu_watchdog.sh               ← KIQ timeout safety monitor
└── tests/
    ├── hip_probe.cpp                 ← Device diagnostic (6 steps)
    ├── hip_minimal_test.cpp          ← Minimal kernel compute test
    └── hip_vector_add.cpp            ← Full managed-memory vector add test
```

---

*Based on research and patches by Dani (2026-02-22), adapted and verified on CachyOS kernel
6.19.6-2-cachyos with ROCm 7.2.0.*