Uploaded sanitized BC250/ROCm Repository.
This commit is contained in:
Fabian
@@ -1,2 +1,872 @@
|
||||
# ROCm-Research-Archive
|
||||
!!!warning ""
|
||||
This was my try on implementing ROCm to the BC250. It is discontinued as of March 22th 2026 (Latest Commit).
|
||||
This Repository is a sanitized re-upload of my findings based off of [Dani's research](./_TestScripts/Danis%20ROCm%20Kernel%20Patch%20Research/) including the Patches that are now applyed to [project-ariel](https://github.com/cachenetics/project-ariel). This Repository is now a public Archive under the GPL-2.0 License and can be used as such.
|
||||
|
||||
|
||||
# ROCm on AsRock BC-250 — Complete Installation Guide
|
||||
|
||||
**Hardware**: AsRock BC-250 (AMD Cyan Skillfish / gfx1013)
|
||||
|
||||
**Target ROCm**: 7.2.0 | **OS**: CachyOS (Arch-based) | **Shell**: fish / bash
|
||||
|
||||
**Bootloader**: Limine | **Compiler**: Clang (LLVM)
|
||||
|
||||
**Author**: [Fabian](https://git.sudx.de/Fabian) | **Last Updated**: 2026-03-17
|
||||
|
||||
**Honorable Mention**: [Dani](https://git.sudx.de/Dani) (Creator of first KIQ Timeout fix)
|
||||
|
||||
|
||||
# This is not working Standalone.
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. [Hardware Overview](#1-hardware-overview)
|
||||
2. [Prerequisites](#2-prerequisites)
|
||||
3. [Step 1 — Install ROCm Packages](#3-step-1--install-rocm-packages)
|
||||
4. [Step 2 — User Group Configuration](#4-step-2--user-group-configuration)
|
||||
5. [Step 3 — Environment Variables](#5-step-3--environment-variables)
|
||||
6. [Step 4 — Kernel Boot Parameters](#6-step-4--kernel-boot-parameters)
|
||||
7. [Step 5 — Modprobe Configuration](#7-step-5--modprobe-configuration)
|
||||
8. [Step 6 — Build Patched amdgpu Kernel Module](#8-step-6--build-patched-amdgpu-kernel-module)
|
||||
9. [Step 7 — Install Module & Reboot](#9-step-7--install-module--reboot)
|
||||
10. [Step 8 — Post-Reboot Verification](#10-step-8--post-reboot-verification)
|
||||
11. [Step 9 — HIP Compute Tests](#11-step-9--hip-compute-tests)
|
||||
12. [Cyan Skillfish Governor (GPU Clock Management)](#12-cyan-skillfish-governor-gpu-clock-management)
|
||||
13. [GPU Watchdog (Safety Monitor)](#13-gpu-watchdog-safety-monitor)
|
||||
14. [HIP Programming Guidelines for BC-250](#14-hip-programming-guidelines-for-bc-250)
|
||||
15. [Kernel Module Rebuild After Updates](#15-kernel-module-rebuild-after-updates)
|
||||
16. [Technical Reference — Why This GPU Needs Patches](#16-technical-reference--why-this-gpu-needs-patches)
|
||||
17. [Troubleshooting](#17-troubleshooting)
|
||||
18. [File Inventory](#18-file-inventory)
|
||||
|
||||
---
|
||||
|
||||
## 1. Hardware Overview
|
||||
|
||||
```
|
||||
GPU: AMD BC-250 (Cyan Skillfish)
|
||||
Device ID: 0x13FE (Vendor: 0x1002 AMD)
|
||||
Architecture: RDNA 1.5 (GFX 10.1.3 / gfx1013)
|
||||
ROCm Target: gfx10-1-generic (auto-mapped), runs as gfx1010 via override
|
||||
Compute Units: 24 (reported by some tools as 12 due to SIMD config)
|
||||
Wavefront Size: 32 (RDNA-style)
|
||||
Memory: ~14.75 GB shared system GDDR6 (NO dedicated VRAM)
|
||||
Memory Type: APU-style unified memory (system RAM shared with CPU)
|
||||
PCIe: Internal SoC fabric (Completion Timeout: Not Supported)
|
||||
```
|
||||
|
||||
### Why This GPU Is Special
|
||||
|
||||
The BC-250 is a **cryptocurrency mining board repurposed as a compute accelerator**. It behaves
|
||||
like an APU — there is **no dedicated VRAM**; all GPU memory operations use system RAM. This has
|
||||
critical implications:
|
||||
|
||||
| Behavior | Explanation |
|
||||
|----------|-------------|
|
||||
| **No standard hipMalloc** | Standard device memory allocation targets non-existent VRAM |
|
||||
| **hipMallocManaged required** | Unified memory works on shared RAM |
|
||||
| **hipHostMalloc works** | Pinned host memory is safe for APU |
|
||||
| **GPU reset = system crash** | Resetting the GPU corrupts shared system RAM |
|
||||
| **SDMA engine unreliable** | Must disable via `HSA_ENABLE_SDMA=0` |
|
||||
| **GFXOFF is fatal** | GPU enters power-save and can't wake — CPU hangs on MMIO read |
|
||||
| **KIQ ring is broken** | KIQ commands cause fence timeouts → system hang |
|
||||
|
||||
---
|
||||
|
||||
## 2. Prerequisites
|
||||
|
||||
- CachyOS (or Arch-based distro) with a working internet connection
|
||||
- `sudo` access for the installation user
|
||||
- `paru` or another AUR helper (for the GPU governor)
|
||||
- Sufficient disk space (~2 GB for kernel source + build artifacts)
|
||||
|
||||
Install base build dependencies:
|
||||
|
||||
```fish
|
||||
sudo pacman -S --needed --noconfirm base-devel bc python
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. Step 1 — Install ROCm Packages
|
||||
|
||||
Install all 16 required ROCm 7.2.0 packages:
|
||||
|
||||
```fish
|
||||
sudo pacman -S --needed --noconfirm \
|
||||
rocm-core hsa-rocr rocminfo rocm-smi-lib rocm-device-libs \
|
||||
rocm-llvm comgr hip-runtime-amd rocm-hip-runtime \
|
||||
rocm-opencl-runtime rocblas hipblas rocrand rocm-cmake \
|
||||
rocm-language-runtime hipblas-common
|
||||
```
|
||||
|
||||
You can also run the provided script:
|
||||
```fish
|
||||
bash scripts/01_install_rocm_packages.sh
|
||||
```
|
||||
|
||||
### Package List
|
||||
|
||||
| Package | Description |
|
||||
|---------|-------------|
|
||||
| `rocm-core` | ROCm core (version files) |
|
||||
| `hsa-rocr` | HSA Runtime API |
|
||||
| `rocminfo` | ROCm system info tool |
|
||||
| `rocm-smi-lib` | ROCm SMI library |
|
||||
| `rocm-device-libs` | ROCm device libraries |
|
||||
| `rocm-llvm` | ROCm LLVM/Clang compiler (~4.5 GB) |
|
||||
| `comgr` | AMDGPU Code Object Manager |
|
||||
| `hip-runtime-amd` | HIP Runtime (AMD backend) |
|
||||
| `rocm-hip-runtime` | Meta-package for HIP runtime |
|
||||
| `rocm-opencl-runtime` | ROCm OpenCL runtime |
|
||||
| `rocblas` | ROCm BLAS library |
|
||||
| `hipblas` | ROCm BLAS marshalling library |
|
||||
| `hipblas-common` | hipBLAS common files |
|
||||
| `rocrand` | ROCm random number generator |
|
||||
| `rocm-cmake` | ROCm CMake modules |
|
||||
| `rocm-language-runtime` | ROCm language runtime meta |
|
||||
|
||||
---
|
||||
|
||||
## 4. Step 2 — User Group Configuration
|
||||
|
||||
Add your user to the `render` and `video` groups:
|
||||
|
||||
```fish
|
||||
sudo usermod -aG render,video $USER
|
||||
```
|
||||
|
||||
**You must log out and back in** (or reboot) for group changes to take effect.
|
||||
|
||||
Verify:
|
||||
```fish
|
||||
groups $USER
|
||||
# Should include: render video
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Step 3 — Environment Variables
|
||||
|
||||
The BC-250 requires specific environment variables for ROCm to function. With the v3 kernel
|
||||
patches (Step 6), only a **minimal set** is needed.
|
||||
|
||||
### For fish shell
|
||||
|
||||
Add to `~/.config/fish/config.fish`:
|
||||
|
||||
```fish
|
||||
# === ROCm Configuration for AMD BC-250 ===
|
||||
set -gx PATH /opt/rocm/bin $PATH
|
||||
set -gx LD_LIBRARY_PATH /opt/rocm/lib $LD_LIBRARY_PATH
|
||||
set -gx ROCM_PATH /opt/rocm
|
||||
set -gx HSA_OVERRIDE_GFX_VERSION 10.1.0
|
||||
set -gx HIP_VISIBLE_DEVICES 0
|
||||
set -gx HSA_ENABLE_SDMA 0
|
||||
set -gx HSA_TOOLS_LIB ""
|
||||
set -gx HSA_TOOLS_REPORT_LOAD_FAILURE 0
|
||||
```
|
||||
|
||||
### For bash shell
|
||||
|
||||
Add to `~/.bashrc`:
|
||||
|
||||
```bash
|
||||
# === ROCm Configuration for AMD BC-250 ===
|
||||
export PATH="/opt/rocm/bin:$PATH"
|
||||
export LD_LIBRARY_PATH="/opt/rocm/lib:$LD_LIBRARY_PATH"
|
||||
export ROCM_PATH=/opt/rocm
|
||||
export HSA_OVERRIDE_GFX_VERSION=10.1.0
|
||||
export HIP_VISIBLE_DEVICES=0
|
||||
export HSA_ENABLE_SDMA=0
|
||||
export HSA_TOOLS_LIB=""
|
||||
export HSA_TOOLS_REPORT_LOAD_FAILURE=0
|
||||
```
|
||||
|
||||
You can also run the provided script:
|
||||
```fish
|
||||
bash scripts/02_configure_environment.sh
|
||||
```
|
||||
|
||||
### Variable Reference
|
||||
|
||||
| Variable | Value | Why Required |
|
||||
|----------|-------|--------------|
|
||||
| `HSA_OVERRIDE_GFX_VERSION` | `10.1.0` | Maps gfx1013 → gfx1010 (closest supported target) |
|
||||
| `HIP_VISIBLE_DEVICES` | `0` | Explicit GPU device selection |
|
||||
| `HSA_ENABLE_SDMA` | `0` | SDMA engine is broken on gfx1013, use shader DMA |
|
||||
| `HSA_TOOLS_LIB` | `""` | Prevent profiling tools from destabilizing GPU |
|
||||
| `HSA_TOOLS_REPORT_LOAD_FAILURE` | `0` | Suppress tool load warnings |
|
||||
|
||||
### Variables You Must NOT Set
|
||||
|
||||
These were pre-v3 workarounds that **severely hurt performance**:
|
||||
|
||||
| Variable | Why It's Harmful |
|
||||
|----------|------------------|
|
||||
| `GPU_MAX_HW_QUEUES=1` | Serializes all GPU ops to 1 queue — severe slowdown |
|
||||
| `HIP_LAUNCH_BLOCKING=1` | Forces synchronous kernel launches — prevents pipelining |
|
||||
| `HSA_DISABLE_FRAGMENT_ALLOCATOR=1` | Not needed with v3 patches |
|
||||
|
||||
---
|
||||
|
||||
## 6. Step 4 — Kernel Boot Parameters
|
||||
|
||||
### Limine Bootloader
|
||||
|
||||
Edit `/etc/default/limine` and add these parameters to the `KERNEL_CMDLINE`:
|
||||
|
||||
```
|
||||
amdgpu.gpu_recovery=1 amdgpu.noretry=0 amdgpu.dc=0 amdgpu.lockup_timeout=120000 amdgpu.ppfeaturemask=0xfff73ef7
|
||||
```
|
||||
|
||||
Apply changes:
|
||||
```fish
|
||||
sudo limine-update
|
||||
```
|
||||
|
||||
### GRUB Bootloader (if applicable)
|
||||
|
||||
Edit `/etc/default/grub` and add to `GRUB_CMDLINE_LINUX_DEFAULT`:
|
||||
|
||||
```
|
||||
amdgpu.gpu_recovery=1 amdgpu.noretry=0 amdgpu.dc=0 amdgpu.lockup_timeout=120000 amdgpu.ppfeaturemask=0xfff73ef7
|
||||
```
|
||||
|
||||
Apply changes:
|
||||
```bash
|
||||
sudo grub-mkconfig -o /boot/grub/grub.cfg
|
||||
```
|
||||
|
||||
### Parameter Reference
|
||||
|
||||
| Parameter | Value | Purpose |
|
||||
|-----------|-------|---------|
|
||||
| `amdgpu.gpu_recovery=1` | Enabled | **CRITICAL**: Auto-recover from GPU hangs |
|
||||
| `amdgpu.noretry=0` | Retry enabled | Allow page fault retry (required for shared memory) |
|
||||
| `amdgpu.dc=0` | Display disabled | Disable display controller (headless — prevents IRQ errors) |
|
||||
| `amdgpu.lockup_timeout=120000` | 120 seconds | Time before declaring GPU hung (allows heavy compute) |
|
||||
| `amdgpu.ppfeaturemask=0xfff73ef7` | Custom mask | Disable GFXOFF + DeepSleep + ULV (see below) |
|
||||
|
||||
### ppfeaturemask Calculation
|
||||
|
||||
```
|
||||
Default value: 0xfff7bfff
|
||||
BC-250 value: 0xfff73ef7
|
||||
|
||||
Bits disabled:
|
||||
Bit 15 (0x8000) — PP_GFXOFF_MASK: GPU enters unrecoverable power-save
|
||||
Bit 8 (0x0100) — PP_ULV_MASK: Ultra-low voltage may destabilize GPU
|
||||
Bit 3 (0x0008) — PP_SCLK_DEEP_SLEEP_MASK: Deep clock sleep prevents GPU wake
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 7. Step 5 — Modprobe Configuration
|
||||
|
||||
Copy the provided config file:
|
||||
```fish
|
||||
sudo cp config/amdgpu.conf /etc/modprobe.d/amdgpu.conf
|
||||
```
|
||||
|
||||
Or create manually — file `/etc/modprobe.d/amdgpu.conf`:
|
||||
|
||||
```
|
||||
# AMD BC-250 (Cyan Skillfish / gfx1013) — ROCm Stability Parameters
|
||||
# noretry=0 — Allow page fault retry (critical for shared memory / APU)
|
||||
# gpu_recovery=1 — Enable GPU recovery on timeout
|
||||
# sched_hw_submission=2 — Limit concurrent HW submissions (prevent queue overload)
|
||||
# ppfeaturemask=0xfff73ef7 — Disable GFXOFF (bit 15), DeepSleep (bit 3), ULV (bit 8)
|
||||
options amdgpu noretry=0 gpu_recovery=1 sched_hw_submission=2 ppfeaturemask=0xfff73ef7
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 8. Step 6 — Build Patched amdgpu Kernel Module
|
||||
|
||||
This is the most critical step. The BC-250 has a hardware/firmware bug where the KIQ (Kernel
|
||||
Interface Queue) ring causes fatal GPU hangs. The v3 patch applies 8 surgical modifications
|
||||
across 3 kernel source files to bypass KIQ and add dead-GPU detection.
|
||||
|
||||
### Why a Custom Module is Needed
|
||||
|
||||
The stock amdgpu driver routes TLB (Translation Lookaside Buffer) invalidation commands through
|
||||
the KIQ ring. On the BC-250, this causes:
|
||||
|
||||
1. **KIQ fence timeouts** → GPU becomes unresponsive
|
||||
2. **GFXOFF power-save** → GPU can't wake, CPU hangs on MMIO read (no PCIe timeout)
|
||||
3. **System freeze** → No recovery possible, hard power-off required
|
||||
|
||||
The v3 patch has three protection layers:
|
||||
- **Layer 1**: Disable GFXOFF for Cyan Skillfish (`gfx_v10_0.c`)
|
||||
- **Layer 2**: KIQ bypass + dead-GPU detection (`gmc_v10_0.c`, `amdgpu_gmc.c`)
|
||||
- **Layer 3**: Boot parameters disable GFXOFF/DeepSleep/ULV (Step 4)
|
||||
|
||||
### CRITICAL: Use CachyOS Kernel Source (Not Vanilla!)
|
||||
|
||||
**You MUST build from the CachyOS kernel source**, not vanilla kernel.org source. CachyOS patches
|
||||
their kernel with struct layout changes that are incompatible with vanilla source headers. Building
|
||||
from vanilla kernel source will produce a module that causes a **NULL pointer dereference** at boot
|
||||
in `drm_vma_offset_add` during `gmc_v10_0_sw_init`.
|
||||
|
||||
### Build Process
|
||||
|
||||
#### Option A: Automated (Recommended)
|
||||
|
||||
The provided build script handles everything:
|
||||
|
||||
```fish
|
||||
# Edit scripts/04_build_patched_module.sh and verify your kernel version first!
|
||||
bash scripts/04_build_patched_module.sh
|
||||
```
|
||||
|
||||
#### Option B: Manual Step-by-Step
|
||||
|
||||
##### 1. Identify Your Kernel Version
|
||||
|
||||
```fish
|
||||
uname -r
|
||||
# Example output: 6.19.6-2-cachyos
|
||||
```
|
||||
|
||||
Note both the **kernel version** (e.g., `6.19.6`) and the **full string** (e.g., `6.19.6-2-cachyos`).
|
||||
|
||||
##### 2. Find the Matching CachyOS Source
|
||||
|
||||
Go to https://github.com/CachyOS/linux/releases and find the release matching your kernel.
|
||||
|
||||
For example, kernel `6.19.6-2-cachyos` → release `cachyos-6.19.6-1` or similar.
|
||||
|
||||
```fish
|
||||
mkdir -p ~/kernel-build
|
||||
cd ~/kernel-build
|
||||
|
||||
# Download (replace URL with your matching version)
|
||||
curl -L -o cachyos-source.tar.gz \
|
||||
"https://github.com/CachyOS/linux/archive/refs/tags/YOUR_TAG.tar.gz"
|
||||
|
||||
# Extract
|
||||
tar xf cachyos-source.tar.gz
|
||||
cd linux-* # or whatever the extracted directory is named
|
||||
```
|
||||
|
||||
##### 3. Prepare Build Environment
|
||||
|
||||
```fish
|
||||
set KVER (uname -r)
|
||||
|
||||
# Copy the running kernel's config and build files
|
||||
cp /usr/lib/modules/$KVER/build/.config .
|
||||
cp /usr/lib/modules/$KVER/build/Module.symvers .
|
||||
|
||||
# Copy localversion files (these set the kernel version string)
|
||||
cp /usr/lib/modules/$KVER/build/localversion.* . 2>/dev/null
|
||||
|
||||
# Prepare the build system
|
||||
make LLVM=1 olddefconfig
|
||||
make LLVM=1 modules_prepare
|
||||
```
|
||||
|
||||
> **Note**: `LLVM=1` is **mandatory** — CachyOS kernels are compiled with Clang, not GCC.
|
||||
> Using GCC will produce an incompatible module.
|
||||
|
||||
##### 4. Apply the Patches
|
||||
|
||||
Copy the three patch scripts from this guide's `patches/` directory to the BC-250, then run them:
|
||||
|
||||
```fish
|
||||
set AMDGPU_DIR (pwd)/drivers/gpu/drm/amd/amdgpu
|
||||
|
||||
python3 patches/patch1_gfxoff.py $AMDGPU_DIR
|
||||
python3 patches/patch2_gmc.py $AMDGPU_DIR
|
||||
python3 patches/patch3_amdgpu_gmc.py $AMDGPU_DIR
|
||||
```
|
||||
|
||||
Verify the patches were applied:
|
||||
```fish
|
||||
grep -c "BC-250" $AMDGPU_DIR/gfx_v10_0.c # Should be ≥ 4
|
||||
grep -c "BC-250" $AMDGPU_DIR/gmc_v10_0.c # Should be ≥ 14
|
||||
grep -c "BC-250" $AMDGPU_DIR/amdgpu_gmc.c # Should be ≥ 9
|
||||
```
|
||||
|
||||
##### 5. Build the Module
|
||||
|
||||
```fish
|
||||
make LLVM=1 -j(nproc) M=drivers/gpu/drm/amd/amdgpu modules
|
||||
```
|
||||
|
||||
This takes 3-10 minutes depending on CPU. The output should end with:
|
||||
```
|
||||
LD [M] drivers/gpu/drm/amd/amdgpu/amdgpu.ko
|
||||
```
|
||||
|
||||
##### 6. Strip and Compress
|
||||
|
||||
```fish
|
||||
# Strip debug symbols: ~600MB → ~30MB
|
||||
strip --strip-debug drivers/gpu/drm/amd/amdgpu/amdgpu.ko
|
||||
|
||||
# Compress: ~30MB → ~4.5MB
|
||||
zstd -19 -f drivers/gpu/drm/amd/amdgpu/amdgpu.ko
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 9. Step 7 — Install Module & Reboot
|
||||
|
||||
##### 1. Backup the Original Module
|
||||
|
||||
```fish
|
||||
set KVER (uname -r)
|
||||
set MODULE_DIR /usr/lib/modules/$KVER/kernel/drivers/gpu/drm/amd/amdgpu
|
||||
|
||||
sudo cp $MODULE_DIR/amdgpu.ko.zst $MODULE_DIR/amdgpu.ko.zst.original
|
||||
```
|
||||
|
||||
##### 2. Install the Patched Module
|
||||
|
||||
```fish
|
||||
sudo cp drivers/gpu/drm/amd/amdgpu/amdgpu.ko.zst $MODULE_DIR/amdgpu.ko.zst
|
||||
sudo depmod -a
|
||||
```
|
||||
|
||||
##### 3. Rebuild Initramfs
|
||||
|
||||
For Limine:
|
||||
```fish
|
||||
sudo limine-update
|
||||
```
|
||||
|
||||
For GRUB/mkinitcpio:
|
||||
```fish
|
||||
sudo mkinitcpio -P
|
||||
```
|
||||
|
||||
##### 4. Verify Before Reboot
|
||||
|
||||
```fish
|
||||
# Check the installed module has all 9 BC-250 strings
|
||||
zstd -d -c $MODULE_DIR/amdgpu.ko.zst | strings | grep "BC-250"
|
||||
```
|
||||
|
||||
Expected output (9 strings):
|
||||
```
|
||||
BC-250: GFXOFF disabled to prevent GPU power-state hangs
|
||||
BC-250: GPU unreachable (MMIO returned 0xFFFFFFFF)...
|
||||
BC-250: GPU died during sem acquire (0xFFFFFFFF)
|
||||
BC-250: GPU died during TLB flush ACK wait (0xFFFFFFFF)
|
||||
BC-250: GPU unreachable in fw_reg_write_reg_wait...
|
||||
BC-250: GPU died during reg_write_reg_wait (0xFFFFFFFF)
|
||||
BC-250 KIQ bypass active
|
||||
BC-250 KIQ bypass active in fw_reg_write_reg_wait
|
||||
BC-250: MMIO reg write/wait timeout
|
||||
```
|
||||
|
||||
##### 5. Reboot
|
||||
|
||||
```fish
|
||||
sudo reboot
|
||||
```
|
||||
|
||||
##### Emergency Rollback
|
||||
|
||||
If the system doesn't boot or the GPU crashes, restore the original module:
|
||||
|
||||
```fish
|
||||
set KVER (uname -r)
|
||||
set MODULE_DIR /usr/lib/modules/$KVER/kernel/drivers/gpu/drm/amd/amdgpu
|
||||
|
||||
sudo cp $MODULE_DIR/amdgpu.ko.zst.original $MODULE_DIR/amdgpu.ko.zst
|
||||
sudo depmod -a
|
||||
sudo limine-update # or: sudo mkinitcpio -P
|
||||
sudo reboot
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 10. Step 8 — Post-Reboot Verification
|
||||
|
||||
After reboot, run the verification script:
|
||||
|
||||
```fish
|
||||
bash scripts/06_verify_installation.sh
|
||||
```
|
||||
|
||||
Or check manually:
|
||||
|
||||
### Check dmesg for v3 Module
|
||||
|
||||
```fish
|
||||
sudo dmesg | grep -E "BC-250|GFXOFF|out-of-tree"
|
||||
```
|
||||
|
||||
Expected:
|
||||
```
|
||||
amdgpu: loading out-of-tree module taints kernel.
|
||||
amdgpu 0000:01:00.0: amdgpu: BC-250: GFXOFF disabled to prevent GPU power-state hangs
|
||||
```
|
||||
|
||||
### Check for Errors
|
||||
|
||||
```fish
|
||||
# KIQ timeout errors (should be 0)
|
||||
sudo dmesg | grep -ci "timeout waiting for kiq fence"
|
||||
|
||||
# GPU unreachable events (should be 0)
|
||||
sudo dmesg | grep -ci "GPU unreachable\|GPU died"
|
||||
|
||||
# NULL pointer / BUG (should be 0)
|
||||
sudo dmesg | grep -ci "BUG\|NULL pointer"
|
||||
```
|
||||
|
||||
### Check Device Nodes
|
||||
|
||||
```fish
|
||||
ls /dev/dri/
|
||||
# Expected: card0 renderD128 (and by-path/)
|
||||
|
||||
ls /dev/kfd
|
||||
# Expected: /dev/kfd (character device)
|
||||
```
|
||||
|
||||
### Check ppfeaturemask
|
||||
|
||||
```fish
|
||||
cat /sys/module/amdgpu/parameters/ppfeaturemask
|
||||
# Expected: 0xfff73ef7
|
||||
```
|
||||
|
||||
### Check rocminfo
|
||||
|
||||
```fish
|
||||
rocminfo | grep -E "Name:|Marketing|gfx|Done"
|
||||
```
|
||||
|
||||
Expected output includes:
|
||||
```
|
||||
Name: gfx1010
|
||||
Marketing Name: AMD BC-250
|
||||
*** Done ***
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 11. Step 9 — HIP Compute Tests
|
||||
|
||||
Compile and run the provided HIP test programs:
|
||||
|
||||
### Compile Tests
|
||||
|
||||
```fish
|
||||
cd tests/
|
||||
hipcc --offload-arch=gfx1010 -o hip_probe hip_probe.cpp
|
||||
hipcc --offload-arch=gfx1010 -o hip_minimal_test hip_minimal_test.cpp
|
||||
hipcc --offload-arch=gfx1010 -o hip_vector_add hip_vector_add.cpp
|
||||
```
|
||||
|
||||
### Run Tests (in order)
|
||||
|
||||
```fish
|
||||
# Test 1: Device probe (no compute)
|
||||
./hip_probe
|
||||
|
||||
# Test 2: Minimal kernel compute (32 values)
|
||||
./hip_minimal_test
|
||||
|
||||
# Test 3: Full vector add with managed memory (65536 elements, sin²+cos²=1.0)
|
||||
./hip_vector_add
|
||||
```
|
||||
|
||||
### Sequential Stress Test
|
||||
|
||||
The critical test — on unpatched kernels, the **second** consecutive HIP run crashes the system:
|
||||
|
||||
```fish
|
||||
for i in (seq 1 5)
|
||||
echo "=== Round $i ==="
|
||||
./hip_minimal_test
|
||||
sleep 2
|
||||
end
|
||||
```
|
||||
|
||||
All 5 rounds should pass. If any round hangs or crashes, the module patch may not be applied
|
||||
correctly.
|
||||
|
||||
You can also run the comprehensive test script:
|
||||
|
||||
```fish
|
||||
bash scripts/07_run_hip_tests.sh
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 12. Cyan Skillfish Governor (GPU Clock Management)
|
||||
|
||||
The BC-250's GPU clock/voltage management is handled by the `cyan-skillfish-governor` service.
|
||||
The kernel patches handle GFXOFF/power-state prevention; the governor handles DPM clock scaling.
|
||||
|
||||
### Installation
|
||||
|
||||
```fish
|
||||
paru -S cyan-skillfish-governor
|
||||
```
|
||||
|
||||
### Enable & Start
|
||||
|
||||
```fish
|
||||
sudo systemctl enable --now cyan-skillfish-governor
|
||||
```
|
||||
|
||||
### Verify
|
||||
|
||||
```fish
|
||||
systemctl is-active cyan-skillfish-governor
|
||||
# Expected: active
|
||||
|
||||
cat /sys/class/drm/card0/device/pp_dpm_sclk
|
||||
# Expected: Shows clock levels with governor managing transitions
|
||||
```
|
||||
|
||||
### Safe Operating Points
|
||||
|
||||
| Clock | Voltage | Use Case |
|
||||
|-------|---------|----------|
|
||||
| 1000 MHz | 700 mV | Idle |
|
||||
| 1500 MHz | 900 mV | Light load |
|
||||
| 2000 MHz | 1000 mV | Compute (max recommended) |
|
||||
|
||||
---
|
||||
|
||||
## 13. GPU Watchdog (Safety Monitor)
|
||||
|
||||
A safety watchdog script monitors kernel logs for KIQ timeouts and auto-kills GPU processes
|
||||
before the timeout cascade crashes the system. The window to act is ~10 seconds after first timeout.
|
||||
|
||||
### Usage
|
||||
|
||||
```fish
|
||||
# Monitor and auto-kill on KIQ timeout
|
||||
bash scripts/gpu_watchdog.sh
|
||||
|
||||
# Monitor only (no kill)
|
||||
bash scripts/gpu_watchdog.sh --dry-run
|
||||
|
||||
# Kill after 2 KIQ timeouts instead of 1
|
||||
bash scripts/gpu_watchdog.sh --max-kiq 2
|
||||
```
|
||||
|
||||
### Running as a Background Monitor
|
||||
|
||||
```fish
|
||||
nohup bash scripts/gpu_watchdog.sh > /tmp/gpu_watchdog.log 2>&1 &
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 14. HIP Programming Guidelines for BC-250
|
||||
|
||||
### Memory Allocation
|
||||
|
||||
| Function | Safe? | Notes |
|
||||
|----------|-------|-------|
|
||||
| `hipMallocManaged()` | **YES** | Preferred — unified memory for APU |
|
||||
| `hipHostMalloc()` | **YES** | Pinned host memory — safe |
|
||||
| `hipHostMallocCoherent` | **YES** | Cache-coherent host memory |
|
||||
| `hipMalloc()` | **CAUTION** | Works with v3 patches but may be slower |
|
||||
| `hipMemcpy()` | **CAUTION** | Works with v3 patches |
|
||||
|
||||
### Critical Rules
|
||||
|
||||
1. **Never call `hipDeviceReset()`** — triggers KIQ queue teardown, crashes the system
|
||||
2. **Use `_exit(0)` instead of `return 0`** in HIP programs — bypasses C++ static destructors
|
||||
that trigger HIP runtime cleanup, which can cause KIQ fence timeouts
|
||||
3. **Never call `rocm-smi` repeatedly** — GPU management queries can destabilize the device
|
||||
4. **Run a single long-lived daemon** for GPU workloads when possible — each process startup/exit
|
||||
cycles TLB flush paths
|
||||
|
||||
### Example: Safe HIP Program Exit
|
||||
|
||||
```cpp
|
||||
#include <hip/hip_runtime.h>
|
||||
#include <unistd.h> // _exit()
|
||||
|
||||
int main() {
|
||||
// ... your HIP code ...
|
||||
|
||||
// CRITICAL: Use _exit() to avoid HIP runtime destructor crash
|
||||
printf("Done\n");
|
||||
fflush(stdout);
|
||||
_exit(0); // Bypasses C++ destructors and HIP cleanup
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 15. Kernel Module Rebuild After Updates
|
||||
|
||||
**Any CachyOS kernel update will replace your patched module.** After updating the kernel,
|
||||
you must rebuild and reinstall the patched module.
|
||||
|
||||
### Quick Rebuild Steps
|
||||
|
||||
```fish
|
||||
set OLD_KVER "6.19.6-2-cachyos" # Previous kernel
|
||||
set NEW_KVER (uname -r) # New kernel after update
|
||||
|
||||
# If kernel version changed, download new CachyOS source and repeat Step 6
|
||||
# If only pkgrel changed (e.g., -2-cachyos → -3-cachyos), you may be able to
|
||||
# just copy the module and rebuild initramfs:
|
||||
|
||||
set MODULE_DIR /usr/lib/modules/$NEW_KVER/kernel/drivers/gpu/drm/amd/amdgpu
|
||||
sudo cp /usr/lib/modules/$OLD_KVER/kernel/drivers/gpu/drm/amd/amdgpu/amdgpu.ko.zst $MODULE_DIR/
|
||||
sudo depmod -a
|
||||
sudo limine-update
|
||||
```
|
||||
|
||||
> **Note**: This copy approach only works if the kernel ABI hasn't changed between versions.
|
||||
> If the module fails to load, you must do a full rebuild from the new kernel source.
|
||||
|
||||
---
|
||||
|
||||
## 16. Technical Reference — Why This GPU Needs Patches
|
||||
|
||||
### The KIQ Problem
|
||||
|
||||
KIQ (Kernel Interface Queue) is a privileged ring buffer used by the amdgpu driver for TLB
|
||||
invalidation and GPU management. On the BC-250, KIQ commands cause **fence timeouts** — the GPU
|
||||
never acknowledges the command, and the CPU waits indefinitely.
|
||||
|
||||
There are exactly **4 code paths** that use the KIQ ring at runtime:
|
||||
|
||||
| # | Function | File | Patched? |
|
||||
|---|----------|------|----------|
|
||||
| 1 | `gmc_v10_0_flush_gpu_tlb()` | gmc_v10_0.c | **YES** — goto use_mmio |
|
||||
| 2 | `amdgpu_gmc_flush_gpu_tlb_pasid()` | amdgpu_gmc.c | **YES** — direct callout |
|
||||
| 3 | `amdgpu_gmc_fw_reg_write_reg_wait()` | amdgpu_gmc.c | **YES** — MMIO write+poll |
|
||||
| 4 | `amdgpu_gfx_enable/disable_kcq()` | amdgpu_gfx.c | No (boot only, works fine) |
|
||||
|
||||
### The GFXOFF Problem
|
||||
|
||||
After a HIP process exits, the GPU enters GFXOFF (a power-saving state). When the next process
|
||||
starts and triggers a TLB flush, the GPU is unresponsive. MMIO reads via `readl()` inside a
|
||||
spinlock-protected loop hang the CPU indefinitely because the BC-250's internal PCIe fabric has
|
||||
**no completion timeout** (`DevCap2: Completion Timeout: Not Supported`).
|
||||
|
||||
### The v3 Three-Layer Solution
|
||||
|
||||
| Layer | Mechanism | Purpose |
|
||||
|-------|-----------|---------|
|
||||
| 1 — GFXOFF Disable | `gfx_v10_0.c`: `PP_GFXOFF_MASK` cleared for IP 10.1.3 | Prevents GPU from entering unrecoverable power-save |
|
||||
| 2 — KIQ Bypass + Dead-GPU | `gmc_v10_0.c` + `amdgpu_gmc.c`: MMIO instead of KIQ, 0xFFFFFFFF checks | Avoids KIQ ring entirely; detects dead GPU before hanging |
|
||||
| 3 — Boot Parameters | `ppfeaturemask=0xfff73ef7` | Belt-and-suspenders: disables GFXOFF+DeepSleep+ULV at power management level |
|
||||
|
||||
### Patch Summary (8 modifications across 3 files)
|
||||
|
||||
| # | File | Function | Type | Purpose |
|
||||
|---|------|----------|------|---------|
|
||||
| 1 | `gfx_v10_0.c` | `gfx_v10_0_check_gfxoff_flag` | v3 | Disable GFXOFF for IP 10.1.3 |
|
||||
| 2 | `gmc_v10_0.c` | `gmc_v10_0_flush_gpu_tlb` | v2 | KIQ bypass → `goto use_mmio` |
|
||||
| 3 | `gmc_v10_0.c` | `gmc_v10_0_flush_gpu_tlb` | v3 | Pre-spinlock 0xFFFFFFFF health check |
|
||||
| 4 | `gmc_v10_0.c` | `gmc_v10_0_flush_gpu_tlb` | v3 | Sem acquire loop dead-GPU bail |
|
||||
| 5 | `gmc_v10_0.c` | `gmc_v10_0_flush_gpu_tlb` | v3 | ACK-wait loop dead-GPU bail |
|
||||
| 6 | `gmc_v10_0.c` | `gmc_v10_0_hw_init` | v2 | `flush_pasid_uses_kiq = false` |
|
||||
| 7 | `amdgpu_gmc.c` | `amdgpu_gmc_flush_gpu_tlb_pasid` | v2 | Direct MMIO TLB flush |
|
||||
| 8 | `amdgpu_gmc.c` | `amdgpu_gmc_fw_reg_write_reg_wait` | v2+v3 | MMIO write+poll + dead-GPU detection |
|
||||
|
||||
### Why Mining OS (Kernel 5.10) Worked Without Patches
|
||||
|
||||
In kernel 5.10 + AMDGPU-PRO 22.20, TLB flush was simpler and more localized. The centralized
|
||||
`amdgpu_gmc_flush_gpu_tlb_pasid()` in `amdgpu_gmc.c` and the deferred work
|
||||
`amdgpu_vm_tlb_fence_work()` didn't exist yet. KIQ failures returned `-ETIME` gracefully instead
|
||||
of cascading to GPU death. Modern kernels (6.x) refactored TLB flushing into a shared centralized
|
||||
path that defaults to KIQ for all hardware — which breaks BC-250.
|
||||
|
||||
---
|
||||
|
||||
## 17. Troubleshooting
|
||||
|
||||
### System freezes after HIP program runs
|
||||
|
||||
**Cause**: v3 module not loaded, or initramfs not rebuilt after installation.
|
||||
|
||||
**Fix**: Boot from recovery, restore original module, verify initramfs was rebuilt:
|
||||
```fish
|
||||
sudo cp /usr/lib/modules/(uname -r)/kernel/drivers/gpu/drm/amd/amdgpu/amdgpu.ko.zst.original \
|
||||
/usr/lib/modules/(uname -r)/kernel/drivers/gpu/drm/amd/amdgpu/amdgpu.ko.zst
|
||||
sudo depmod -a
|
||||
sudo limine-update # MUST rebuild initramfs!
|
||||
sudo reboot
|
||||
```
|
||||
|
||||
### NULL pointer dereference at boot (drm_vma_offset_add)
|
||||
|
||||
**Cause**: Module was built from vanilla kernel.org source instead of CachyOS source.
|
||||
|
||||
**Fix**: Rebuild from CachyOS-specific kernel source (see Step 6).
|
||||
|
||||
### `rocminfo` shows no GPU / "No HSA GPU agents found"
|
||||
|
||||
**Cause**: Environment variables not set, or `/dev/kfd` doesn't exist.
|
||||
|
||||
**Fix**:
|
||||
```fish
|
||||
# Check env vars
|
||||
echo $HSA_OVERRIDE_GFX_VERSION # Should be 10.1.0
|
||||
|
||||
# Check /dev/kfd exists
|
||||
ls -la /dev/kfd
|
||||
|
||||
# Check user groups
|
||||
groups # Should include 'render'
|
||||
```
|
||||
|
||||
### "KIQ timeout" messages in dmesg
|
||||
|
||||
**Cause**: Stock (unpatched) module is loaded instead of v3.
|
||||
|
||||
**Fix**: Verify the v3 module is installed:
|
||||
```fish
|
||||
zstd -d -c /usr/lib/modules/(uname -r)/kernel/drivers/gpu/drm/amd/amdgpu/amdgpu.ko.zst \
|
||||
| strings | grep "BC-250"
|
||||
# Should show 9 BC-250 strings
|
||||
```
|
||||
|
||||
### Module version mismatch after kernel update
|
||||
|
||||
**Cause**: CachyOS updated the kernel, replacing the patched module.
|
||||
|
||||
**Fix**: Rebuild the module from matching CachyOS source (see Section 15).
|
||||
|
||||
---
|
||||
|
||||
## 18. File Inventory
|
||||
|
||||
```
|
||||
BC250_ROCm_Guide/
|
||||
├── README.md ← This guide
|
||||
├── config/
|
||||
│ └── amdgpu.conf ← Modprobe configuration
|
||||
├── patches/
|
||||
│ ├── patch1_gfxoff.py ← Layer 1: Disable GFXOFF
|
||||
│ ├── patch2_gmc.py ← Layer 2: KIQ bypass + dead-GPU (gmc_v10_0.c)
|
||||
│ └── patch3_amdgpu_gmc.py ← Layer 2: KIQ bypass + dead-GPU (amdgpu_gmc.c)
|
||||
├── scripts/
|
||||
│ ├── 01_install_rocm_packages.sh ← Package installation
|
||||
│ ├── 02_configure_environment.sh ← Environment variables + modprobe + boot params
|
||||
│ ├── 04_build_patched_module.sh ← Download source, patch, build module
|
||||
│ ├── 05_install_module.sh ← Install module + rebuild initramfs
|
||||
│ ├── 06_verify_installation.sh ← Post-reboot verification
|
||||
│ ├── 07_run_hip_tests.sh ← Compile and run HIP tests
|
||||
│ └── gpu_watchdog.sh ← KIQ timeout safety monitor
|
||||
└── tests/
|
||||
├── hip_probe.cpp ← Device diagnostic (6 steps)
|
||||
├── hip_minimal_test.cpp ← Minimal kernel compute test
|
||||
└── hip_vector_add.cpp ← Full managed-memory vector add test
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
*Based on research and patches by Dani (2026-02-22), adapted and verified on CachyOS kernel
|
||||
6.19.6-2-cachyos with ROCm 7.2.0.*
|
||||
|
||||
Reference in New Issue
Block a user