SIGFPE crash in gralloc.gbm.so (gralloc_gbm_bo_create) with AMD GPU on host mode — RGBA_1010102 format (Unity 6 apps)

[fogelmanjg]

Summary

When running RedDroid in GPU host mode on an AMD GPU (tested: AMD Ryzen 5700G iGPU, Mesa 24.0.8, radeonsi), any app using Unity 6 — or any app that requests the HAL_PIXEL_FORMAT_RGBA_1010102 (10-bit) pixel format — causes an immediate SIGFPE (integer division by zero) crash in the android.hardware.graphics.allocator@2.0-service, taking down the entire Android container.

Tested image: erstt/redroid:15.0.0_ndk_magisk_litegapps_AVD (Android 15). Guest mode (SwiftShader) works fine; the crash only happens in host GPU mode.

---

Crash log

signal 8 (SIGFPE), code 1 (FPE_INTDIV), fault addr 0x...
pid: NNN (allocator@2.0-s)

backtrace:
  #00 /vendor/lib64/hw/gralloc.gbm.so (gralloc_gbm_bo_create+621)
  #01 /vendor/lib64/hw/gralloc.gbm.so
  #02 /vendor/lib64/hw/android.hardware.graphics.allocator@2.0-impl.so (allocateBuffers+328)
  #03 /vendor/lib64/hw/android.hardware.graphics.allocator@2.0-impl.so (allocate+90)

---

Root cause

gralloc_gbm_bo_create contains a jump table for bytes-per-pixel (bpp) calculation covering HAL pixel formats up to HAL_PIXEL_FORMAT_FLEX_RGBA_8888 (value 0x2A = 42). For formats above 0x2A, it only handles YV12. Any other format in that range falls through to bpp = 0, which then causes division by zero when computing the stride.

Disassembly of the relevant section (x86_64, file offset 0x57d2):

; Format > 0x2A reaches here
67bc:  cmpl  $0x32315659, 0x18(%rsp)   ; compare with 'YV12'
67c9:  jne   67d2                       ; not YV12 → fall through to bpp=0
67cb:  mov   $0x1, %ecx                ; YV12 → bpp=1
67d0:  jmp   67db

67d2:  xor   %ecx, %ecx               ; ← bpp = 0  (BUG)
67d4:  jmp   67db

67db:  xor   %edx, %edx
67dd:  div   %ecx                      ; ← SIGFPE when ecx=0
67df:  mov   %eax, (%rdi)             ; store stride

Affected format: HAL_PIXEL_FORMAT_RGBA_1010102 = 0x2B = 43, which is > 0x2A and not YV12. Unity 6 requests this format for its HDR/10-bit render pipeline.

The source-level fix should be to add HAL_PIXEL_FORMAT_RGBA_1010102 (and likely HAL_PIXEL_FORMAT_RGBA_FP16) to the handled cases in gralloc_gbm_bo_create, or to guard the division against bpp == 0 before dividing.

---

Binary workaround (until fixed in source)

For users affected by this, a 2-byte patch to gralloc.gbm.so resolves it:

File: /vendor/lib64/hw/gralloc.gbm.so
Offset: 0x57d2
Change: 31 C9 → B1 04 (xor ecx,ecx → mov cl, 4)

This sets bpp=4 for unknown formats (correct for RGBA_1010102, a 32-bit packed format), preventing the division by zero. Applied with Python:

with open('gralloc.gbm.so', 'rb') as f:
    data = bytearray(f.read())
data[0x57d2] = 0xb1
data[0x57d3] = 0x04
with open('gralloc.gbm.so', 'wb') as f:
    f.write(data)

A patched Docker image is available as a reference: android-redroid:15-gapps-magisk-amd-fix (based on erstt/redroid:15.0.0_ndk_magisk_litegapps_AVD with only this change applied).

---

Environment

Host CPU	AMD Ryzen 7 5700G (Vega iGPU)
Host kernel	6.8.0-124-generic
Mesa	24.0.8, radeonsi driver
RedDroid image	erstt/redroid:15.0.0_ndk_magisk_litegapps_AVD
Android version	15
GPU mode	host (/dev/dri/renderD128, /dev/dri/card1)
Guest mode	works (ANGLE + SwiftShader)

---

Steps to reproduce

1. Run RedDroid Android 15 in host GPU mode on any AMD GPU
2. Install any Unity 6 game (e.g. Loop Sort)
3. Launch the game — the graphics allocator service crashes immediately, taking down the container

Any app requesting HAL_PIXEL_FORMAT_RGBA_1010102 will reproduce this.