OOB read in tom_render_16bpp_cry_scanline (1 byte past tomRam8)

[JoeMatt]

Labels: bug :bug:, tom, gpu, crash :fire:

Summary

ASAN catches a 1-byte global-buffer-overflow read at the end of tom_render_16bpp_cry_scanline (src/tom/tom.c:625). Was masked by the leak in #125; surfaced after PR #126 fixed that leak and the sanitizer job actually reached the rendering path.

ASAN trace

==3052==ERROR: AddressSanitizer: global-buffer-overflow on address 0x7f647f497260
READ of size 1 at 0x7f647f497260 thread T0
    #0 tom_render_16bpp_cry_scanline (src/tom/tom.c:625:25)
    #1 HalflineCallback                (src/core/jaguar.c:717:4)
    #2 JaguarExecuteNew                (src/core/jaguar.c)
    #3 retro_run                       (libretro.c:1091:4)
    #4 run_frames                      (test/test_subsystem_timeline.c:319)
    #5 test_68k_frame_counter          (test/test_subsystem_timeline.c:438)
    #6 main                            (test/test_subsystem_timeline.c:1015)

0x7f647f497260 is located 0 bytes after global variable 'tomRam8' defined
in src/tom/tom.c:386 (size 16384)

What the code looks like

// src/tom/tom.c:619 — tom_render_16bpp_cry_scanline
backbuffer += 2 * startPos, width -= startPos;

while (width >= pwidth_scale)
{
   uint16_t color = (*current_line_buffer++) << 8;     // line 625 — READ past tomRam8 end
   color |= *current_line_buffer++;
   for (s = 0; s < pwidth_scale; s++)
      *backbuffer++ = CRY16ToRGB32[color];
   width -= pwidth_scale;
}

current_line_buffer is (uint8_t *)&tomRam8[0x1800] (line buffer at offset 6 KB inside the 16 KB tomRam8). The loop runs while width >= pwidth_scale; on the affected halfline it consumes width bytes and walks past the end of tomRam8.

Reads exactly 1 byte past the end — usually reads zero/adjacent BSS in production, but a real OOB. ASAN catches it cleanly.

Reproducer

Run make test with sanitizers built in (the existing CI sanitizers job, or locally):

make clean
make TEST_EXPORTS=1 \
  CC="clang -fsanitize=address,undefined -shared-libasan -fno-omit-frame-pointer -O1 -g" \
  LD="clang -fsanitize=address,undefined -shared-libasan -fno-omit-frame-pointer -O1 -g"
ASAN_OPTIONS=detect_leaks=1:halt_on_error=1:print_stacktrace=1 \
  make test TEST_EXPORTS=1 CC="..."
# fails in test_subsystem_timeline / test_68k_frame_counter

What needs to happen

• Find the actual upper bound of current_line_buffer walk and either clamp width or change the loop condition to honor the buffer end. Likely off-by-one on the per-halfline budget (8 KB line buffer at 0x1800–0x37FF in tomRam8).
• Add a test in test/test_subsystem_timeline.c (or a new test/test_tom_render_bounds.c) that exercises the boundary case, so future regressions trip ASAN immediately.
• Once fixed, the sanitizers CI job should go fully clean — that's the trigger to flip continue-on-error: true → false on the job.

Related

• PR fix: free table68k + branch_condition_table on shutdown (closes #125) #126 — leak fix that surfaced this
• Issue ASAN: ~1.5 MB leak surfaced by sanitizers job (test harness or core deinit) #125 — original leak issue (closed by fix: free table68k + branch_condition_table on shutdown (closes #125) #126 once merged)

[JoeMatt]

Fixed by PR #126 (merged at b73a11c). New tom_clamp_line_buffer_width() helper applied to all five tom_render_*_scanline() functions; clamps width based on actual remaining bytes in the line buffer. ASAN now clean on the rendering path.