ASAN: ~1.5 MB leak surfaced by sanitizers job (test harness or core deinit)

[JoeMatt]

Labels: bug :bug:, tests, performance :rocket:

Summary

The sanitizers CI job (added in PR #121, commit 9f8e831) now correctly runs make test under ASAN+UBSAN with -shared-libasan and an UBSAN ignorelist for the machine-generated UAE 68K core (src/m68000/). After those false-positive sources were filtered out, the first clean run surfaces two real ASAN leaks that should be triaged before tightening the sanitizer gate from advisory (continue-on-error: true) to mandatory.

Evidence

From the sanitizer job on PR #121 / commit ba042e4 (job 73905947783):

Direct leak of N byte(s) in 1 object(s) allocated from:
    #0 0x... in malloc (libclang_rt.asan-x86_64.so)
    #1 0x...  (<unknown module>)

Direct leak of 256 byte(s) in 1 object(s) allocated from:
    #0 0x... in malloc (libclang_rt.asan-x86_64.so)
    #1 0x...  (<unknown module>)

SUMMARY: AddressSanitizer: 1573120 byte(s) leaked in 2 allocation(s).
make: *** [Makefile:749: test] Aborted (core dumped)

Two allocations:

• ~1.5 MB — likely a ROM buffer or core-state buffer that make test mallocs once and never frees on shutdown.
• 256 B — small fixture allocation, possibly an event-queue / cheat-list / SRAM scratch.

Both have <unknown module> frames because the ASAN runtime can't resolve symbols from the dlopen'd .so reliably with -shared-libasan. Need to re-run with stack-resolution help (ASAN_SYMBOLIZER_PATH + matching debug info, or build the test binary with the asan runtime linked statically into the test binary itself rather than the shared lib).

What needs to happen

• Get usable stacks — either:
  • Run locally with ASAN_OPTIONS=detect_leaks=1:print_stacktrace=1:fast_unwind_on_malloc=0 and a debug build (make DEBUG=1 CC="clang -fsanitize=address,undefined ...")
  • OR re-link the test binaries with -fsanitize=address directly (not -shared-libasan) so the asan runtime is in the test executable
• Identify the two allocations — most likely candidates: test/test_subsystem_init.c, test/test_audio_pipeline.c, or one of the harnesses that info.data = malloc(fsize) for ROM bytes (see test/tools/test_benchmark.c:262 for the pattern -- almost every test does this).
• Decide: real leak (fix) vs intentional (test fixture, owned by process lifetime — suppress via LSan_suppressions.txt).
• Once known clean: flip sanitizers from continue-on-error: true to false in .github/workflows/c-cpp.yml so future leaks gate.

Pre-work for the perf phase

Closing this in tandem with #123 (OP perf audit) makes sense — the OP profile work needs make benchmark running clean, and a leaky test harness contaminates Instruments / perf record attribution if the leaked memory grows during a long benchmark run.

Related

• PR chore: CI and code-quality improvements #121 — added the sanitizer job
• Issue [spike] Object Processor (OP) performance audit + optimization plan #123 — OP performance audit (next-up)
• .github/workflows/c-cpp.yml job sanitizers
• .ubsan-ignorelist — current UBSAN suppressions (only src/m68000/*)

[JoeMatt]

Fixed by PR #126 (merged at b73a11c). m68k_done() + GPUDone() wired into JaguarDone, with free_table68k() in readcpu.c keeping ownership in the module that allocates. Also fixed a missing p_retro_unload_game() in test_hle_bios that prevented JaguarDone from running for the second load.