gfx/common/vulkan: plug partial-init leak in emulated mailbox + CI

Found during the v2 audit of gfx/common/vulkan_common.c, deferred
from the v3 patch (da5e650) which addressed the three high-severity
heap-corruption findings.  Adds an AddressSanitizer + LSan
regression test under samples/gfx/vulkan_mailbox_init_leak/ wired
into the existing Linux-samples-gfx CI workflow as a fifth step.

vulkan_emulated_mailbox_init has three sequential allocations
(scond_new, slock_new, sthread_create) and on any of the latter
two failures returns `false` directly, leaking the already-
allocated cond and/or lock.  Both production call sites in
vulkan_create_swapchain ignore the return value -- so an init
failure also leaves vk->mailbox.lock == NULL and vk->mailbox.cond
== NULL while VK_DATA_FLAG_EMULATING_MAILBOX is still set,
setting up a NULL-deref the next time vulkan_acquire_next_image()
routes into the emulated path (slock_lock(NULL) inside
vulkan_emulated_mailbox_acquire_next_image()).

The leak itself is small (the libretro-common slock_t / scond_t
each hold a pthread mutex / cond worth of data); the trigger
requires scond_new or slock_new to fail, which means OOM or
pthread resource exhaustion.  Realistic mainly on memory-
constrained devices (older Android, Vita, 3DS, Wii U) where
filter-chain teardown / re-init under thrashing memory pressure
can put the system in this state.

* gfx/common/vulkan_common.c

  Route every early-failure path in vulkan_emulated_mailbox_init
  through `goto error` to a single cleanup that calls
  vulkan_emulated_mailbox_deinit().  The deinit function is
  null-safe (each free is guarded by an `if (mailbox->X)` check)
  and ends with a memset, so the deinit-on-failure shape matches
  the deinit-on-shutdown shape exactly -- the two production
  callers that ignore the return value get a self-healing
  failure mode.

  The post-deinit memset also clears mailbox->swapchain, which
  trips the existing guard at vulkan_acquire_next_image:

     if (vk->mailbox.swapchain == VK_NULL_HANDLE)
        err = VK_ERROR_OUT_OF_DATE_KHR;

  closing the NULL-deref crash that the bare `return false` left
  open.  No caller-side changes needed.

* samples/gfx/vulkan_mailbox_init_leak/,
  .github/workflows/Linux-samples-gfx.yml

  Regression test following the convention of the v3 vulkan/
  tests, the v4 slang test, and the security regression tests
  under samples/tasks/.  Keeps verbatim copies of both
  vulkan_emulated_mailbox_init() and vulkan_emulated_mailbox_
  deinit() demarcated by `=== verbatim copy ===` markers, with
  an `IMPORTANT: ... must follow` comment.  Plain C, asserts
  manually, exits nonzero on failure.

  Five probes: scond fails first (was already correct, baseline
  case); slock fails second (pre-fix: scond leak); sthread fails
  third (pre-fix: scond + slock leak); full success (sanity check
  on alloc/free pairing post-deinit); and an explicit
  __lsan_do_recoverable_leak_check() call after running all
  three failure stages.  Mocks scond_new / slock_new /
  sthread_create with controllable failure injection so each
  stage can be exercised deterministically.

  Verified pre/post-patch discrimination: under the pre-fix
  `return false` shape, ASan's LeakSanitizer reports 384 bytes
  leaked across 6 allocations (1 scond from slock-fails + 2
  alloc/leak pairs from sthread-fails + 3 from the lsan-check
  probe), each with a full backtrace pointing into
  vulkan_emulated_mailbox_init.  Under the post-fix shape all
  five probes pass clean.  The CI step runs with
  ASAN_OPTIONS=detect_leaks=1 so leaks fail the run with a
  non-zero exit.

Author: LibretroAdmin

.github/workflows/Linux-samples-gfx.yml

@@ -139,3 +139,36 @@ jobs:
           test -x slang_texture_index_bounds_test
           timeout 60 ./slang_texture_index_bounds_test
           echo "[pass] slang_texture_index_bounds_test"
+
+      - name: Build and run vulkan_mailbox_init_leak_test (ASan + LSan)
+        shell: bash
+        working-directory: samples/gfx/vulkan_mailbox_init_leak
+        run: |
+          set -eu
+          # Regression test for the partial-init leak fix in
+          # gfx/common/vulkan_common.c::vulkan_emulated_mailbox_init.
+          # Pre-fix the function had three sequential allocations
+          # (scond_new, slock_new, sthread_create) and on any of the
+          # latter two failures returned `false` directly, leaking
+          # the already-allocated cond and/or lock.  Both production
+          # call sites in vulkan_create_swapchain ignore the return
+          # value, so an init failure also left vk->mailbox.lock ==
+          # NULL and vk->mailbox.cond == NULL while VK_DATA_FLAG_
+          # EMULATING_MAILBOX was still set, setting up a NULL-deref
+          # the next time vulkan_acquire_next_image routed into
+          # vulkan_emulated_mailbox_acquire_next_image (slock_lock
+          # on a NULL pointer).  Fix routes every early failure
+          # through `goto error` to a single deinit call, which is
+          # null-safe and ends with a memset, leaving the struct in
+          # the same shape the deinit-on-shutdown path produces and
+          # tripping the existing `mailbox.swapchain == VK_NULL_
+          # HANDLE` guard at vulkan_acquire_next_image.  Build under
+          # AddressSanitizer with leak detection so any
+          # reintroduction is caught at the leak level.  If
+          # vulkan_common.c amends the init or deinit, the verbatim
+          # copies in vulkan_mailbox_init_leak_test.c must follow.
+          make clean all SANITIZER=address
+          test -x vulkan_mailbox_init_leak_test
+          ASAN_OPTIONS=detect_leaks=1 timeout 60 \
+             ./vulkan_mailbox_init_leak_test
+          echo "[pass] vulkan_mailbox_init_leak_test"

gfx/common/vulkan_common.c

@@ -265,13 +265,26 @@ static bool vulkan_emulated_mailbox_init(
    mailbox->flags               = 0;
 
    if (!(mailbox->cond      = scond_new()))
-      return false;
+      goto error;
    if (!(mailbox->lock      = slock_new()))
-      return false;
+      goto error;
    if (!(mailbox->thread    = sthread_create(vulkan_emulated_mailbox_loop,
                mailbox)))
-      return false;
+      goto error;
    return true;
+
+error:
+   /* Tear down anything we managed to allocate before failing.
+    * vulkan_emulated_mailbox_deinit() is null-safe and ends with
+    * a memset, so the struct is left in the same shape a caller
+    * would see after a successful init+deinit cycle -- callers
+    * that ignore our return value (the two sites in
+    * vulkan_create_swapchain) will then take the
+    * mailbox.swapchain == VK_NULL_HANDLE branch in
+    * vulkan_acquire_next_image and skip the emulated path
+    * cleanly instead of dereferencing a NULL lock/cond. */
+   vulkan_emulated_mailbox_deinit(mailbox);
+   return false;
 }
 
 static void vulkan_debug_mark_object(VkDevice device,

samples/gfx/vulkan_mailbox_init_leak/Makefile

@@ -0,0 +1,25 @@
+TARGET := vulkan_mailbox_init_leak_test
+
+SOURCES := vulkan_mailbox_init_leak_test.c
+
+OBJS := $(SOURCES:.c=.o)
+
+CFLAGS += -Wall -pedantic -std=gnu99 -g -O0
+
+ifneq ($(SANITIZER),)
+   CFLAGS  := -fsanitize=$(SANITIZER) -fno-omit-frame-pointer $(CFLAGS)
+   LDFLAGS := -fsanitize=$(SANITIZER) $(LDFLAGS)
+endif
+
+all: $(TARGET)
+
+%.o: %.c
+	$(CC) -c -o $@ $< $(CFLAGS)
+
+$(TARGET): $(OBJS)
+	$(CC) -o $@ $^ $(LDFLAGS)
+
+clean:
+	rm -f $(TARGET) $(OBJS)
+
+.PHONY: clean