gfx/font_driver: plug renderer-state leak on OOM in font_driver_init_first

font_driver_init_first starts by calling font_init_first (or
video_thread_font_init in the threaded-video case), which on success
populates font_driver and font_handle.  That handle owns the raster
font's allocated state - glyph atlas texture, GL names, D3D COM
objects, etc. - allocated and held inside the chosen renderer
(gl2_raster_font / d3d11_font / vulkan_raster_font / ...).

Only after that succeeds do we malloc the small wrapper struct
(font_data_t).  Before this commit, if that wrapper malloc failed,
the function returned NULL directly and the entire raster-font
state was orphaned with no path to free it.  Cores that hit OOM
during font init would leak that state on every failed attempt.

Also refactors font_driver_free's renderer-side teardown into a
shared helper, font_driver_release_renderer_state, so both the
normal teardown path and the new OOM cleanup path handle the
threaded-vs-non-threaded dispatch the same way.  This matters:
renderer->free cannot be called on the main thread when threaded
video is active, because it touches GPU state owned by the video
thread (GL context, D3D ImmediateContext, D3D12 fence, Vulkan
command buffer recording).  The existing teardown already dispatches
via video_thread_texture_handle + font_driver_free_wrap; the OOM
path now takes the same route.

Before the refactor, font_driver_free had its own inline thread-
dispatch block with the safety rationale as a comment.  That comment
is preserved verbatim on the shared helper so the knowledge isn't
lost.

Thread-safety:
  - Non-threaded video: renderer->free runs synchronously on the
    caller's thread.  Identical to the previous behaviour for both
    call sites.
  - Threaded video: renderer->free is dispatched to the video thread
    via video_thread_texture_handle.  For the OOM cleanup path this
    is a behaviour fix (previous code didn't free at all), not a
    change to the teardown path itself.  video_thread_texture_handle
    is documented self-safe: falls back to direct call if the
    wrapper isn't active, calls func directly if already on the
    video thread.

Control flow in font_driver_free is slightly reorganised - the
original code had an early 'free(font); return;' inside the threaded
branch and let the non-threaded path fall through.  The refactor
unifies the post-renderer-free cleanup (NULL ivars + free(font)) at
the end of the function body.  Equivalent in behaviour, easier to
read.

No functional change to font_driver_init_first's success path; the
wrapper struct allocation, field assignment, and return are
unchanged.

Author: LibretroAdmin

gfx/font_driver.c

@@ -918,63 +918,71 @@ static uintptr_t font_driver_free_wrap(void *data)
 }
 #endif
 
+/* Free the renderer-owned state (glyph atlas / GPU textures / etc.)
+ * behind a (renderer, handle) pair.  Shared between the normal
+ * font_driver_free teardown path and the OOM cleanup path in
+ * font_driver_init_first; keeping the thread-dispatch logic in one
+ * place avoids the two call sites drifting out of sync.
+ *
+ * When threaded video is active, font resources (GPU textures, GL
+ * names, D3D COM objects) belong to the video thread's rendering
+ * context.  Freeing them on the main thread races with the video
+ * thread's draw calls:
+ *
+ *  - GL: context is single-threaded; gl2_raster_font_free
+ *    calls make_current to steal the context, but the video
+ *    thread may be mid-frame.
+ *  - D3D11: ImmediateContext is not thread-safe; Release on
+ *    the main thread while the video thread draws is UB.
+ *  - D3D12: fenceValue++ from the main thread races with
+ *    the video thread's own fence signalling.
+ *  - Vulkan: vkQueueWaitIdle under queue_lock only drains
+ *    submitted work, not command buffers being recorded.
+ *
+ * Dispatch renderer->free to the video thread via
+ * video_thread_texture_handle so it runs serialised with
+ * the video thread's frame rendering.  This is the same
+ * pattern used by texture load/unload.
+ *
+ * video_thread_texture_handle is self-safe: if the wrapper
+ * is not active (VIDEO_FLAG_THREAD_WRAPPER_ACTIVE not set),
+ * it falls back to calling func(data) on the current thread.
+ * If called from the video thread itself, it calls func
+ * directly (no deadlock). */
+static void font_driver_release_renderer_state(
+      const font_renderer_t *renderer, void *renderer_data,
+      bool is_threaded)
+{
+   if (!renderer || !renderer->free)
+      return;
+
+#ifdef HAVE_THREADS
+   if (is_threaded)
+   {
+      font_free_cmd_t cmd;
+      cmd.renderer      = renderer;
+      cmd.renderer_data = renderer_data;
+      cmd.is_threaded   = is_threaded;
+      video_thread_texture_handle(&cmd, font_driver_free_wrap);
+      return;
+   }
+#endif
+
+   renderer->free(renderer_data, is_threaded);
+}
+
 void font_driver_free(font_data_t *font)
 {
    if (font)
    {
-      const font_renderer_t *renderer;
       bool is_threaded        = false;
 #ifdef HAVE_THREADS
       bool *is_threaded_tmp   = video_driver_get_threaded();
       is_threaded             = *is_threaded_tmp;
 #endif
-      renderer                = font ? font->renderer : NULL;
 
-#ifdef HAVE_THREADS
-      /* When threaded video is active, font resources (GPU
-       * textures, GL names, D3D COM objects) belong to the
-       * video thread's rendering context.  Freeing them on
-       * the main thread races with the video thread's draw
-       * calls:
-       *
-       *  - GL: context is single-threaded; gl2_raster_font_free
-       *    calls make_current to steal the context, but the video
-       *    thread may be mid-frame.
-       *  - D3D11: ImmediateContext is not thread-safe; Release on
-       *    the main thread while the video thread draws is UB.
-       *  - D3D12: fenceValue++ from the main thread races with
-       *    the video thread's own fence signalling.
-       *  - Vulkan: vkQueueWaitIdle under queue_lock only drains
-       *    submitted work, not command buffers being recorded.
-       *
-       * Dispatch renderer->free to the video thread via
-       * video_thread_texture_handle so it runs serialised with
-       * the video thread's frame rendering.  This is the same
-       * pattern used by texture load/unload.
-       *
-       * video_thread_texture_handle is self-safe: if the wrapper
-       * is not active (VIDEO_FLAG_THREAD_WRAPPER_ACTIVE not set),
-       * it falls back to calling func(data) on the current thread.
-       * If called from the video thread itself, it calls func
-       * directly (no deadlock). */
-      if (is_threaded && renderer && renderer->free)
-      {
-         font_free_cmd_t cmd;
-         cmd.renderer      = renderer;
-         cmd.renderer_data = font->renderer_data;
-         cmd.is_threaded   = is_threaded;
-         video_thread_texture_handle(&cmd, font_driver_free_wrap);
-
-         font->renderer      = NULL;
-         font->renderer_data = NULL;
-
-         free(font);
-         return;
-      }
-#endif
-
-      if (renderer && renderer->free)
-         renderer->free(font->renderer_data, is_threaded);
+      font_driver_release_renderer_state(font->renderer,
+            font->renderer_data, is_threaded);
 
       font->renderer      = NULL;
       font->renderer_data = NULL;
@@ -1014,6 +1022,18 @@ font_data_t *font_driver_init_first(
          font->size          = font_size;
          return font;
       }
+
+      /* Wrapper malloc failed after font_init_first (or
+       * video_thread_font_init) had already succeeded.  The raster
+       * font's init path allocates the glyph atlas / GPU textures /
+       * COM objects behind font_handle; returning NULL here without
+       * releasing them would leak the entire raster-font state and,
+       * on subsequent re-init attempts, accumulate.  Dispatch via
+       * the shared helper so threaded-video builds free GPU state
+       * on the video thread, matching the normal teardown path. */
+      font_driver_release_renderer_state(
+            (const font_renderer_t*)font_driver,
+            font_handle, is_threaded);
    }
 
    return NULL;