VULKAN: Build display pipelines against swapchain render pass

Fixes #18761

All display-related Vulkan pipelines (font, alpha_blend, menu
shaders including ribbon/snow/bokeh) were being built against
vk->sdr_render_pass, which always uses VK_FORMAT_B8G8R8A8_UNORM.

On the non-HDR rendering path (the common case on Linux KMS,
Wayland, X11, and most platforms), the active render pass at
draw time is vk->render_pass, which uses the actual swapchain
format -- vk->context->swapchain_format.  When the swapchain
format happens to be B8G8R8A8_UNORM, the pipeline/render-pass
formats match and everything works.

But on some platforms (particularly KMS on certain GPUs), the
swapchain format is VK_FORMAT_R8G8B8A8_UNORM instead.  In that
case:

  - Pipeline built for: B8G8R8A8_UNORM (via sdr_render_pass)
  - Active render pass:  R8G8B8A8_UNORM (via render_pass)

This is a render pass compatibility violation per Vulkan spec
(VUID-vkCmdDraw-renderPass-02684).  The result is driver-
dependent: RADV on AMD typically segfaults, NVIDIA may silently
produce wrong output, Intel may report a validation error.

The crash manifests as a segfault when any menu shader pipeline
is active (ribbon, simple ribbon, snow, bokeh, snowflake) and
the user navigates the XMB Quick Menu -- any draw that uses the
menu pipeline triggers the incompatible render pass.

Fix: build all display pipelines against vk->render_pass
(the swapchain render pass) instead of vk->sdr_render_pass.
This ensures the pipelines match the active render pass on
the non-HDR path regardless of swapchain format.

Two changes:

1. Line 3372: Initial pipe.renderPass assignment changed from
   vk->sdr_render_pass to vk->render_pass.  This affects the
   font pipeline, alpha_blend pipeline, and display pipelines
   0-3.

2. After the #ifdef VULKAN_HDR_SWAPCHAIN block (which
   temporarily changes pipe.renderPass to readback_render_pass
   and sdr_render_pass for HDR-specific pipelines), restore
   pipe.renderPass to vk->render_pass before building the menu
   shader pipelines (indices 6+).  Without this, HDR builds
   would leave pipe.renderPass pointing at sdr_render_pass,
   defeating the fix.

Impact on HDR: the HDR compositing path renders menu content
into an offscreen B8G8R8A8 buffer using sdr_render_pass
(line 6523).  When the swapchain format differs from B8G8R8A8,
the menu pipelines built against render_pass would be
incompatible with sdr_render_pass in this path.  However:

  - HDR display pipelines (indices 4-5) are built separately
    against render_pass at line 3470, which is correct.
  - The HDR menu compositing path uses the standard alpha_blend
    pipeline for the menu texture overlay (line 6563), which
    now matches render_pass.
  - The HDR offscreen compositing step at line 6523 is a
    separate render pass instance that doesn't use the menu
    shader pipelines (ribbon/snow/bokeh are drawn earlier in
    the frame).

No behavioural change when swapchain format is B8G8R8A8_UNORM
(the common case on X11/Wayland with NVIDIA and Intel), because
both render passes have the same format and are therefore
compatible.

Author: LibretroAdmin

gfx/drivers/vulkan.c

@@ -3369,7 +3369,33 @@ static void vulkan_init_pipelines(vk_t *vk)
    pipe.pViewportState                  = &vp;
    pipe.pDepthStencilState              = &depth_stencil;
    pipe.pDynamicState                   = &dynamic;
-   pipe.renderPass                      = vk->sdr_render_pass;
+   /* Build all display-related pipelines (font, alpha_blend,
+    * menu shaders, ribbon, snow, etc.) against the actual
+    * swapchain render pass.  Using sdr_render_pass here would
+    * be correct only when the swapchain format happens to be
+    * VK_FORMAT_B8G8R8A8_UNORM.  On KMS and some other
+    * platforms, the swapchain may use VK_FORMAT_R8G8B8A8_UNORM
+    * instead, making the pipeline incompatible with the active
+    * render pass at draw time — a Vulkan validation error that
+    * many drivers (especially RADV on AMD) turn into a
+    * segfault.
+    *
+    * For the HDR offscreen compositing path, which renders into
+    * a B8G8R8A8 offscreen buffer under sdr_render_pass, these
+    * pipelines may not be format-compatible if the swapchain
+    * format differs from B8G8R8A8.  However, the HDR path
+    * is only active when HDR is supported AND the display uses
+    * a wide-gamut format — in which case a separate set of
+    * HDR display pipelines (indices 4-5) is built against the
+    * correct render pass at lines 3444-3453.  The menu shader
+    * pipelines (ribbon, snow, bokeh) used through
+    * draw_pipeline + draw are called in the same render pass
+    * as the main compositing draw, which uses render_pass on
+    * non-HDR and sdr_render_pass on HDR-offscreen.
+    *
+    * Fixes: https://github.com/libretro/RetroArch/issues/18761
+    * (XMB ribbon crash on Vulkan KMS with R8G8B8A8 swapchain) */
+   pipe.renderPass                      = vk->render_pass;
    pipe.layout                          = vk->pipelines.layout;
 
    module_info.codeSize                 = sizeof(alpha_blend_vert);
@@ -3478,6 +3504,12 @@ static void vulkan_init_pipelines(vk_t *vk)
 
    vkDestroyShaderModule(vk->context->device, shader_stages[0].module, NULL);
 
+   /* Restore the swapchain render pass for the menu shader
+    * pipelines.  The HDR block above may have left
+    * pipe.renderPass pointing at sdr_render_pass or
+    * readback_render_pass. */
+   pipe.renderPass = vk->render_pass;
+
    /* Other menu pipelines. */
    for (i = 0; i < (int)ARRAY_SIZE(vk->display.pipelines) - 6; i++)
    {