gfx/d3d9cg: Recover from shader compile failures instead of black-

screening

Three closely-related issues stacked on top of each other made any
single-pass-or-later compile failure inside a multi-pass preset
permanently brick the driver until restart:

1. d3d9_cg_renderchain_add_pass discarded d3d9_cg_load_program's
   return value.  When a pass like mdapt-pass3.cg failed to compile
   (C5109 domain-conflict, etc.), the pass was still appended to the
   chain with vprg/fprg = NULL.  The render path then called
   cgD3D9BindProgram(NULL) every frame, which leaves the device in
   an unspecified state -- on the user's GPU, that's a black core
   frame with menu/widgets/overlay still drawing fine (those use
   fixed-function or stock-blend, not the renderchain).

2. d3d9_cg_load_program's error: label leaked the partially-created
   Cg program(s).  In the cgD3D9LoadProgram failure case, pass->fprg
   was a valid Cg program object (compile succeeded, GPU-load did
   not), and any subsequent code path that didn't NULL-check would
   happily try to bind it.  Worse, the leaked program objects pin
   the surrounding context until cgDestroyContext runs.

3. d3d9_cg_set_shader's failure handling left d3d->renderchain_data
   pointing at a half-built chain (whatever passes succeeded before
   the failure, plus the bad pass with NULL programs after #1).  The
   driver kept rendering through that chain.  Even reapplying the
   stock shader from the menu wouldn't recover, because the menu's
   "Apply Shader" path also goes through set_shader -> process_shader
   -> restore, hitting the same broken state if anything along the
   way faulted.

Fix all three:

- add_pass now memsets the local pass struct (so the error path has
  a well-defined struct to walk), checks load_program's return, and
  funnels every failure through a single error: label that releases
  the Cg programs, vertex declaration, vertex buffer, texture, and
  attrib_map it created before the failure point.

- load_program's error: label cgDestroyProgram()s pass->vprg/fprg
  (whichever is non-NULL) and zeroes them, so callers that ignore
  the return value still see a clean pass struct.

- set_shader, on process_shader-or-restore failure, frees
  d3d->shader_path, runs process_shader again to populate the stock
  defaults, and calls restore once more to rebuild the chain
  against the stock shader.  The original return value (false) is
  preserved, so the menu still surfaces the failure to the user --
  they just don't get stuck on a black screen anymore.

The init-ordering fix (call process_shader before initialize on
the first init path so init_chain doesn't read a calloc-zeroed
fbo struct) is also included, since both regressions would have
needed to be tested together anyway.

Tested with snes9x2010 + DKC2 + sequence: stock -> testie-crt
(7-pass, works) -> mdapt+xbr-hybrid+aa (mdapt-pass3 C5109 failure)
-> stock (recovered).  Pre-fix the third step stayed black; post-
fix it goes back to plain stock rendering.

Author: LibretroAdmin

gfx/drivers/d3d9cg.c

@@ -2045,6 +2045,22 @@ static bool d3d9_cg_load_program(cg_renderchain_t *chain,
    free(listing_f);
    free(listing_v);
 
+   /* Drop any partially-created program objects so add_pass doesn't
+    * inherit half-loaded handles -- previously these leaked, and on
+    * cgD3D9LoadProgram failure pass->fprg was a valid Cg object
+    * (just not GPU-loaded) which the renderer would happily try to
+    * bind every frame. */
+   if (pass->fprg)
+   {
+      cgDestroyProgram((CGprogram)pass->fprg);
+      pass->fprg = NULL;
+   }
+   if (pass->vprg)
+   {
+      cgDestroyProgram((CGprogram)pass->vprg);
+      pass->vprg = NULL;
+   }
+
    return false;
 }
 
@@ -2697,17 +2713,27 @@ static bool d3d9_cg_renderchain_add_pass(void *data, const struct LinkInfo *info
    cg_renderchain_t *chain     = (cg_renderchain_t*)data;
    d3d9_cg_renderchain_t *_chain = &chain->chain;
 
+   /* Zero everything first so the bail-out path has a well-defined
+    * struct to clean up: load_program may leave vprg/fprg as NULL
+    * on failure (stack garbage otherwise), and the post-fail cleanup
+    * in renderchain_free / set_shader fallback walks these fields. */
+   memset(&pass, 0, sizeof(pass));
    pass.info                   = *info;
-   pass.last_width             = 0;
-   pass.last_height            = 0;
    pass.attrib_map             = (struct unsigned_vector_list*)
       unsigned_vector_list_new();
    pass.pool                   = D3DPOOL_DEFAULT;
 
-   d3d9_cg_load_program((cg_renderchain_t*)chain, &pass, info->pass->source.path, true);
+   /* Cg compile/link failure was previously swallowed here, leaving
+    * vprg/fprg NULL but the pass still appended to the chain.  The
+    * render path then bound NULL programs every frame -> black
+    * screen with no recovery short of restarting the driver.
+    * Bail now and let init_chain unwind. */
+   if (!d3d9_cg_load_program((cg_renderchain_t*)chain, &pass,
+            info->pass->source.path, true))
+      goto error;
 
    if (!d3d9_cg_renderchain_init_shader_fvf(_chain, &pass))
-      return false;
+      goto error;
 
    if (!SUCCEEDED(IDirect3DDevice9_CreateVertexBuffer(
                _chain->dev, 4 * sizeof(struct Vertex),
@@ -2717,7 +2743,7 @@ static bool d3d9_cg_renderchain_add_pass(void *data, const struct LinkInfo *info
       vertbuf = NULL;
 
    if (!vertbuf)
-      return false;
+      goto error;
 
    pass.vertex_buf = vertbuf;
 
@@ -2735,7 +2761,7 @@ static bool d3d9_cg_renderchain_add_pass(void *data, const struct LinkInfo *info
    }
 
    if (!tex)
-      return false;
+      goto error;
 
    pass.tex        = tex;
 
@@ -2747,6 +2773,25 @@ static bool d3d9_cg_renderchain_add_pass(void *data, const struct LinkInfo *info
    shader_pass_vector_list_append(_chain->passes, pass);
 
    return true;
+
+error:
+   /* Release whatever we managed to create before the failure point.
+    * pass is a stack local that's about to be discarded, so anything
+    * the chain doesn't take ownership of must be freed here. */
+   if (pass.vprg)
+      cgDestroyProgram((CGprogram)pass.vprg);
+   if (pass.fprg)
+      cgDestroyProgram((CGprogram)pass.fprg);
+   if (pass.vertex_decl)
+      IDirect3DVertexDeclaration9_Release(
+            (LPDIRECT3DVERTEXDECLARATION9)pass.vertex_decl);
+   if (vertbuf)
+      IDirect3DVertexBuffer9_Release(vertbuf);
+   if (tex)
+      IDirect3DTexture9_Release(tex);
+   if (pass.attrib_map)
+      free(pass.attrib_map);
+   return false;
 }
 
 static void d3d9_cg_renderchain_calc_and_set_shader_mvp(
@@ -3793,7 +3838,25 @@ static bool d3d9_cg_set_shader(void *data,
 
    if (!d3d9_cg_process_shader(d3d) || !d3d9_cg_restore(d3d))
    {
-      RARCH_ERR("[D3D9 Cg] Failed to set shader.\n");
+      /* The new preset failed to compile/link, and restore() left
+       * d3d->renderchain_data pointing at a half-built chain whose
+       * passes have NULL vprg/fprg.  Rendering through that chain
+       * produces a black screen with no recovery path.
+       *
+       * Drop the bad preset and rebuild the chain against the stock
+       * shader so the user lands back in a working state.  Returning
+       * false tells the caller their requested preset didn't take. */
+      RARCH_ERR("[D3D9 Cg] Failed to set shader, falling back to stock.\n");
+
+      if (d3d->shader_path)
+      {
+         free(d3d->shader_path);
+         d3d->shader_path = NULL;
+      }
+
+      if (!d3d9_cg_process_shader(d3d) || !d3d9_cg_restore(d3d))
+         RARCH_ERR("[D3D9 Cg] Stock-shader fallback also failed.\n");
+
       return false;
    }
 
@@ -3884,6 +3947,24 @@ static bool d3d9_cg_init_internal(d3d9_video_t *d3d,
 
    d3d->video_info = *info;
 
+   /* Populate d3d->shader with stock-shader defaults (or parse a
+    * preset if d3d->shader_path is set) BEFORE the renderchain
+    * is built.  d3d9_cg_initialize -> d3d9_cg_init_chain reads
+    * d3d->shader.pass[0].fbo to size the final pass; on the first
+    * init the d3d_video_t was just calloc'd, so without this call
+    * fbo.type_x/y are RARCH_SCALE_INPUT (=0) with scale=0.0f, and
+    * the renderchain ends up emitting a zero-sized quad — no core
+    * frame on screen until something later forces a chain rebuild
+    * via d3d9_cg_set_shader (which does call process_shader).
+    *
+    * d3d9hlsl gets away without this because its vertex positions
+    * are constant [0,1], so out_width/out_height being zero just
+    * means a uselessly-positioned-but-still-on-screen quad.  The
+    * Cg renderchain uses pixel-space vertices (0..out_width,
+    * 0..out_height) and collapses to a point when those are zero. */
+   if (!d3d9_cg_process_shader(d3d))
+      return false;
+
    if (!d3d9_cg_initialize(d3d, &d3d->video_info))
       return false;