gfx/drivers/d3d9hlsl: NULL-check realloc in d3d9_hlsl_convert_macro_loops

Six realloc sites inside d3d9_hlsl_convert_macro_loops had no NULL
check at all, just

    while (opos + N >= cap)
    { cap *= 2; out = (char*)realloc(out, cap); }
    memcpy(out + opos, ...);

On OOM realloc returns NULL, the self-assign overwrites the only
pointer to the old buffer (leaking it), and the very next line
memcpys into NULL+opos - a segfault crash during Cg-style HLSL
shader preprocessing.

This function rewrites macro-unrolled loops in Cg shaders into
HLSL [loop] form.  It runs during shader-preset compilation, so
every slang preset load that contains a qualifying macro
(pattern: single-param function-like #define called N times with
monotonically increasing integer arguments, N >= 4) hits this
code path.  Shader presets are user-provided and can be
arbitrary size; a memory-tight system loading a big CRT-Royale-
style preset at the wrong moment will crash here rather than
fail gracefully.

The other five callers inside the function (lines ~5749, 5760,
5772, 5786, 5795, 5803 pre-patch) all had the same bug.  All six
are fixed by switching to the realloc-into-tmp idiom:

    while (opos + N >= cap)
    {
       char *tmp;
       cap *= 2;
       if (!(tmp = (char*)realloc(out, cap)))
       { free(out); return NULL; }
       out = tmp;
    }

This also plugs the self-assign leak that the pre-patch form
would have triggered on OOM - since we free(out) on the failure
path, the old buffer is reclaimed before we return NULL.

Thread-safety: unchanged.  Shader compilation runs on whichever
thread loads the preset; d3d9_hlsl_convert_macro_loops does not
touch any shared state beyond the const-source argument.

Reachability: every Cg-style macro-expanded shader preset load
on the D3D9-HLSL backend.  The worst of the crash modes (writes
past end of buffer on OOM) has been latent for as long as this
preprocessing pass has been in tree.

Scope: this commit only fixes the six sites in
d3d9_hlsl_convert_macro_loops.  The surrounding 8200-line file
has ~13 more realloc-assign-self patterns in other HLSL-fixup
functions (d3d9_hlsl_preprocess_includes, d3d9_hlsl_add_struct_
semantics, d3d9_hlsl_decompose_struct_samplers,
d3d9_hlsl_fixup_cg_source, and the d3d9_hlsl_buf_append helper)
that have the same leak-on-OOM pattern but do NULL-check after
the realloc so they don't crash - just leak and return NULL.
Those are worth fixing too but are a separate follow-up; this
commit focuses on the six crash sites.

Author: LibretroAdmin

gfx/drivers/d3d9hlsl.c

@@ -5746,7 +5746,13 @@ static char *d3d9_hlsl_convert_macro_loops(const char *source)
                                        || macro_define_start[comment_len-1] == '\r'))
                                  comment_len--;
                               while (opos + dll + comment_len + 10 >= cap)
-                              { cap *= 2; out = (char*)realloc(out, cap); }
+                              {
+                                 char *tmp;
+                                 cap *= 2;
+                                 if (!(tmp = (char*)realloc(out, cap)))
+                                 { free(out); return NULL; }
+                                 out = tmp;
+                              }
                               memcpy(out + opos, dl, dll); opos += dll;
                               memcpy(out + opos, macro_define_start, comment_len);
                               opos += comment_len;
@@ -5757,7 +5763,13 @@ static char *d3d9_hlsl_convert_macro_loops(const char *source)
                            {
                               size_t mid = (size_t)(seq_start - macro_define_end);
                               while (opos + mid + 1 >= cap)
-                              { cap *= 2; out = (char*)realloc(out, cap); }
+                              {
+                                 char *tmp;
+                                 cap *= 2;
+                                 if (!(tmp = (char*)realloc(out, cap)))
+                                 { free(out); return NULL; }
+                                 out = tmp;
+                              }
                               memcpy(out + opos, macro_define_end, mid);
                               opos += mid;
                            }
@@ -5769,7 +5781,13 @@ static char *d3d9_hlsl_convert_macro_loops(const char *source)
                                     "[loop] for (int %s = 0; %s <= %d; %s++) {\n",
                                     macro_param, macro_param, max_seq, macro_param);
                               while (opos + lh_len + 1 >= cap)
-                              { cap *= 2; out = (char*)realloc(out, cap); }
+                              {
+                                 char *tmp;
+                                 cap *= 2;
+                                 if (!(tmp = (char*)realloc(out, cap)))
+                                 { free(out); return NULL; }
+                                 out = tmp;
+                              }
                               memcpy(out + opos, loop_header, lh_len);
                               opos += lh_len;
                            }
@@ -5783,7 +5801,13 @@ static char *d3d9_hlsl_convert_macro_loops(const char *source)
                                        && bp_src[1] == '\n')
                                  { bp_src += 2; continue; }
                                  while (opos + 2 >= cap)
-                                 { cap *= 2; out = (char*)realloc(out, cap); }
+                                 {
+                                    char *tmp;
+                                    cap *= 2;
+                                    if (!(tmp = (char*)realloc(out, cap)))
+                                    { free(out); return NULL; }
+                                    out = tmp;
+                                 }
                                  out[opos++] = *bp_src++;
                               }
                            }
@@ -5792,15 +5816,27 @@ static char *d3d9_hlsl_convert_macro_loops(const char *source)
                            {
                               const char *cl = "\n}\n";
                               while (opos + 4 >= cap)
-                              { cap *= 2; out = (char*)realloc(out, cap); }
+                              {
+                                 char *tmp;
+                                 cap *= 2;
+                                 if (!(tmp = (char*)realloc(out, cap)))
+                                 { free(out); return NULL; }
+                                 out = tmp;
+                              }
                               memcpy(out + opos, cl, 3); opos += 3;
                            }
 
                            /* Copy everything after the macro call run */
                            {
                               size_t tail = src_len_full - (size_t)(seq_end_ptr - source);
                               while (opos + tail + 1 >= cap)
-                              { cap *= 2; out = (char*)realloc(out, cap); }
+                              {
+                                 char *tmp;
+                                 cap *= 2;
+                                 if (!(tmp = (char*)realloc(out, cap)))
+                                 { free(out); return NULL; }
+                                 out = tmp;
+                              }
                               memcpy(out + opos, seq_end_ptr, tail);
                               opos += tail;
                            }