HTTP/3 DATA prefix can exceed its heap allocation

[xintenseapple]

Summary

The HTTP/3 body filter allocates an 8-byte heap buffer for delayed frame
prefixes, but a DATA frame prefix can require 9 bytes: one byte for the DATA
frame type and eight bytes for a large QUIC varint length. The same fixed-size
prefix allocation pattern is also used when creating trailing HEADERS frame
prefixes.

Affected Area

HTTP/3 response filtering in:

• src/http/v3/ngx_http_v3.h
• src/http/v3/ngx_http_v3_encode.c
• src/http/v3/ngx_http_v3_filter_module.c

Steps to Reproduce

1. Build freenginx with HTTP/3, AddressSanitizer, and NGX_DEBUG_PALLOC=1 so
   request-pool allocations receive ASAN redzones.
2. Configure an HTTP/3 server with trailers enabled, for example through
   add_trailer.
3. Serve an HTTP/3 response from a content producer that passes one in-memory
   body buffer of at least 0x40000000 bytes through the output filter.
4. Request that response over HTTP/3.

The common static-file path did not trigger the large-size condition in local
testing because the copy filter split a sparse 1 GiB file into 32 KiB memory
chunks before the HTTP/3 body filter. A custom or third-party content producer
that passes one large in-memory buffer can reach the vulnerable path.

Expected Behavior

The delayed DATA and trailer HEADERS prefix allocations should account for the
actual encoded QUIC varint lengths before writing the prefixes.

Actual Behavior

NGX_HTTP_V3_VARLEN_INT_LEN is defined as four bytes:

#define NGX_HTTP_V3_VARLEN_INT_LEN                 4

However, ngx_http_v3_encode_varlen_int() emits eight bytes for values larger
than 0x3fffffff:

if (p == NULL) {
    return 8;
}

*p++ = 0xc0 | (value >> 56);
*p++ = (value >> 48);
*p++ = (value >> 40);
*p++ = (value >> 32);
*p++ = (value >> 24);
*p++ = (value >> 16);
*p++ = (value >> 8);
*p++ = value;

The delayed DATA-frame path allocates only NGX_HTTP_V3_VARLEN_INT_LEN * 2,
then writes the frame type varint followed by the payload-size varint:

chunk = ngx_palloc(r->pool, NGX_HTTP_V3_VARLEN_INT_LEN * 2);
b->end = chunk + NGX_HTTP_V3_VARLEN_INT_LEN * 2;

b->last = (u_char *) ngx_http_v3_encode_varlen_int(chunk,
                                                   NGX_HTTP_V3_FRAME_DATA);
b->last = (u_char *) ngx_http_v3_encode_varlen_int(b->last, size);

For size >= 0x40000000, this writes 1 + 8 bytes into an 8-byte allocation.

The trailer-prefix path repeats the same fixed allocation and two varint writes:

p = ngx_palloc(r->pool, NGX_HTTP_V3_VARLEN_INT_LEN * 2);
b->end = p + NGX_HTTP_V3_VARLEN_INT_LEN * 2;

b->last = (u_char *) ngx_http_v3_encode_varlen_int(p,
                                                   NGX_HTTP_V3_FRAME_HEADERS);
b->last = (u_char *) ngx_http_v3_encode_varlen_int(b->last, n);

Impact

This is a remote, configuration- and producer-dependent heap overwrite in an
HTTP/3 response path. The likely impact is worker crash or adjacent request-pool
metadata corruption when an HTTP/3 response with delayed DATA frame prefixing is
generated as a single body-filter chunk of at least 0x40000000 bytes.
Built-in static-file serving did not trigger the large chunk condition in local
testing, which reduces practical exploitability for default deployments.

Evidence

The source invariant is:

allocated prefix bytes = NGX_HTTP_V3_VARLEN_INT_LEN * 2 = 8
required DATA prefix bytes when size >= 0x40000000 = 1 + 8 = 9

ASAN reproduction against freenginx/1.31.2 confirmed the overwrite in the
HTTP/3 body filter. The debug log first showed the large chunk:

http3 chunk: 1073741824

ASAN then reported:

ERROR: AddressSanitizer: heap-buffer-overflow
WRITE of size 1
    #0 ngx_http_v3_encode_varlen_int src/http/v3/ngx_http_v3_encode.c:58
    #1 ngx_http_v3_body_filter src/http/v3/ngx_http_v3_filter_module.c:673

0xf7a36f9edf98 is located 0 bytes after 8-byte region
[0xf7a36f9edf90,0xf7a36f9edf98)
allocated by thread T0 here:
    #0 malloc
    #1 ngx_alloc src/os/unix/ngx_alloc.c:22
    #2 ngx_palloc_large src/core/ngx_palloc.c:220
    #3 ngx_palloc src/core/ngx_palloc.c:131
    #4 ngx_http_v3_body_filter src/http/v3/ngx_http_v3_filter_module.c:657

Suggested Fix

Use exact varint length accounting before allocating these prefixes:

len = ngx_http_v3_encode_varlen_int(NULL, NGX_HTTP_V3_FRAME_DATA)
      + ngx_http_v3_encode_varlen_int(NULL, size);

chunk = ngx_palloc(r->pool, len);

Apply the same exact length calculation to the trailing HEADERS prefix path. A
defensive alternative is to set NGX_HTTP_V3_VARLEN_INT_LEN to eight, but exact
length allocation matches the existing header-filter pattern and avoids relying
on a global constant.

[mdounin]

Thanks for reporting this. Looks like a valid bug in the experimental HTTP/3 code, which might be in theoretically triggered when using very large memory buffers. It seems to be already fixed in F5 NGINX by bumping NGX_HTTP_V3_VARLEN_INT_LEN to 8 (0f9f43b).

[mdounin]

Suggested patch is posted here:
https://freenginx.org/pipermail/nginx-devel/2026-June/001189.html