8386562: JVM crashes when StackMapTable attribute is too long

[iklam]

When an unusually large StackMapTable is found in a classfile, we throw an OutOfMemoryError. Note that we don't throw a
ClassFormatException as the class file's format is correct, but the JVM cannot allocate enough memory to for this table.

The offending class is artificially generated. Normal Java classes do not have such large tables.

See https://docs.oracle.com/en/java/javase/26/docs/specs/jvms/jvms-4.html#jvms-4.7.4

StackMapTable_attribute {
    u2 attribute_name_index;
    u4 attribute_length;
    u2 number_of_entries;
    stack_map_frame entries[number_of_entries];
}

Please note that the fix is not in the class file parser but rather in the Metaspace::allocate() interface. The behavior has changed when the requested size is larger than Metaspace::max_allocation_word_size():

• Before: a debug JVM would assert(); a product JVM would abort with a guarantee()
• After: the JVM throws an OutOfMemoryError.

This makes the interface safer -- otherwise it would be tedious and error prone to require every caller to check for Metaspace::max_allocation_word_size(). Usually, the code that make the allocation does not know the actual required size (e.g., the actual allocation size of Array<b> depends on object header size as well as alignment paddings).

---

• I confirm that I make this contribution in accordance with the OpenJDK Interim AI Policy.

---

Progress

• Change must not contain extraneous whitespace
• Commit message must refer to an issue
• Change must be properly reviewed (2 reviews required, with at least 1 Reviewer, 1 Author)

Issue

• JDK-8386562: JVM crashes when StackMapTable attribute is too long (Bug - P3)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk.git pull/31493/head:pull/31493
$ git checkout pull/31493

Update a local copy of the PR:
$ git checkout pull/31493
$ git pull https://git.openjdk.org/jdk.git pull/31493/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 31493

View PR using the GUI difftool:
$ git pr show -t 31493

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/31493.diff

Using Webrev

Link to Webrev Comment

[iklam]

@tstuefe @coleenp please take a look.

[bridgekeeper]

👋 Welcome back iklam! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

[openjdk]

❗ This change is not yet ready to be integrated.
See the Progress checklist in the description for automated requirements.

[openjdk]

@iklam The following label will be automatically applied to this pull request:

• hotspot-runtime

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

[openjdk]

The total number of required reviews for this PR has been set to 2 based on the presence of this label: hotspot-runtime. This can be overridden with the /reviewers command.

[mlbridge]

Webrevs

• 00: Full (da1e479f)

[tstuefe]

I'm on PTO but can take a look next week

[sendaoYan]

Suggested change

	// Generate a StackMapTable that's larger than to Metaspace::max_allocation_word_size()
	// Generate a StackMapTable that's larger than Metaspace::max_allocation_word_size()

[sendaoYan]

Should we use Utils.adjustTimeout(COMPILE_TIMEOUT) to mutiply the timeoutFactor.