TODO list: Move the following function calls into work packets and run them in the right work buckets, or remove them if they are not necessary.
The problem
mmtk_resume_mutators currently does much things other than actually resuming mutators.
In the master branch for OpenJDK 11:
static void mmtk_resume_mutators(void *tls) {
nmethod::oops_do_marking_epilogue();
// ClassLoaderDataGraph::purge();
CodeCache::gc_epilogue();
JvmtiExport::gc_epilogue();
#if COMPILER2_OR_JVMCI
DerivedPointerTable::update_pointers();
#endif
// ...In the jdk-21.0.9+3-mmtk for OpenJDK 21:
static void mmtk_resume_mutators(void *tls) {
nmethod::oops_do_marking_epilogue();
// ClassLoaderDataGraph::purge();
#if COMPILER2_OR_JVMCI
DerivedPointerTable::update_pointers();
#endif
{
CodeBlobFixRelocationClosure cb_cl;
CodeCache::blobs_do(&cb_cl);
}
// ...Those "epilogues" are actually proper finalization work of a GC. For example, in OpenJDK 11, CodeCache::gc_epilogue cleans up the "scavenge root nmethods", and in OpenJDK 21 CodeBlobFixRelocationClosure calls nm->fix_oop_relocations for nmethods with forwarded embedded pointers.
The performance impact is OK for OpenJDK 11, but on OpenJDK 21, we observed an abnormal latency between the invocation of mmtk_resume_mutator and the actual resumption of mutators, as shown in the timeline below.
The delay is caused by fix_oop_relocations invoked for every nmethod in the code cache. One consequence of this is that the STW time is systematically under-measured, and the mutator time is systematically over-measured. And mmtk_resume_mutators is run by a single thread, and cannot make use of parallel GC workers.
This can be fixed by moving the fix_oop_relocations invocations to work packets in the work bucket after Closure (i.e. SoftRefClosure for now) and only do this during copying GC, and only do this for young nmethods during nursery GC, as shown below.
Similarly, there are "prologues" executed in mmtk_stop_all_mutators, but they are less problematic than the code cache epilogue in the OpenJDK 21 binding. Here is a timeline for OpenJDK 11. ClassLoaderDataGraph::clear_claimed_marks(); is taking more time than other prologues, but not as much as the invocation that actually stops mutators.
We should systematically inspect all the "epilogues" we have, and move them to proper GC phases, usually reference-processing or reference-updating phases.
Details
ClassLoaderDataGraph::clear_claimed_marks();
Each ClassLoaderData has a _claim field. When iterating over all ClassLoaderData, it allows "claiming" those instances so that they won't be visited again when iterating over all ClassLoaderData again. This is useful for concurrent GCs so that they can scan ClassLoaderDataGraph roots multiple times and not visit ClassLoaderData instance that have been visited before. It is also useful for parallel GCs so that multiple GC workers can avoid processing the same instance at the same time. In OpenJDK 21, this field is changed from boolean to a bit field so that it can be claimed for multiple different reasons in a GC.
ClassLoaderDataGraph::clear_claimed_marks(); simply clears the _claim field of all ClassLoaderData instances. It is similar to clearing per-object metadata during Prepare when using MMTk. The OpenJDK documentation says
The claimed marks of the CLDs in the ClassLoaderDataGraph are cleared
outside a safepoint and without locking the ClassLoaderDataGraph_lock.
This is required to avoid a deadlock between concurrent GC threads and safepointing.
That's probably why we called it outside STW, too.
When using MMTk, we never "claim" any ClassLoaderData. So calling ClassLoaderDataGraph::clear_claimed_marks(); is unnecessary.
It may be useful if there are too many ClassLoaderData instances (There actually are! ClassLoaderDataGraph roots are the second most time-consuming root-scanning task in mmtk-openjdk, only after code cache roots.), and we need parallel root processing. But it is obviously easier to follow the linked list once and extract its nodes into vectors, rather than letting MMTk's GC worker threads compete to "claim" each individual node. Alternatively, we can implement our own Rust-side cache using vectors instead of linked lists, which is clearly more friendly to parallelism.
CodeCache::gc_prologue and CodeCache::gc_epilogue (OpenJDK 11)
CodeCache::gc_prologue does nothing at all.
CodeCache::gc_epilogue calls prune_scavenge_root_nmethods. It removes "zombie" and "unloaded" nmethods from the list of "scavenge root nmethods". Note: An nmethod is added to the list of "scavenge root" by certain GC algorithms when the nmethod is created. We may remove this, too, because when using MMTk, we never add it to the "scavenge root" list.
DerivedPointerTable::clear() and DerivedPointerTable::update_pointers()
According to the documentation:
Derived pointer support. This table keeps track of all derived points on a
stack. It is cleared before each scavenge/GC. During the traversal of all
oops, it is filled in with references to all locations that contains a
derived oop (assumed to be very few). When the GC is complete, the derived
pointers are updated based on their base pointers new value and an offset.
In OpenJDK, a GC discovers derived pointers during transitive closure. It calls DerivedPointerTable::clear() at the beginning of a GC, add discover derived pointers using DerivedPointerTable::add, and call DerivedPointerTable::update_pointers() in the end of a GC to update derived pointers.
The preprocessor directive #if COMPILER2_OR_JVMCI indicates that it is only needed when using C2 or JVMCI.
Derived pointer are only added during stack root scanning. So it is sufficient to call DerivedPointerTable::clear() before scanning any mutator. According to our current mmtk-core-to-binding API, mmtk_stop_all_mutators seems to be the best place to call DerivedPointerTable::clear() because mmtk-core may start scanning mutator while mutators are gradually stopped.
DerivedPointerTable::update_pointers() visits all recorded derived pointer locations, computes their base pointers, find the new address of the object if moved, and update the derived pointer location, preserving the offset from the base pointer to the derived pointer. Because base pointer locations are enumerated during root scanning, and they are updated during Closure (scavenging plans) or SecondRoots (MarkCompact), it is sufficient to call DerivedPointerTable::update_pointers() after those two work bucket stages (i.e. SoftRefClosure and RefForwarding, respectively), just like fixing nmethod relocations.
nmethod::oops_do_marking_prologue(); and nmethod::oops_do_marking_epilogue();
In OpenJDK's stock GC algorithms, GC worker threads compete to "claim" nmethods to process so that two GC worker threads don't work on the same nmethod at the same time. For this, every nmethod has a field _oops_do_mark_link which is nullptr when not claimed, but linked into a linked list after claimed. In nmethod::oops_do_marking_epilogue();, it visits all claimed nmethods and reverts them back to the unclaimed state by clearing their _oops_do_mark_link fields. The role of the _oops_do_mark_link field is like the mark bit metadata for nmethod, and clearing the field is like clearing the mark bits during Release. nmethod::oops_do_marking_prologue(); verifies that no nmethods have been "claimed".
When using MMTk, we never modify the _oops_do_mark_link field, so nmethod::oops_do_marking_epilogue(); is effectively a no-op. We can safely remove the nmethod::oops_do_marking_epilogue(); statement. We also need to remove the nmethod::oops_do_marking_prologue(); statement in mmtk_stop_all_mutators which pairs with the epilogue.
TODO list: Move the following function calls into work packets and run them in the right work buckets, or remove them if they are not necessary.
ClassLoaderDataGraph::clear_claimed_marks();(can be removed)CodeCache::gc_prologue();(can be removed)DerivedPointerTable::clear();(keep inmmtk_stop_all_mutators)nmethod::oops_do_marking_prologue();(can be removed)nmethod::oops_do_marking_epilogue();(can be removed)CodeCache::gc_epilogue();(can be removed)JvmtiExport::gc_epilogue();DerivedPointerTable::update_pointers();(move toSoftRefClosureorRefForwarding)ClassLoaderDataGraph::clear_claimed_marks();(can be removed)DerivedPointerTable::clear();(keep inmmtk_stop_all_mutators)nmethod::oops_do_marking_prologue();(can be removed)nmethod::oops_do_marking_epilogue();(can be removed)DerivedPointerTable::update_pointers();(move toSoftRefClosureorRefForwarding)SoftRefClosureorRefForwarding)The problem
mmtk_resume_mutatorscurrently does much things other than actually resuming mutators.In the
masterbranch for OpenJDK 11:In the
jdk-21.0.9+3-mmtkfor OpenJDK 21:Those "epilogues" are actually proper finalization work of a GC. For example, in OpenJDK 11,
CodeCache::gc_epiloguecleans up the "scavenge root nmethods", and in OpenJDK 21CodeBlobFixRelocationClosurecallsnm->fix_oop_relocationsfor nmethods with forwarded embedded pointers.The performance impact is OK for OpenJDK 11, but on OpenJDK 21, we observed an abnormal latency between the invocation of
mmtk_resume_mutatorand the actual resumption of mutators, as shown in the timeline below.The delay is caused by
fix_oop_relocationsinvoked for every nmethod in the code cache. One consequence of this is that the STW time is systematically under-measured, and the mutator time is systematically over-measured. Andmmtk_resume_mutatorsis run by a single thread, and cannot make use of parallel GC workers.This can be fixed by moving the
fix_oop_relocationsinvocations to work packets in the work bucket afterClosure(i.e.SoftRefClosurefor now) and only do this during copying GC, and only do this for young nmethods during nursery GC, as shown below.Similarly, there are "prologues" executed in
mmtk_stop_all_mutators, but they are less problematic than the code cache epilogue in the OpenJDK 21 binding. Here is a timeline for OpenJDK 11.ClassLoaderDataGraph::clear_claimed_marks();is taking more time than other prologues, but not as much as the invocation that actually stops mutators.We should systematically inspect all the "epilogues" we have, and move them to proper GC phases, usually reference-processing or reference-updating phases.
Details
ClassLoaderDataGraph::clear_claimed_marks();Each
ClassLoaderDatahas a_claimfield. When iterating over allClassLoaderData, it allows "claiming" those instances so that they won't be visited again when iterating over allClassLoaderDataagain. This is useful for concurrent GCs so that they can scan ClassLoaderDataGraph roots multiple times and not visitClassLoaderDatainstance that have been visited before. It is also useful for parallel GCs so that multiple GC workers can avoid processing the same instance at the same time. In OpenJDK 21, this field is changed from boolean to a bit field so that it can be claimed for multiple different reasons in a GC.ClassLoaderDataGraph::clear_claimed_marks();simply clears the_claimfield of allClassLoaderDatainstances. It is similar to clearing per-object metadata during Prepare when using MMTk. The OpenJDK documentation saysThat's probably why we called it outside STW, too.
When using MMTk, we never "claim" any
ClassLoaderData. So callingClassLoaderDataGraph::clear_claimed_marks();is unnecessary.It may be useful if there are too many
ClassLoaderDatainstances (There actually are! ClassLoaderDataGraph roots are the second most time-consuming root-scanning task in mmtk-openjdk, only after code cache roots.), and we need parallel root processing. But it is obviously easier to follow the linked list once and extract its nodes into vectors, rather than letting MMTk's GC worker threads compete to "claim" each individual node. Alternatively, we can implement our own Rust-side cache using vectors instead of linked lists, which is clearly more friendly to parallelism.CodeCache::gc_prologueandCodeCache::gc_epilogue(OpenJDK 11)CodeCache::gc_prologuedoes nothing at all.CodeCache::gc_epiloguecallsprune_scavenge_root_nmethods. It removes "zombie" and "unloaded" nmethods from the list of "scavenge root nmethods". Note: An nmethod is added to the list of "scavenge root" by certain GC algorithms when the nmethod is created. We may remove this, too, because when using MMTk, we never add it to the "scavenge root" list.DerivedPointerTable::clear()andDerivedPointerTable::update_pointers()According to the documentation:
In OpenJDK, a GC discovers derived pointers during transitive closure. It calls
DerivedPointerTable::clear()at the beginning of a GC, add discover derived pointers usingDerivedPointerTable::add, and callDerivedPointerTable::update_pointers()in the end of a GC to update derived pointers.The preprocessor directive
#if COMPILER2_OR_JVMCIindicates that it is only needed when using C2 or JVMCI.Derived pointer are only added during stack root scanning. So it is sufficient to call
DerivedPointerTable::clear()before scanning any mutator. According to our current mmtk-core-to-binding API,mmtk_stop_all_mutatorsseems to be the best place to callDerivedPointerTable::clear()because mmtk-core may start scanning mutator while mutators are gradually stopped.DerivedPointerTable::update_pointers()visits all recorded derived pointer locations, computes their base pointers, find the new address of the object if moved, and update the derived pointer location, preserving the offset from the base pointer to the derived pointer. Because base pointer locations are enumerated during root scanning, and they are updated during Closure (scavenging plans) or SecondRoots (MarkCompact), it is sufficient to callDerivedPointerTable::update_pointers()after those two work bucket stages (i.e.SoftRefClosureandRefForwarding, respectively), just like fixing nmethod relocations.nmethod::oops_do_marking_prologue();andnmethod::oops_do_marking_epilogue();In OpenJDK's stock GC algorithms, GC worker threads compete to "claim" nmethods to process so that two GC worker threads don't work on the same nmethod at the same time. For this, every nmethod has a field
_oops_do_mark_linkwhich isnullptrwhen not claimed, but linked into a linked list after claimed. Innmethod::oops_do_marking_epilogue();, it visits all claimed nmethods and reverts them back to the unclaimed state by clearing their_oops_do_mark_linkfields. The role of the_oops_do_mark_linkfield is like the mark bit metadata fornmethod, and clearing the field is like clearing the mark bits during Release.nmethod::oops_do_marking_prologue();verifies that no nmethods have been "claimed".When using MMTk, we never modify the
_oops_do_mark_linkfield, sonmethod::oops_do_marking_epilogue();is effectively a no-op. We can safely remove thenmethod::oops_do_marking_epilogue();statement. We also need to remove thenmethod::oops_do_marking_prologue();statement inmmtk_stop_all_mutatorswhich pairs with the epilogue.