fix(broker): jemalloc global allocator to stop churn-driven RSS growth (#23) - #24
Merged
Conversation
glibc's malloc fragments its per-thread arenas under the broker's high connection open/close churn: a 20-minute churn soak (8 client pairs, reconnect every 500 msgs, ~1.2M reconnects) grows broker RSS +64% with no plateau and collapses throughput -76% (719k -> 170k msg/s). jemalloc keeps both flat (RSS +12%, throughput -11%) — established by an LD_PRELOAD A/B on the identical workload and confirmed here with the compiled-in build (the churn soak passes, RSS plateaus). This is a glibc allocator behavior under churn, not a leak in broker logic (profiling ruled out connection/consumer/fd/task/unacked leaks). tikv-jemallocator is wired as `#[global_allocator]` in the `ramqp-brokerd` bin, behind a default-on `jemalloc` feature; `--no-default-features` opts out on targets jemalloc does not support. Only the daemon sets the allocator — the library never imposes it on downstream users. ramqp-broker 0.8.28 -> 0.8.29.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Make
ramqp-brokerduse jemalloc as its global allocator. Closes #23.(This is the follow-on to #22, which merged before this commit landed on the
branch — the deps (#20/#21) and test-battery work went in, this did not.)
Why
The 20-minute connection-churn soak added in #22 surfaced a severe degradation:
broker RSS climbs +64% with no plateau and throughput collapses −76%
(719k → 170k msg/s) over the run. An
LD_PRELOADA/B on the identical workloadpinned the cause to glibc malloc arena fragmentation under churn — not a
broker logic leak (profiling ruled out connection/consumer/fd/task/unacked
leaks; a debug pass showed the queue actor's unacked scan is O(0) at close). With
jemalloc, RSS plateaus (+12%) and throughput holds (−11%).
Change
tikv-jemallocatorwired as#[global_allocator]in theramqp-brokerdbin,behind a default-on
jemallocfeature.--no-default-featuresopts out ontargets jemalloc doesn't support.
downstream users.
Validation
cargo fmt --checkclean;clippy --all-targets --all-features -D warningsclean.LD_PRELOAD) churn soak passes: RSS plateaus (+14.5%),throughput holds (−13%) — matching the preload A/B, unlike glibc's failure.
ramqp-broker0.8.28 → 0.8.29.