Skip to content

Fix shutdown race condition to avoid double processing and unnecessary dead-set publishes#288

Merged
Vruttant1403 merged 2 commits into
gojek:masterfrom
vsr625:fix/rabbitmq-shutdown-drain-in-flight-messages
Jun 16, 2026
Merged

Fix shutdown race condition to avoid double processing and unnecessary dead-set publishes#288
Vruttant1403 merged 2 commits into
gojek:masterfrom
vsr625:fix/rabbitmq-shutdown-drain-in-flight-messages

Conversation

@vsr625

@vsr625 vsr625 commented Jun 3, 2026

Copy link
Copy Markdown

During shutdown, consumer connections could be torn down while messages were
still being processed, leaving those messages unacked. RabbitMQ then redelivered
them, causing the same messages to be processed again after restart.

Track in-flight messages with a counter (incremented before processing and
decremented in a finally block) and, when stopping the consumers, wait for the
in-flight handlers to finish after cancelling subscriptions but before the
connection is closed. The wait is bounded by a configurable drain timeout
([:shutdown :drain-timeout-ms], default 5000ms).

Also fix the JVM shutdown hook to run (stop ...) and (shutdown-agents)
sequentially via do; previously it invoked the result of stop as a function.

Adds tests covering in-flight tracking, wait-for-in-flight-messages (including
the configured-timeout path), and the drain step in the consumers :stop lifecycle.

@vsr625
vsr625 force-pushed the fix/rabbitmq-shutdown-drain-in-flight-messages branch from 4b123a6 to baa43d9 Compare June 5, 2026 05:44
@vsr625
vsr625 marked this pull request as ready for review June 5, 2026 05:52
Subramani and others added 2 commits June 16, 2026 12:39
…ssing

During shutdown, consumer connections could be torn down while messages were
still being processed, leaving those messages unacked. RabbitMQ then redelivered
them, causing the same messages to be processed again after restart.

Track in-flight messages with a counter (incremented before processing and
decremented in a finally block) and, when stopping the consumers, wait for the
in-flight handlers to finish after cancelling subscriptions but before the
connection is closed. The wait is bounded by a configurable drain timeout
([:shutdown :drain-timeout-ms], default 5000ms).

Also fix the JVM shutdown hook to run (stop ...) and (shutdown-agents)
sequentially via do; previously it invoked the result of stop as a function.

Adds tests covering in-flight tracking, wait-for-in-flight-messages (including
the configured-timeout path), and the drain step in the consumers :stop lifecycle.

Co-authored-by: Cursor <[email protected]>
@Vruttant1403
Vruttant1403 force-pushed the fix/rabbitmq-shutdown-drain-in-flight-messages branch from 21c718e to 59ae2ee Compare June 16, 2026 07:09
@Vruttant1403
Vruttant1403 merged commit 83cca31 into gojek:master Jun 16, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants