Skip to content

Stop all application states in dependency order on shutdown#293

Merged
Vruttant1403 merged 1 commit into
gojek:masterfrom
vsr625:fix/unified-shutdown-ordering
Jul 20, 2026
Merged

Stop all application states in dependency order on shutdown#293
Vruttant1403 merged 1 commit into
gojek:masterfrom
vsr625:fix/unified-shutdown-ordering

Conversation

@vsr625

@vsr625 vsr625 commented Jul 14, 2026

Copy link
Copy Markdown

Previously each mode's stop-fn tore down its own subset of states, and modes were stopped in sorted order. :api-server's stop closed the shared consumer-connection before :worker cancelled its RabbitMQ subscribers and drained in-flight messages. Handlers mid-processing then failed to ack on the closed channel, publishing successfully processed messages to the dead set while the broker requeued them, causing double processing. Shared states (kafka-producers, producer-connection, channel-pool) had the same problem: whichever mode stopped first closed them while other modes' flows were still publishing through them.

Replace the per-mode stop dispatch with a single stop-application-states fn that stops every state in dependency order regardless of modes: stop ingestion (server, streams, batch consumers), then cancel RabbitMQ subscribers and drain in-flight messages, and only then close the connections and producers those handlers publish through. Stopping a never-started state is a no-op in mount, so the sequence is safe for every mode combination.

Each state is stopped with its own mount/stop call because mount orders a multi-state call by namespace load order, not by argument order; separate calls are the only way to guarantee this dependency ordering.

With stop no longer dispatched per mode, drop the now-dead :stop-fn entries from valid-modes-fns. The public stop-* fns are kept as-is for backward compatibility.

@vsr625
vsr625 force-pushed the fix/unified-shutdown-ordering branch 2 times, most recently from 7c0dc48 to 25221a5 Compare July 20, 2026 08:59
Previously each mode's stop-fn tore down its own subset of states, and
modes were stopped in sorted order. :api-server's stop closed the shared
consumer-connection before :worker cancelled its RabbitMQ subscribers and
drained in-flight messages. Handlers mid-processing then failed to ack on
the closed channel, publishing successfully processed messages to the dead
set while the broker requeued them, causing double processing. Shared
states (kafka-producers, producer-connection, channel-pool) had the same
problem: whichever mode stopped first closed them while other modes' flows
were still publishing through them.

Replace the per-mode stop dispatch with a single stop-application-states fn
that stops every state in dependency order regardless of modes: stop
ingestion (server, streams, batch consumers), then cancel RabbitMQ
subscribers and drain in-flight messages, and only then close the
connections and producers those handlers publish through. Stopping a
never-started state is a no-op in mount, so the sequence is safe for every
mode combination.

Each state is stopped with its own mount/stop call because mount orders a
multi-state call by namespace load order, not by argument order; separate
calls are the only way to guarantee this dependency ordering.

With stop no longer dispatched per mode, drop the now-dead :stop-fn entries
from valid-modes-fns. The public stop-* fns are kept as-is for backward
compatibility.
@vsr625
vsr625 force-pushed the fix/unified-shutdown-ordering branch from 25221a5 to 841286f Compare July 20, 2026 09:02
@Vruttant1403
Vruttant1403 merged commit 5a35774 into gojek:master Jul 20, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants