Stop all application states in dependency order on shutdown#293
Merged
Conversation
vsr625
force-pushed
the
fix/unified-shutdown-ordering
branch
2 times, most recently
from
July 20, 2026 08:59
7c0dc48 to
25221a5
Compare
Previously each mode's stop-fn tore down its own subset of states, and modes were stopped in sorted order. :api-server's stop closed the shared consumer-connection before :worker cancelled its RabbitMQ subscribers and drained in-flight messages. Handlers mid-processing then failed to ack on the closed channel, publishing successfully processed messages to the dead set while the broker requeued them, causing double processing. Shared states (kafka-producers, producer-connection, channel-pool) had the same problem: whichever mode stopped first closed them while other modes' flows were still publishing through them. Replace the per-mode stop dispatch with a single stop-application-states fn that stops every state in dependency order regardless of modes: stop ingestion (server, streams, batch consumers), then cancel RabbitMQ subscribers and drain in-flight messages, and only then close the connections and producers those handlers publish through. Stopping a never-started state is a no-op in mount, so the sequence is safe for every mode combination. Each state is stopped with its own mount/stop call because mount orders a multi-state call by namespace load order, not by argument order; separate calls are the only way to guarantee this dependency ordering. With stop no longer dispatched per mode, drop the now-dead :stop-fn entries from valid-modes-fns. The public stop-* fns are kept as-is for backward compatibility.
vsr625
force-pushed
the
fix/unified-shutdown-ordering
branch
from
July 20, 2026 09:02
25221a5 to
841286f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Previously each mode's stop-fn tore down its own subset of states, and modes were stopped in sorted order. :api-server's stop closed the shared consumer-connection before :worker cancelled its RabbitMQ subscribers and drained in-flight messages. Handlers mid-processing then failed to ack on the closed channel, publishing successfully processed messages to the dead set while the broker requeued them, causing double processing. Shared states (kafka-producers, producer-connection, channel-pool) had the same problem: whichever mode stopped first closed them while other modes' flows were still publishing through them.
Replace the per-mode stop dispatch with a single stop-application-states fn that stops every state in dependency order regardless of modes: stop ingestion (server, streams, batch consumers), then cancel RabbitMQ subscribers and drain in-flight messages, and only then close the connections and producers those handlers publish through. Stopping a never-started state is a no-op in mount, so the sequence is safe for every mode combination.
Each state is stopped with its own mount/stop call because mount orders a multi-state call by namespace load order, not by argument order; separate calls are the only way to guarantee this dependency ordering.
With stop no longer dispatched per mode, drop the now-dead :stop-fn entries from valid-modes-fns. The public stop-* fns are kept as-is for backward compatibility.