perf(postgres): install notification triggers lazily on first Subscribe#1935
Merged
adecaro merged 1 commit intoJul 17, 2026
Merged
Conversation
Notification triggers fire per-row pg_notify, and PostgreSQL serializes every notifying transaction's commit on a global queue lock. Installing the triggers at store creation makes every write pay that cost even when nothing subscribes. Install the notification schema on the first subscription instead; SkipCreateTable deployments skip runtime DDL entirely. Signed-off-by: Evan <[email protected]>
Contributor
|
lovely, thanks @EvanYan1024, great insight 🙏 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
The postgres stores install their notification triggers (per-row
pg_notify) at store creation. PostgreSQL serializes the commit of every notifying transaction on a global notification-queue lock, so once the triggers exist, every write to those tables pays that lock — whether or not anything ever subscribes. The token and transaction notifiers currently have no subscriber in the code base (the only production subscriber is the identity-configuration sync intoken/services/identity/membership/lm.go), so on write-heavy tables the triggers broadcast to nobody while serializing all commits.In our test environments under sustained load, the database showed hundreds of connections blocked on
COMMITwaiting for the notification-queue lock (wait_event_type=Lock, wait_event=object); dropping the triggers took the lock-wait peaks from ~800 to zero.Changes
Notifierinstalls the notification schema on the firstSubscribe(inside the existingstartOnce) instead of at store creation. A failed installation is final: it is recorded and returned by every subsequentSubscribe; closing the notifier while the schema is installing does not start the listener.SkipCreateTablethe notifier runs no DDL at all (skipSchemaManagement) and the deployment pre-creates the notification schema like the rest of the schema.IdentityNotifierjust for schema creation; the store keeps a reference to the one it already has.Existing subscribers are unaffected: their first
Subscribeinstalls the same schema that store creation used to.Tests
go test -race ./token/...passes.Fixes #1934