Skip to content

perf(postgres): install notification triggers lazily on first Subscribe#1935

Merged
adecaro merged 1 commit into
LFDT-Panurus:mainfrom
Built-by-Sign:perf/postgres-lazy-notify-triggers
Jul 17, 2026
Merged

perf(postgres): install notification triggers lazily on first Subscribe#1935
adecaro merged 1 commit into
LFDT-Panurus:mainfrom
Built-by-Sign:perf/postgres-lazy-notify-triggers

Conversation

@EvanYan1024

Copy link
Copy Markdown
Contributor

Motivation

The postgres stores install their notification triggers (per-row pg_notify) at store creation. PostgreSQL serializes the commit of every notifying transaction on a global notification-queue lock, so once the triggers exist, every write to those tables pays that lock — whether or not anything ever subscribes. The token and transaction notifiers currently have no subscriber in the code base (the only production subscriber is the identity-configuration sync in token/services/identity/membership/lm.go), so on write-heavy tables the triggers broadcast to nobody while serializing all commits.

In our test environments under sustained load, the database showed hundreds of connections blocked on COMMIT waiting for the notification-queue lock (wait_event_type=Lock, wait_event=object); dropping the triggers took the lock-wait peaks from ~800 to zero.

Changes

  • Notifier installs the notification schema on the first Subscribe (inside the existing startOnce) instead of at store creation. A failed installation is final: it is recorded and returned by every subsequent Subscribe; closing the notifier while the schema is installing does not start the listener.
  • The token/transaction/identity store providers no longer create the notification schema at startup. With SkipCreateTable the notifier runs no DDL at all (skipSchemaManagement) and the deployment pre-creates the notification schema like the rest of the schema.
  • The identity store provider no longer builds a second, throwaway IdentityNotifier just for schema creation; the store keeps a reference to the one it already has.

Existing subscribers are unaffected: their first Subscribe installs the same schema that store creation used to.

Tests

  • Unit: the schema is installed exactly once across concurrent subscribers; an installation failure surfaces to every subscriber and is not retried; close-during-install starts no listener; skip mode runs no DDL and still listens.
  • Against a real postgres: no trigger exists before the first subscription; subscribing installs it and notifications are delivered.
  • Full go test -race ./token/... passes.

Fixes #1934

Notification triggers fire per-row pg_notify, and PostgreSQL serializes
every notifying transaction's commit on a global queue lock. Installing
the triggers at store creation makes every write pay that cost even when
nothing subscribes. Install the notification schema on the first
subscription instead; SkipCreateTable deployments skip runtime DDL
entirely.

Signed-off-by: Evan <[email protected]>
@adecaro

adecaro commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

lovely, thanks @EvanYan1024, great insight 🙏

@adecaro
adecaro self-requested a review July 17, 2026 05:23
@adecaro adecaro self-assigned this Jul 17, 2026

@adecaro adecaro left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@adecaro
adecaro merged commit d7e192d into LFDT-Panurus:main Jul 17, 2026
91 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Postgres notification triggers serialize every commit even with no subscribers

2 participants