From 1a47cee03394ef70f5f70a1a93648be2369611ee Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 25 Jul 2026 06:37:21 +0000 Subject: [PATCH 1/3] notify-bench: harness isolating the NOTIFY commit lock (fork-local) Adds the benchmark harness, raw results and measurement instrumentation used to characterize the cluster-wide lock that PreCommit_Notify() takes. Fork-local; not proposed for upstream in this form. The central difficulty is that comparing INSERT against INSERT+NOTIFY measures the lock plus the SLRU queue insert, the utility statement and an extra round trip. The instrumentation patch resolves this by making the identical binary skip only LockSharedObject() when PG_NOTIFY_NOLOCK is set in the postmaster environment, so the two arms differ by that one call. With synchronous_commit=on and zero listeners, removing it takes throughput from 1,027 to 7,893 TPS at 16 clients and from 1,055 to 12,369 at 64 -- recovering essentially all of the gap to a NOTIFY-free workload up to 16 clients and 74% of it at 64. This shows the SLRU NotifyQueueLock does not simply become the next bottleneck. Stated without reference to this disk: the wait-event data shows both arms with one fsync in flight, but the control queues ~21 backends behind it while NOTIFY queues none, because they are stuck a step earlier. Group-commit batch factor ~21 versus exactly 1. The README documents the harness's known defects rather than hiding them, including one that invalidates the "interested listener" results and whose bias favours the conclusion the harness was built to test: psql reading from a fifo never calls PQconsumeInput, so those listeners jam in ClientWrite instead of draining. scripts/checkjam.sh demonstrates it. Those numbers are retracted; the uninterested-listener results are unaffected. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01M3RLizCyeXBBFJdBHGDvKW --- notify-bench/README.md | 139 ++++++++++++++++++ ...UREMENT-ONLY-skip-notify-commit-lock.patch | 33 +++++ notify-bench/results/ab_lock.csv | 39 +++++ notify-bench/results/master.csv | 31 ++++ notify-bench/results/master.listeners.csv | 8 + notify-bench/results/master.listeners2.csv | 9 ++ notify-bench/results/master.waits.csv | 107 ++++++++++++++ notify-bench/results/pg18.listeners2.csv | 9 ++ notify-bench/results/pg18.waits.csv | 101 +++++++++++++ notify-bench/results/pg18.writers.csv | 29 ++++ notify-bench/scripts/ab_lock.sh | 117 +++++++++++++++ notify-bench/scripts/checkjam.sh | 49 ++++++ notify-bench/scripts/listeners.sh | 73 +++++++++ notify-bench/scripts/verify.sh | 53 +++++++ notify-bench/scripts/writers.sh | 78 ++++++++++ 15 files changed, 875 insertions(+) create mode 100644 notify-bench/README.md create mode 100644 notify-bench/instrumentation/0001-MEASUREMENT-ONLY-skip-notify-commit-lock.patch create mode 100644 notify-bench/results/ab_lock.csv create mode 100644 notify-bench/results/master.csv create mode 100644 notify-bench/results/master.listeners.csv create mode 100644 notify-bench/results/master.listeners2.csv create mode 100644 notify-bench/results/master.waits.csv create mode 100644 notify-bench/results/pg18.listeners2.csv create mode 100644 notify-bench/results/pg18.waits.csv create mode 100644 notify-bench/results/pg18.writers.csv create mode 100644 notify-bench/scripts/ab_lock.sh create mode 100644 notify-bench/scripts/checkjam.sh create mode 100644 notify-bench/scripts/listeners.sh create mode 100644 notify-bench/scripts/verify.sh create mode 100644 notify-bench/scripts/writers.sh diff --git a/notify-bench/README.md b/notify-bench/README.md new file mode 100644 index 0000000000000..298698d6023fe --- /dev/null +++ b/notify-bench/README.md @@ -0,0 +1,139 @@ +# notify-bench — measuring the LISTEN/NOTIFY commit lock + +Fork-local benchmark harness for the cluster-wide heavyweight lock that +`PreCommit_Notify()` takes at `src/backend/commands/async.c:1309`: + +```c +LockSharedObject(DatabaseRelationId, InvalidOid, 0, AccessExclusiveLock); +``` + +Because `LockSharedObject()` builds the locktag with dbid `InvalidOid`, this lock is +cluster-wide (all databases). It is released at +`ResourceOwnerRelease(RESOURCE_RELEASE_LOCKS)` (`xact.c:2480`), i.e. **after** +`RecordTransactionCommit()` (`:2407`, containing `XLogFlush()` at `:1544` and +`SyncRepWaitForLSN()` at `:1599`) and **after** `ProcArrayEndTransaction()` (`:2431`). +Notifying transactions therefore cannot group-commit with one another. + +Context, full analysis and discussion: issue #82 in this fork. + +> **This directory is fork-local.** It is not proposed for upstream inclusion in this +> form. The instrumentation patch is a measurement device and is deliberately unsafe. + +## Headline result: the lock, isolated + +The problem with comparing `INSERT` against `INSERT; NOTIFY` is that the delta contains +the lock *plus* the SLRU queue insert, the utility statement, and an extra client round +trip. To isolate the lock, `instrumentation/0001-MEASUREMENT-ONLY-skip-notify-commit-lock.patch` +makes the identical binary skip **only** that one call when `PG_NOTIFY_NOLOCK` is set in +the postmaster environment. + +`synchronous_commit=on`, zero listeners, 3 reps of 10 s, median [min–max]: + +| clients | control | NOTIFY (lock) | NOTIFY (no lock) | gap recovered | +|--:|--:|--:|--:|--:| +| 1 | 824 [794–835] | 790 [618–844] | 733 [695–778] | — (no gap at c=1) | +| 4 | 2,059 [1946–2075] | 1,033 [934–1051] | 2,177 [2115–2300] | 111% | +| 16 | 7,811 [7430–8006] | 1,027 [911–1074] | 7,893 [7606–8290] | 101% | +| 64 | 16,307 [16044–16469] | 1,055 [979–1135] | 12,369 [11077–12973] | 74% | + +Lock-removal speedup: 2.1x (4 clients), 7.7x (16), 11.7x (64). Arm separation is +asserted by the harness: 72 `Lock`/`object` wait samples in the lock arm, **0** in the +no-lock arm. A run where that check fails to separate is void. + +This answers the open question from the 2025 pgsql-hackers thread — whether +`NotifyQueueLock` would simply become the next bottleneck. It does not: it is a short +LWLock around the SLRU insert and never spans the fsync. At 64 clients the residual +(NotifyQueueLock + SLRU + one extra round trip) is ~26% of the gap. + +### The mechanism, stated hardware-independently + +From `results/master.waits.csv`, both arms show exactly one backend in `IO`/`WalSync` at +any instant — one fsync in flight. The difference is who is queued behind it: + +| | `Lock`/`object` | `LWLock`/`WALWrite` | `IO`/`WalSync` | +|---|--:|--:|--:| +| NOTIFY, 32 clients | 610 | **0** | 18 | +| control, 32 clients | 0 | **373** | 18 | + +Control queues ~21 backends per in-flight fsync — group commit working. NOTIFY queues +**zero**, because everyone is stuck one step earlier on the heavyweight lock. +**Group-commit batch factor ~21 versus exactly 1.** Throughput ratios on any particular +disk are a consequence of that; this statement is the portable one. + +## Scripts + +All run as an unprivileged user; PostgreSQL refuses to start as root. + +``` +scripts/writers.sh