Add physical differential oracle test for incremental backups

[NikolayS]

Physical oracle for incremental backup completeness

New TAP test src/bin/pg_combinebackup/t/013_physical_differential.pl (registered in meson.build; the Makefile picks it up via the t/*.pl glob).

Why

The WAL summarizer only learns about relation blocks that WAL records register. Any code path that modifies a page without registering its buffer (or without WAL-logging at all) is invisible to it: the incremental backup omits the block and pg_combinebackup silently reconstructs a stale page from an older backup. Bugs of this class (most recently the unregistered VM buffers fixed in ed62d26) were previously only detectable by enumerating known operations — 012_vm_consistency.pl checks INSERT/UPDATE/DELETE/LOCK/COPY explicitly. This test is the complementary generic oracle: it doesn't need to know which operation is broken; a stale block of any kind, in any compared fork, in any relation, fails the byte comparison.

Design choice: audited-quiescent-window comparison, not replay-to-target-LSN

Reviewer feedback correctly noted that separately taken backups do not share a stop LSN — each emits its own checkpoint/backup-end/WAL-switch records — and suggested restoring both paths and replaying to one target LSN. After weighing both, this PR deliberately keeps the zero-replay design, and the test header now states the actual equivalence argument instead of implying stop-LSN equality:

• What makes C and R comparable is not a shared LSN but that no relation block changes on disk between the two final backups. This is audited, not assumed: pg_waldump scans the WAL window from just before I3's start checkpoint to just after R completes; if any record in the window carries a block reference, the pair is rejected and retaken (≤3 attempts) before failing loudly.
• Why the audit is conservative-sufficient: with data checksums on (default, asserted by the test), a relation page can only reach disk in modified form if a WAL record referencing it was inserted first — normal modifications are WAL-logged in the critical section that dirties the buffer, and hint-bit-only changes emit a full-page hint on first touch after a checkpoint. Since I3's own start checkpoint lies inside the audited window, an in-window hint-bit change cannot hide behind an earlier FPI without that FPI also being in the window. Known exceptions (FSM forks: not fully WAL-logged; unlogged main forks: never backed up) are excluded from comparison.
• Why not replay-to-LSN: recovery applies full-page images, which would repair exactly the stale blocks this oracle exists to catch — any missed change whose page is touched again by replayed WAL gets overwritten wholesale, masking the bug. The zero-replay comparison cannot be fooled that way, is empirically deterministic here (attempt 1 accepted in every run), and avoids the extra cost/noise of two restored clusters. Limits are stated honestly in the test header: the argument leans on data_checksums=on (asserted, fails fast otherwise) and on the audit window's coverage, which is itself verified by a post-window txid_current() sentinel record proving pg_waldump read through the window end.

Test structure

1. Workload: heap with btree indexes fed by a sequence, TOAST-heavy table, VM set/clear cycling (VACUUM FREEZE / DELETE / UPDATE / SELECT FOR UPDATE), bulk COPY extension, TRUNCATE + re-extend, index build/drop, table drop, unlogged table (init-fork coverage).
2. Backup chain with real concurrency: full F0, then incrementals I1, I2 taken while a background_psql session runs bounded DO-block churn (backups rate-limited --max-rate=4M so they genuinely overlap: I1 ≈ 1.2 s, I2 ≈ 2.6 s vs ≈ 2.4 s churn).
3. Quiesce + final pair: settle (VACUUM, hint-bit sweep over catalogs and user tables, CHECKPOINT), then I3 and reference full R back to back, accepted only if the audited WAL window is free of block references (retake procedure above).
4. Oracle: pg_combinebackup(F0,I1,I2,I3) must be physically identical to R — block-by-block over every main, _vm, _init fork under base/ and global/ (923 files / 3344 blocks per run). File-set equality asserted both directions; anti-triviality guards require >20 files and >500 blocks compared.

Normalization: per-fork/per-AM hook, page-header mask in effect

Per review, blanket masking was restructured into a normalizer dispatch keyed by fork and access method ("fork:am" → "fork:*" → "*:*"), with relkind/AM resolved from pg_class and included in every divergence report. The dispatch currently routes everything to the page-header mask — pd_lsn + pd_checksum only — which is sufficient under this test's documented preconditions (audited-quiescent window, checksums on, heap/btree/sequence/toast relations only). The comparator comment documents, per fork/AM, what could legitimately diverge under relaxed conditions and where the corresponding masker would slot in as a local change:

• heap main fork: infomask hint bits (HEAP_XMIN/XMAX_COMMITTED …) and pd_prune_xid (cf. heap's rm_mask) — not needed here because checksums=on forces hint changes through WAL;
• btree main fork: LP_DEAD line-pointer hints, btpo_cycleid — same reasoning;
• _vm: pure bitmap payload behind the header, no legitimate unlogged content when heap VM operations are WAL-correct (which is what the oracle checks);
• _init: written once, no legitimate divergence;
• _fsm: intentionally not WAL-logged — excluded entirely rather than masked.

Empirical result: across all runs, 0 blocks even needed the pd_lsn/pd_checksum mask (counter reported via note when nonzero) — the combined output is bit-identical to the reference.

Kill test (committed, deterministic)

The corruption-injection check is now part of the test (subtests 16–21), not a prose procedure: after the main comparison, one byte is flipped — outside the masked header bytes — in copies of a heap main-fork block (block 3, offset 4000) and a VM block (block 0, offset 100) from the combined output; the comparator must report exactly one divergent block each with correct file/block/offset attribution:

ok 17 - kill test: heap main fork: exactly one divergent block detected
ok 18 - kill test: heap main fork: divergence attributed to block 3 offset 4000
ok 20 - kill test: visibility map fork: exactly one divergent block detected
ok 21 - kill test: visibility map fork: divergence attributed to block 0 offset 100

This also guards the main comparison against rotting into a trivial pass via a future masking or file-collection regression.

Divergence findings on master

None. On current master (post-ed62d26) the combined backup is bit-identical to the reference full backup in every run. No new summarizer blind spot detected by this workload.

Provenance & run evidence

• Base commit: d374280 (v19devel master tip; ed62d26 in ancestry). PR head: 52e346d (test) on 8177c62 (initial version).
• Build: meson setup build --buildtype=debug -Dcassert=true -Dtap_tests=enabled; gcc, Linux x86_64; TAP driver meson test --num-processes 1 pg_combinebackup/013_physical_differential, run as an unprivileged user.
• GUCs: summarize_wal=on, autovacuum=off, wal_keep_size=1GB (prevents segment recycling within the ~16-segment test lifetime so the audit window stays readable), allows_streaming node init, data checksums at initdb default (on — asserted as subtest 1). Combine mode --copy (env-selectable via PG_TEST_PG_COMBINEBACKUP_MODE like sibling tests).
• Runs (final code): 3/3 green at 14 s / 9 s / 10 s (21 subtests each), plus full pg_combinebackup suite green (13 tests, 013 at 10.08 s). Earlier iterations of the same design: 8/8 green at 9–12 s. Every run accepted the final backup pair on attempt 1; the retake path has never been exercised outside fault injection.
• Logs: build/testrun/pg_combinebackup/013_physical_differential/log/regress_log_013_physical_differential (TAP output incl. LSN window notes and any DIVERGENCE reports) and .../log/013_physical_differential_primary.log (server log); build/meson-logs/testlog.txt (suite summary).
• Retake/minimization procedure: on a failed window audit the test diags the offending blkref records from pg_waldump (first 10), retakes settle+I3+R up to 3 attempts, and aborts loudly if none is clean — so a genuine quiescence breaker is characterized, not averaged away. On comparison failure, reports carry relpath, relname, relkind, AM, fork, absolute block number (segment-aware) and differing byte ranges with hex from both sides (capped at 50 reports / 5 ranges each) — enough to minimize straight to the responsible relation and operation.

Caveats

• Oracle sensitivity is bounded by the workload: unregistered-buffer bugs in operations never performed (BRIN/GIN/GiST/SP-GiST/hash indexes, logical structures) won't be caught; the workload is deliberately extensible, and the normalizer dispatch is structured so AM additions are local.
• With --no-data-checksums the strict hint-bit comparison would be unsound; the test fails fast on the checksums assertion rather than producing false positives.
• pg_waldump quirk worked around: an --end LSN sitting exactly at a fresh segment's first-record position triggers a spurious prev-link error from a clamped page read (stale read-buffer bytes; reproducible on static files) — the test therefore runs without --end and verifies coverage via the sentinel record. Possibly worth an independent upstream report.

Closes #74

🤖 Generated with Claude Code

https://claude.ai/code/session_01PtxA6B4d47ietk5dGsseuP