Strengthen restored-cluster oracle in pg_combinebackup tests (amcheck + pg_visibility + IOS/seqscan comparison)

[NikolayS]

What

Upgrades the verification oracle in the incremental-backup TAP tests and adds a committed negative control proving the oracle fires:

• src/bin/pg_combinebackup/t/CombineBackupTest.pm — new shared helper module (loaded via the FindBin pattern used by pg_rewind's RewindTest.pm) exporting check_physical_consistency, vm_sweep_errors, ios_vs_seqscan, connectable_databases.
• src/bin/pg_combinebackup/t/002_compare_backups.pl — after the existing pg_dumpall comparison, runs the full oracle on both the full-backup PITR and the incremental (combined) PITR, so a divergence is attributable to pg_combinebackup. Gains an indexed_table (1000 rows, PK, VACUUM (FREEZE)) whose heap, index, and VM change inside the incremental window; autovacuum disabled on restored nodes.
• src/bin/pg_combinebackup/t/012_vm_consistency.pl — per-operation restored-cluster validation now also runs pg_check_frozen; then the full shared oracle runs against the restored cluster using a new vm_ios_test indexed table modified inside the incremental window.
• src/bin/pg_combinebackup/t/013_stale_vm_detection.pl — new committed negative control (kill test), see below.

The oracle, per restored cluster:

1. pg_amcheck --all --install-missing --heapallindexed — heap and index consistency in every database.
2. pg_visibility sweep in every connectable database (pg_database WHERE datallowconn): pg_check_visible() + pg_check_frozen() over every user relation (tables, matviews, toast; pg_class relkind r/m/t, oid >= 16384), asserting zero corrupt TIDs.
3. Forced index-only scan vs forced seqscan: same count/sum/min/max aggregate under enable_seqscan=off, enable_indexonlyscan=on vs the inverse, asserting identical results, plus an EXPLAIN assertion that the forced plan really is an Index Only Scan — the user-visible symptom of stale all-visible bits.

Why

002_compare_backups.pl compared restored clusters via pg_dumpall output only (its own comment admits the limitation). A logical dump reads live rows through sequential scans and is blind to stale visibility-map bits, index corruption, and other page-level damage — exactly why the ed62d26 VM/WAL-logging corruption class survived this test. 012_vm_consistency.pl (5174d15) checks specific operations but its restored-cluster verification only ran pg_check_visible on the per-operation tables.

Negative control (013_stale_vm_detection.pl)

Committed, deterministic kill test proving the oracle demonstrably fires — only a scratch restored cluster is ever corrupted:

1. Primary initialized with --no-data-checksums (so a fabricated stale VM bit is a valid page whose bits are simply wrong — the same shape as a real pg_combinebackup bug), summarize_wal=on, autovacuum=off. Table t + identical t_control: 1000 rows each, PK, VACUUM (FREEZE).
2. Full backup → DELETE ... WHERE id % 10 = 0 on both tables (clears VM bits, leaves dead tuples still present in the indexes; the predicate is non-indexable so no index kill bits are set) → incremental backup → pg_combinebackup restore into a scratch cluster.
3. Sanity: oracle passes on the intact restored cluster (sweep clean; IOS == seqscan on t_control).
4. Stop cluster, write 0xFF over the first 4 bytes of t's VM fork data area (after the 24-byte page header ⇒ first 16 heap pages claimed all-visible+all-frozen), restart. Assertions that must fire:
   • pg_visibility sweep reports corruption: t|100|100 (100 corrupt TIDs from each of pg_check_visible/pg_check_frozen);
   • forced IOS diverges from seqscan: 1000|500500|1|1000 vs 900|450000|1|999.
5. Stop, overwrite 100 bytes of a t_pkey btree page, restart: pg_amcheck must fail (it does: "line pointer points past end of tuple space").

Subtlety encoded in the test: the corruption target must never be index-scanned before its VM is corrupted — an IOS on the intact cluster heap-fetches the dead tuples and sets LP_DEAD kill bits in the index, which would hide the divergence afterwards. Hence the separate t_control for the intact-cluster sanity IOS. Also confirmed: pg_amcheck alone does not detect stale VM bits (verify_heapam has no VM cross-check), so the pg_visibility + IOS/seqscan layers are necessary, not redundant.

Test evidence

1/3 postgresql:pg_combinebackup / pg_combinebackup/012_vm_consistency     OK   8.59s  49 subtests passed
2/3 postgresql:pg_combinebackup / pg_combinebackup/002_compare_backups    OK  13.30s  18 subtests passed
3/3 postgresql:pg_combinebackup / pg_combinebackup/013_stale_vm_detection OK   6.93s   9 subtests passed

Key TAP lines from 013:

ok 7 - pg_visibility sweep detects the stale VM bits
# vm sweep reported: t|100|100
ok 8 - index-only scan diverges from seqscan with stale VM bits
# index-only scan returned [1000|500500|1|1000], seqscan [900|450000|1|999]
ok 9 - pg_amcheck detects the corrupted btree page

Provenance / reproduction

• Base commit: d374280 ("ci: Generate crashlogs on Windows"), v19devel, branch claude/testplan-backup-oracle.
• Build: meson setup build --buildtype=debug -Dcassert=true -Dtap_tests=enabled; gcc 13, Linux x86_64; combine mode --copy (default PG_TEST_PG_COMBINEBACKUP_MODE).
• Non-default GUCs in the tests: summarize_wal=on (all primaries), autovacuum=off (012/013 primaries and all restored/scratch nodes), recovery settings in 002 as before (recovery_target_lsn, recovery_target_action=promote, archive_mode=off). 013 additionally uses initdb --no-data-checksums.
• Reproduce: meson test -C build --suite setup && meson test -C build pg_combinebackup/002_compare_backups pg_combinebackup/012_vm_consistency pg_combinebackup/013_stale_vm_detection (as a non-root user). Runtimes above; total added wall time across the suite is roughly 15 s.

Closes #63

🤖 Generated with Claude Code

https://claude.ai/code/session_01PtxA6B4d47ietk5dGsseuP