Skip read-only subvolumes (snapshots) when deduplicating

[martinus]

What problem would this solve?

Pointed at a btrfs tree that contains snapshots — the normal shape of a NAS or a snapper/Timeshift desktop — oans walks into every read-only snapshot and reads and hashes every file in it, for no possible gain.

Measured on btrfs with one live subvolume (2 files) and one read-only snapshot of it:

$ oans -rq --hashfile=/tmp/s.db $T
$ sqlite3 /tmp/s.db "select count(*) from files"
4
$ sqlite3 /tmp/s.db "select filename from files order by filename"
/…/snap/.snapshots/snap1/a.bin
/…/snap/.snapshots/snap1/b.bin
/…/snap/live/a.bin
/…/snap/live/b.bin

The behaviour is safe — that part is working correctly. At dedupe time the already-shared check catches them before any ioctl:

[0x…] /…/.snapshots/snap1/a.bin already shares the target's extents; skipping.
[0x…] /…/.snapshots/snap1/b.bin already shares the target's extents; skipping.

Nothing is written into the read-only subvolume, no EROFS, exit 0.

The cost is the problem. Every file in every snapshot is opened, read, hashed and written to the hashfile, and only then found to be pointless. The waste scales with snapshot count: 20 snapshots of a 1 TiB tree means reading ~20 TiB and storing ~20× the hashfile rows to reclaim nothing from 19 of them. On the rotational storage this fork targets, that is the difference between a scheduled job finishing overnight and not finishing.

A read-only subvolume can never be a dedupe destination — the kernel refuses — so under -d the work is provably wasted, not merely usually wasted.

oans currently has no read-only-subvolume awareness at all. It tracks subvolume ids only to key UNIQUE(ino, subvol) for the hardlink guard.

Proposed solution

Detect read-only btrfs subvolumes during the walk and skip them when deduplicating.

Why by property rather than by name. The obvious alternative is a built-in exclude list (.snapshots, @eaDir, .Trash-1000). Read-only-ness is a fact about what the kernel will accept; a directory name is a guess. Detecting the property catches snapper, Timeshift and Synology alike, whatever the directory is called, and never fires on a directory that merely happens to be named .snapshots. See Alternatives for the rest of the argument against a name list.

Where it fits. btrfs gives every subvolume a distinct anonymous st_dev — confirmed, dev=107 for the live subvolume vs dev=108 for its snapshot — and file_scan.c already caches per-subvolume state keyed by st_dev (subvol_cache, around line 515, added precisely so per-subvolume ioctls happen once rather than per file). Read-only-ness can ride along in the same cache: one BTRFS_IOC_SUBVOL_GETFLAGS per subvolume, checked for BTRFS_SUBVOL_RDONLY. No per-file cost, and the walk already has the st_dev in hand from its statx.

Two conditions, both important:

1. Only when deduplicating. In report mode (no -d) a user may legitimately want to know what duplicates exist inside snapshots — that is a reasonable thing to ask of a reporting tool. The skip should key off -d, with a flag to force either way (a --dedupe-options token, or a dedicated --[no-]skip-readonly-subvols).

2. It must be reported, not silent. "Skipped 12 read-only subvolumes" in the summary, and a field in --json. A default that quietly shrinks the scan is the same class of surprise as the silent --exclude no-op that --exclude patterns that match nothing are accepted silently #147 fixed, just wearing different clothes. This is the natural first consumer of Scan-phase skips and errors are never counted, reported, or exported #145's skip accounting — if Scan-phase skips and errors are never counted, reported, or exported #145 lands first, this should use its buckets rather than inventing a parallel counter.

Worth deciding in the issue: whether a read-only subvolume should still be hashed so it can serve as a dedupe source (only the destination needs to be writable, so live files could in principle be deduped against a snapshot's extents). My inclination is no: if the content matched, the extents are usually already shared anyway, so the gain is marginal and it would give up the entire saving this issue is about.

Alternatives considered

• A built-in name-based exclude list (.snapshots, @eaDir, …). Rejected on three counts. It is silent, which is the failure mode Scan-phase skips and errors are never counted, reported, or exported #145/--exclude patterns that match nothing are accepted silently #147 exist to remove — a user points oans at /srv and it quietly scans less than they asked. It is wrong sometimes: a real directory named .snapshots holding user data gets skipped with no indication. And it breaks the self-describing hashfile: implicit defaults are not written to scan_excludes, so a replay's scope would depend on which binary version ran it — upgrade oans and a scheduled job silently changes what it covers, which is exactly what the stored-config design prevents.
• Documentation only. Since 1.6.0 made .gitignore patterns work, --exclude '.snapshots' --exclude '@eaDir' finally does the obvious thing, and the NAS guide should recommend it regardless — it is explicit and gets stored in the hashfile. But it requires every user to know to do it, does not help anyone who has already set up a job, and does nothing for snapshot directories in a non-standard location.
• Skipping any subvolume boundary. Too broad: plenty of people keep ordinary writable data in separate subvolumes and legitimately want it deduplicated.
• Relying on the already-shared check, as today. It is what makes the current behaviour safe, and it should stay, but it only avoids the ioctl. All the read and hash I/O has already been spent by the time it runs.