Allow block records to not be written to the database.

[crass]

What problem would this solve?

The database can grow quite large and the majority of records will be block records. However, if the user never does a partial dedupe, these records are never used. To get an idea, a fully mapped 1Tb file need to store 8Gb of digest data when using 4K blocks.

Note: One may wonder, "why use 4K blocks when not doing a partial dedupe?" Because extents are filesystem block aligned, not oans block aligned, having the block sizes be the same increases the chance that two extents contains the same data.

Proposed solution

Allow the user to optionally disable storing block records to the database.

Alternatives considered

No response

[martinus]

Thanks for this — I dug into it, and the answer splits three ways.

1. Block records are already not written by default.

Block hashing is off unless --dedupe-options=partial is passed (src/opt.c: do_block_hash = false), and process_blocks() returns early without it. So a user who never does a partial dedupe already stores zero block rows, at any blocksize:

$ oans -rq -b 4K   --hashfile=d.db tree/   # blocks=0
$ oans -rq -b 128K --hashfile=d.db tree/   # blocks=0

2. The rationale for wanting 4K blocks anyway doesn't hold.

Because extents are filesystem block aligned, not oans block aligned, having the block sizes be the same increases the chance that two extents contains the same data.

blocksize is used only for the block-hash array, the block-hash loop, zero-block detection and hole-skip rounding. The extent digest is computed in process_extents() over the fiemap extent's own bytes, independent of -b. Confirmed empirically — extent and file digests are byte-identical across -b 4K and -b 128K, and there's now a regression test pinning it. So there's no reason to run small blocks without partial to begin with.

3. But the problem you're describing is real, with a different cause.

Block rows are dropped by dbfile_remove_hashes() only when a file is re-hashed — and the point of a hashfile is that unchanged files aren't re-hashed. So a hashfile that ran partial once carries them forever on a stable tree:

run	blocks	hashfile
1. with --dedupe-options=partial	4096	332 KiB
2. same hashfile, without partial	4096	332 KiB
3. after touch forces a re-hash	0	68 KiB

At your scale — a fully-mapped 1 TiB at -b 4K — that's ~8 GiB of digests nothing will ever read. #167 adds --prune-block-hashes to reclaim them, and makes --stats flag them when the stored scan config no longer uses partial matching so they're at least visible.

Kept explicit rather than automatic: dropping them on any non-partial run would make a later partial run silently re-hash the whole tree.

Closing in favour of #167 — thanks for the report, the underlying observation was right even though the mechanism turned out to be a different one.