scan: hash sparse files whose trailing hole ends at an unaligned EOF

[martinus]

The bug

A file ending in a hole was silently and permanently skipped — on every run, forever — if its size was not a multiple of the hashing block size. It was reported as having changed while being read, which was not true:

Show.S01E04.mkv: size changed while hashing (read 2013921280 bytes, expected 2014013223)

Nothing had changed. The file was a 2 GiB sparse stub with 484 KiB actually allocated and a month-old mtime. The read stopped exactly on a 128 KiB block boundary, 91943 bytes short of EOF — one partial block.

hole_run_length() floors a hole run to a block boundary. That is right for an interior hole, where the block holding the first data byte must still be read whole, but wrong for a trailing hole, where no data follows. Flooring left a final partial block that maps no data, so fill_buffer() capped its read at zero bytes, the loop exited with off < filesize, and csum_whole_file() concluded the file had changed under it and threw the digest away. Nothing was stored, so the next run repeated it.

It hits exactly what sparseness is used for: preallocated torrent stubs, VM images, database files.

Only the e == NULL (trailing) branch changes. When EOF happens to be block-aligned the new expression is numerically identical to the old — which is why existing coverage passed: test_trailing_hole_scans truncates to 4 MiB, exactly 32 blocks.

Why here and not at the EOF check

Relaxing if (ctxt.off != ctxt.filesize) would only silence the message. The fix makes off legitimately reach filesize and folds the true {offset, length} descriptor into the running checksum. Tolerating a short read instead would leave the final partial hole unaccounted, so two sparse files identical up to the last block boundary but differing in size would produce the same digest.

Message clarity (how the bug became findable)

The old text was file <path> changed — no reason, no numbers, no consequence. It was the numbers in the reworded message that identified the block-boundary stop and led to the root cause.

Three different conditions used the word "changed" with three different meanings. They now each state what was observed and what follows from it.

A counter was also reporting the wrong cause. dedupe_dest_differs was incremented both by our own pre-flight size check and by the kernel's FILE_DEDUPE_RANGE_DIFFERS verdict, then printed as one number labelled changed since scan — so every kernel byte-compare mismatch was attributed to a file change that had not happened. A mismatch is also a property of the pair, not of the destination we happen to name. They are now counted apart, and the summary names only the causes that occurred:

  Not deduped    3 changed since scan, 2 content mismatch, 1 failed (rerun with -v for detail)

Counters are deliberately not exported to --json/run_history here — that is #145's job and doing it now would collide.

Testing

Two regression tests in test_sparse.py: an unaligned trailing hole, and a fully-sparse unaligned file. I verified both fail without the fix and pass with it, and that every pre-existing test passes either way.

Also verified against the real file that triggered this (read-only -r scan): warning gone, file hashed and recorded.

scripts/verify.sh passes — build with warnings-as-failure, 122 tests, valgrind scan+dedupe+replay smoke.

A /simplify pass is included as the last commit: the summary line is table-driven rather than three copy-pasted branches, and some comments trimmed. No behaviour change.

Note for release

Worth a release note — anyone with sparse files has been silently losing them from dedupe, with a message that pointed at the wrong cause.

🤖 Generated with Claude Code