Hash resume for very large files

[crass]

What problem would this solve?

Currently oans must start hashing a file from the beginning everytime a hash is needed. For example, if a 1Tb file is 6hours in to hashing and the oans process is interrupted (power outtage, OOM, ctrl+c, etc.), rerunning oans will have to start rehashing the large file from the beginning.

Proposed solution

Rehashing from the beginning must be done because the current digest is a hash of individual bytes and xxhash is not additive. Instead have the file hash be a hash of the extent hashes and the extent hash be a hash of the block hashes as suggested in #158.

Also, this will require #157, otherwise there will never be block or extent records for files in the middle of being hashed when an interruption occurs.

Alternatives considered

No response

[martinus]

Parking note, since this issue's two dependencies have now resolved differently than planned:

• Periodically commit db transaction and flush on very large files #157 is closed as already implemented — the scan writer commits every 10 s (COMMIT_INTERVAL_SEC), so an interrupted run already resumes at file granularity. What's missing is a commit mid-file, which is this issue.
• Change file hash and extent hash to be a hash of hashes #158 is closed — making the file digest a hash of extent digests would make fragmentation part of file identity, so byte-identical files with different extent layouts would stop matching (evidence in that thread). So the foundation this issue proposed to build on isn't available.

The problem is still real: a 1 TiB file interrupted six hours into hashing restarts from zero, and that's a shape the NAS/VM-image audience genuinely has.

A resume that doesn't redefine the hash would mean persisting the running xxhash state at a checkpoint offset and reloading it. That's feasible in principle but XXH3_state_t has no stable serialization API, so it would mean versioning an internal library struct inside the hashfile and invalidating checkpoints on an xxhash upgrade. Doable, not cheap, and it needs care around the change-detection path (a file whose row shows the new mtime/size but no digest must not be treated as up to date).

Leaving open as a known limitation rather than closing — but it needs a fresh design, not the one sketched above.