Periodically commit db transaction and flush on very large files

[crass]

What problem would this solve?

I have a file that is estimated to take 12+ hours to hash. The hashing was interrupted by a power outtage. It would be nice if the hashing didn't have to start from the beginning. Presumably, the hash values for blocks and extents haven't changed and now that the file hash is a hash of extent hashes, oans shouldn't need to rehash from the beginning.

Proposed solution

Periodically commit the transaction force the hash file to be written to disk (if --hashfile was given).

Alternatives considered

No response

[martinus]

This is already in place — file_scan.c commits the scan's write batch on a timer rather than at end of run:

#define COMMIT_INTERVAL_SEC	10.0
...
/* Commit the write batch once it has been open COMMIT_INTERVAL_SEC. */
	if (scan_trans_open && elapsed_seconds() - scan_write_start >= COMMIT_INTERVAL_SEC)
		return scan_write_flush();

So a scan that is interrupted loses at most the last ~10 s of completed files, not the whole run, and a --hashfile scan of a large tree is already resumable at file granularity. (The batching is deliberate: committing per file dominated the scan of a large tree — each commit forces its own WAL frames and fcntl locking through the single writer.)

What that doesn't give you is what your 12-hour file actually needs: a commit in the middle of one file, so an interrupted hash of a single huge file can resume part-way. Nothing partial exists to record today — the file's digest is only finalized at the end — so that work is really #159, and it needs a design that doesn't depend on #158 (see my comment there).

Closing this one as already implemented; #159 stays open for the mid-file case.