What problem would this solve?
Do block hashing on an 8Tb file, where blocksize is 4K, requires allocating 48G of memory, which is a really inefficient use of memory because once a block is written to its never used, only saved to the database. So the 48Gb of memory in the above example is only written to and read once, and in sequential order.
Proposed solution
Introduce a block batch size, and only allocate up to batch sized blocks for hashing. When the block batch has all been written to, serialize them to the database. Consider having the block batch size be user definable but default to some multiple of read_buf_size / blocksize.
Alternatives considered
A non-mutually exclusive alternative would be to allow the user to completely disable block hashing and never allocate the memory for them. This would work because neither the file nor extent hashes depend on the block hash. And if the user never want to do a partial dedup, then the block records are never needed (wasted compute, memory, and storage). However, this does not solve the problem when doing a partial dedup, so its not a complete alternative.
What problem would this solve?
Do block hashing on an 8Tb file, where blocksize is 4K, requires allocating 48G of memory, which is a really inefficient use of memory because once a block is written to its never used, only saved to the database. So the 48Gb of memory in the above example is only written to and read once, and in sequential order.
Proposed solution
Introduce a block batch size, and only allocate up to batch sized blocks for hashing. When the block batch has all been written to, serialize them to the database. Consider having the block batch size be user definable but default to some multiple of
read_buf_size / blocksize.Alternatives considered
A non-mutually exclusive alternative would be to allow the user to completely disable block hashing and never allocate the memory for them. This would work because neither the file nor extent hashes depend on the block hash. And if the user never want to do a
partialdedup, then the block records are never needed (wasted compute, memory, and storage). However, this does not solve the problem when doing apartialdedup, so its not a complete alternative.