lore-storage: Batch FastCDC chunking to cut per-chunk overhead#61
lore-storage: Batch FastCDC chunking to cut per-chunk overhead#61Vazcore wants to merge 6 commits into
Conversation
write_fragmented called FastCDC::cut once per chunk, shipping the work to the compute pool, awaiting the result, and repeating. For a 1 GiB buffer with ~4 KiB chunks that is ~256k spawn dispatches, oneshot allocations and await yields — overhead that can rival the chunking work itself for small remaining chunks. Compute all FastCDC boundaries in a single compute_pool task by driving the FastCDC Iterator to completion. Fixed-size chunking has trivial per-step math so its boundaries are computed inline. The single-fragment fast path and the storage dispatch loop are unchanged. Verified by a new parity test that compares the batched boundaries against a reference FastCDC iteration on a 256 KiB random buffer, plus edge cases for empty, sub-min-size, all-zero, and non-aligned fixed-size inputs. Signed-off-by: Oleksii Habrusiev <[email protected]>
2fe3c48 to
80d041b
Compare
…nt-optimization Signed-off-by: Oleksii Habrusiev <[email protected]> # Conflicts: # lore-storage/src/fragment_engine.rs
e560d28 to
f2071c6
Compare
mjansson
left a comment
There was a problem hiding this comment.
The average chunk size is 64KiB, so 1GiB would be 16k tasks - but yes, I agree with the idea.
Some comments, please address them.
Also, please do some micro benchmarks to see if this has any actual measurable impact.
…ation Signed-off-by: Oleksii Habrusiev <[email protected]>
2b89c6a to
7b60d66
Compare
@mjansson thanks for review. I created a draft PR to run some benchmarks using "criterion". I updated this PR description to include a raw log and also some summary of improvement.
At 64 KB the overhead of a single boundary is too small to notice. But for real-world writes (1 MB and up) the batch approach is consistently 40-47% faster. |
Signed-off-by: Oleksii Habrusiev <[email protected]>
…c allocation in FastCDC chunking Signed-off-by: Oleksii Habrusiev <[email protected]>
|
Imported as Lore CR-212. |
|
Imported as Lore CR-218. |
When you store a big file in Lore, write_fragmented cuts it into chunks using FastCDC and ships each cut point to the compute pool separately, waits for the answer, then does it again. For a 1 GiB buffer with 64 KiB average chunks I think that's around 16k spawn dispatches, oneshot allocations and await yields. The actual chunking work is fast, but all that round-tripping added up to a real cost on medium and large writes I believe. ### Why A single chunking pass on the compute pool is enough - there's no reason to wait per boundary. ### How A single compute_pool task drives the FastCDC Iterator to completion and sends back a list of boundaries through one oneshot. The existing storage dispatch loop then iterates them locally. For fixed-size chunking the boundaries are just step arithmetic so those are computed inline. The Arc<FastCDC> wrapper is gone since we only touch the chunker once now. The single-fragment fast path, the JoinSet of storage tasks, hash-only mode and clone_buffer are all preserved, and the public signature of write_fragmented is unchanged. Behaviorally the output is identical to the old code, same cut points for the same input. ### Testing I added a small reference helper that does the standard FastCDC iteration synchronously and compared its output to the batched path on a 256 KiB random buffer. There's also coverage for empty input, sub-min-size input, an all-zero buffer (which defeats the rolling hash), and a non-aligned fixed-size case. cargo test -p lore-storage goes from 154 to 160 passing, clippy is clean with -D warnings, and the whole workspace still compiles. The fastcdc_batch_matches_reference test is the one that would catch a regression in the FastCDC version or a misconfiguration of the chunk sizes. ### Testing using Criterion Draft PR with criterion: #103 #### Micro-benchmark results Comparing the new batched approach against a simulation of the old per-chunk approach. Same FastCDC config as the real code: min=32KB, expected=64KB, threshold=256KB, Level1. #### What's being measured - **batched** — all chunk boundaries found in one compute-pool dispatch (PR approach) - **per-chunk** — one compute-pool spawn + oneshot channel per boundary (old approach) #### Results | Buffer size | Batched | Per-chunk | Speedup | |------------|---------|-----------|---------| | 64 KB | 19.0 µs | 19.8 µs | **1.04x** | | 1 MB | 222 µs | 327 µs | **1.47x** | | 16 MB | 3.79 ms | 5.53 ms | **1.46x** | | 64 MB | 15.4 ms | 21.5 ms | **1.40x** | #### Why it matters At 64 KB the overhead of a single boundary is too small to notice. But for real-world writes (1 MB and up) the batch approach is consistently **40-47% faster**. A 1 GB file with ~16K chunk boundaries saves ~16K spawn dispatches, oneshot allocations and await yields — replacing them with a single dispatch. Run locally: `cargo bench -p lore-storage --bench chunking` #### Raw log: ``` warning: [email protected]: Failed to execute Lore to get revision information, no data extracted Finished `bench` profile [optimized + debuginfo] target(s) in 0.35s Running benches\chunking.rs (target\release\deps\chunking-b21acb5ed9e17d59.exe) Benchmarking fastcdc_chunking/batched_64KiB Benchmarking fastcdc_chunking/batched_64KiB: Warming up for 3.0000 s Benchmarking fastcdc_chunking/batched_64KiB: Collecting 100 samples in estimated 5.0093 s (263k iterations) Benchmarking fastcdc_chunking/batched_64KiB: Analyzing fastcdc_chunking/batched_64KiB time: [18.946 ┬╡s 19.007 ┬╡s 19.072 ┬╡s] change: [ΓêÆ5.5576% ΓêÆ4.0130% ΓêÆ2.6139%] (p = 0.00 < 0.05) Performance has improved. Found 2 outliers among 100 measurements (2.00%) 1 (1.00%) high mild 1 (1.00%) high severe Benchmarking fastcdc_chunking/per_chunk_64KiB Benchmarking fastcdc_chunking/per_chunk_64KiB: Warming up for 3.0000 s Benchmarking fastcdc_chunking/per_chunk_64KiB: Collecting 100 samples in estimated 5.0627 s (258k iterations) Benchmarking fastcdc_chunking/per_chunk_64KiB: Analyzing fastcdc_chunking/per_chunk_64KiB time: [19.718 ┬╡s 19.759 ┬╡s 19.802 ┬╡s] change: [ΓêÆ1.6628% ΓêÆ0.6054% +0.3810%] (p = 0.26 > 0.05) No change in performance detected. Found 9 outliers among 100 measurements (9.00%) 1 (1.00%) low severe 3 (3.00%) low mild 4 (4.00%) high mild 1 (1.00%) high severe Benchmarking fastcdc_chunking/batched_1MiB Benchmarking fastcdc_chunking/batched_1MiB: Warming up for 3.0000 s Benchmarking fastcdc_chunking/batched_1MiB: Collecting 100 samples in estimated 5.6211 s (25k iterations) Benchmarking fastcdc_chunking/batched_1MiB: Analyzing fastcdc_chunking/batched_1MiB time: [221.66 ┬╡s 222.25 ┬╡s 222.95 ┬╡s] change: [ΓêÆ17.166% ΓêÆ16.724% ΓêÆ16.093%] (p = 0.00 < 0.05) Performance has improved. Found 6 outliers among 100 measurements (6.00%) 5 (5.00%) high mild 1 (1.00%) high severe Benchmarking fastcdc_chunking/per_chunk_1MiB Benchmarking fastcdc_chunking/per_chunk_1MiB: Warming up for 3.0000 s Benchmarking fastcdc_chunking/per_chunk_1MiB: Collecting 100 samples in estimated 6.6377 s (20k iterations) Benchmarking fastcdc_chunking/per_chunk_1MiB: Analyzing fastcdc_chunking/per_chunk_1MiB time: [324.99 ┬╡s 326.69 ┬╡s 329.20 ┬╡s] change: [ΓêÆ3.5149% ΓêÆ2.9853% ΓêÆ2.2318%] (p = 0.00 < 0.05) Performance has improved. Found 7 outliers among 100 measurements (7.00%) 2 (2.00%) high mild 5 (5.00%) high severe Benchmarking fastcdc_chunking/batched_16MiB Benchmarking fastcdc_chunking/batched_16MiB: Warming up for 3.0000 s Benchmarking fastcdc_chunking/batched_16MiB: Collecting 100 samples in estimated 5.3552 s (1400 iterations) Benchmarking fastcdc_chunking/batched_16MiB: Analyzing fastcdc_chunking/batched_16MiB time: [3.7744 ms 3.7851 ms 3.7964 ms] change: [+3.0532% +3.4713% +3.8820%] (p = 0.00 < 0.05) Performance has regressed. Found 2 outliers among 100 measurements (2.00%) 2 (2.00%) high mild Benchmarking fastcdc_chunking/per_chunk_16MiB Benchmarking fastcdc_chunking/per_chunk_16MiB: Warming up for 3.0000 s Benchmarking fastcdc_chunking/per_chunk_16MiB: Collecting 100 samples in estimated 5.4254 s (1000 iterations) Benchmarking fastcdc_chunking/per_chunk_16MiB: Analyzing fastcdc_chunking/per_chunk_16MiB time: [5.4582 ms 5.5349 ms 5.6246 ms] change: [+5.0870% +6.5725% +8.0610%] (p = 0.00 < 0.05) Performance has regressed. Found 15 outliers among 100 measurements (15.00%) 7 (7.00%) high mild 8 (8.00%) high severe Benchmarking fastcdc_chunking/batched_64MiB Benchmarking fastcdc_chunking/batched_64MiB: Warming up for 3.0000 s Benchmarking fastcdc_chunking/batched_64MiB: Collecting 100 samples in estimated 6.2283 s (400 iterations) Benchmarking fastcdc_chunking/batched_64MiB: Analyzing fastcdc_chunking/batched_64MiB time: [15.341 ms 15.424 ms 15.513 ms] change: [+1.2826% +1.9061% +2.6672%] (p = 0.00 < 0.05) Performance has regressed. Found 2 outliers among 100 measurements (2.00%) 1 (1.00%) high mild 1 (1.00%) high severe Benchmarking fastcdc_chunking/per_chunk_64MiB Benchmarking fastcdc_chunking/per_chunk_64MiB: Warming up for 3.0000 s Benchmarking fastcdc_chunking/per_chunk_64MiB: Collecting 100 samples in estimated 6.5061 s (300 iterations) Benchmarking fastcdc_chunking/per_chunk_64MiB: Analyzing fastcdc_chunking/per_chunk_64MiB time: [21.428 ms 21.534 ms 21.675 ms] change: [+2.4232% +2.9622% +3.6167%] (p = 0.00 < 0.05) Performance has regressed. Found 6 outliers among 100 measurements (6.00%) 3 (3.00%) high mild 3 (3.00%) high severe Benchmarking fixed_size/16MiB Benchmarking fixed_size/16MiB: Warming up for 3.0000 s Benchmarking fixed_size/16MiB: Collecting 100 samples in estimated 5.0009 s (20M iterations) Benchmarking fixed_size/16MiB: Analyzing fixed_size/16MiB time: [250.77 ns 251.66 ns 252.62 ns] change: [+0.3309% +0.9096% +1.4554%] (p = 0.00 < 0.05) Change within noise threshold. Found 4 outliers among 100 measurements (4.00%) 4 (4.00%) high mild ``` ``` Imported-PR: #61 Imported-From: 8d7aab9 Imported-Base: a00b1c4 Imported-Merge: 30a8d7b Imported-Author: Oleksii Habrusiev (Vazcore) Signed-off-by: Oleksii Habrusiev <[email protected]> GH-URL: #61 ``` Lore-RevId: 323 Lore-Signature: b9e7f3161fda891797a950e7e04b1de54e06b664fabd8e179666b5927175cc3f
|
Closed by mirrored commit 4bbdf03. |
When you store a big file in Lore, write_fragmented cuts it into chunks using FastCDC and ships each cut point to the compute pool separately, waits for the answer, then does it again. For a 1 GiB buffer with 64 KiB average chunks I think that's around 16k spawn dispatches, oneshot allocations and await yields. The actual chunking work is fast, but all that round-tripping added up to a real cost on medium and large writes I believe.
Why
A single chunking pass on the compute pool is enough - there's no reason to wait per boundary.
How
A single compute_pool task drives the FastCDC Iterator to completion and sends back a list of boundaries through one oneshot. The existing storage dispatch loop then iterates them locally. For fixed-size chunking the boundaries are just step arithmetic so those are computed inline.
The Arc wrapper is gone since we only touch the chunker once now. The single-fragment fast path, the JoinSet of storage tasks, hash-only mode and clone_buffer are all preserved, and the public signature of write_fragmented is unchanged. Behaviorally the output is identical to the old code, same cut points for the same input.
Testing
I added a small reference helper that does the standard FastCDC iteration synchronously and compared its output to the batched path on a 256 KiB random buffer. There's also coverage for empty input, sub-min-size input, an all-zero buffer (which defeats the rolling hash), and a non-aligned fixed-size case. cargo test -p lore-storage goes from 154 to 160 passing, clippy is clean with -D warnings, and the whole workspace still compiles. The fastcdc_batch_matches_reference test is the one that would catch a regression in the FastCDC version or a misconfiguration of the chunk sizes.
Testing using Criterion
Draft PR with criterion: #103
Micro-benchmark results
Comparing the new batched approach against a simulation of the old per-chunk approach.
Same FastCDC config as the real code: min=32KB, expected=64KB, threshold=256KB, Level1.
What's being measured
Results
Why it matters
At 64 KB the overhead of a single boundary is too small to notice. But for real-world
writes (1 MB and up) the batch approach is consistently 40-47% faster. A 1 GB file
with ~16K chunk boundaries saves ~16K spawn dispatches, oneshot allocations and await
yields — replacing them with a single dispatch.
Run locally:
cargo bench -p lore-storage --bench chunkingRaw log: