Add incremental tile and entry-bundle writers#53
Conversation
ebb6e57 to
5479ef7
Compare
There was a problem hiding this comment.
Pull request overview
This PR extends the tiled-storage stack (follow-up to #52) by adding incremental writers for producing complete hash tiles across levels (TileWriterT) and optional raw-entry bundles (EntryBundleWriterT), plus corresponding tests and design documentation updates.
Changes:
- Added
TileWriterTandEntryBundleWriterTto incrementally persist newly-complete tiles/bundles while preserving immutability/idempotency and supporting bounded recovery. - Added new integration-style tests (
tiles_writer,tiles_entries) and expandedtiles_storedoctest coverage for sync-failure retry semantics in the writer. - Updated the tlog-tiles design document to specify writer/resume semantics and expected tile counts.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| test/tiles_writer.cpp | Adds end-to-end tests for incremental tile writing, resume behavior, and bounded recovery. |
| test/tiles_store.cpp | Adds doctest cases validating writer behavior across injected directory-sync failures. |
| test/tiles_entries.cpp | Adds end-to-end tests for incremental entry-bundle writing, corruption repair on resume, and bounded recovery. |
| test/CMakeLists.txt | Registers the new tiles_writer and tiles_entries tests with CTest. |
| merklecpp_tiles.h | Introduces perfect_root, TileWriterT, and EntryBundleWriterT APIs and implementations. |
| doc/design/tlog-tiles.md | Documents the new writer APIs and their incremental/resume semantics within the tlog-tiles design. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Add TileWriterT (progressive full-tile writes with all-level rollups and bounded contiguous-prefix recovery) and EntryBundleWriterT (mirrors the writer for optional entry bundles). Both are incremental: existing full tiles/bundles are immutable and never rewritten, resume is idempotent, and recovery is caller-validated (the caller supplies the tree/entry state and re-derives it consistently across restarts). Add tiles_writer and tiles_entries tests covering exact full-tile/bundle counts at several sizes, incremental/idempotent writes, interior-hole repair on resume, and bounded recovery against sparse indices. Move the store durability-retry tests that exercise TileWriter into tiles_store, now that the writer exists. Document the write path, compaction integration, and entry-bundle writer in the design doc, and add the size-256/70000 worked examples. Co-authored-by: Copilot <[email protected]>
Co-authored-by: Copilot App <[email protected]> Copilot-Session: 8d3e4958-68c2-4b72-88ae-bba0b15052ae
Co-authored-by: Copilot App <[email protected]> Copilot-Session: 8d3e4958-68c2-4b72-88ae-bba0b15052ae
Co-authored-by: Copilot App <[email protected]> Copilot-Session: 8d3e4958-68c2-4b72-88ae-bba0b15052ae
eca982d to
6e2c46d
Compare
cjen1-msft
left a comment
There was a problem hiding this comment.
I think this is good.
Agent comments, but no blockers here.
- The validation is noted as being structural only ie not checking for disk corruption
- This should be alright.
- Test for a level 2 roll-up possibly useful?
- sha-384 or sha-512 could be nice to also test
Co-authored-by: Copilot App <[email protected]> Copilot-Session: 8d3e4958-68c2-4b72-88ae-bba0b15052ae
Added both cases to |
Summary
Part 2 of the tiled-storage series; follow up to merged #52.
TileWriterTfor progressive complete-tile production and all-level roll-ups;EntryBundleWriterTfor optional complete raw-entry bundles;Series
TiledTreelifecycle wrapperValidation
tiles_storetiles_writertiles_entries