Skip to content

Add incremental tile and entry-bundle writers#53

Merged
achamayou merged 6 commits into
mainfrom
achamayou/tiles-writers
Jul 25, 2026
Merged

Add incremental tile and entry-bundle writers#53
achamayou merged 6 commits into
mainfrom
achamayou/tiles-writers

Conversation

@achamayou

@achamayou achamayou commented Jul 13, 2026

Copy link
Copy Markdown
Member

Summary

Part 2 of the tiled-storage series; follow up to merged #52.

  • add TileWriterT for progressive complete-tile production and all-level roll-ups;
  • add EntryBundleWriterT for optional complete raw-entry bundles;
  • recover from a caller-validated contiguous prefix without trusting sparse high-index files;
  • preserve immutable/idempotent writes while repairing malformed interior files;
  • retain and exercise durability retry behavior across writer and store instances;
  • specify write, resume, ownership, and entry-bundle semantics.

Series

  1. Add durable tlog-tiles storage primitives #52 — tile formats and durable storage (merged)
  2. This PR: incremental writers
  3. tile-backed proof generation
  4. TiledTree lifecycle wrapper
  5. user documentation and performance coverage

Validation

  • tiles_store
  • tiles_writer
  • tiles_entries
  • header clang-tidy

@achamayou
achamayou force-pushed the achamayou/tiles-writers branch from ebb6e57 to 5479ef7 Compare July 18, 2026 12:15
@achamayou
achamayou marked this pull request as ready for review July 20, 2026 17:59
@achamayou
achamayou requested a review from a team as a code owner July 20, 2026 17:59
@achamayou
achamayou requested review from Copilot and removed request for a team July 20, 2026 17:59

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR extends the tiled-storage stack (follow-up to #52) by adding incremental writers for producing complete hash tiles across levels (TileWriterT) and optional raw-entry bundles (EntryBundleWriterT), plus corresponding tests and design documentation updates.

Changes:

  • Added TileWriterT and EntryBundleWriterT to incrementally persist newly-complete tiles/bundles while preserving immutability/idempotency and supporting bounded recovery.
  • Added new integration-style tests (tiles_writer, tiles_entries) and expanded tiles_store doctest coverage for sync-failure retry semantics in the writer.
  • Updated the tlog-tiles design document to specify writer/resume semantics and expected tile counts.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 6 comments.

Show a summary per file
File Description
test/tiles_writer.cpp Adds end-to-end tests for incremental tile writing, resume behavior, and bounded recovery.
test/tiles_store.cpp Adds doctest cases validating writer behavior across injected directory-sync failures.
test/tiles_entries.cpp Adds end-to-end tests for incremental entry-bundle writing, corruption repair on resume, and bounded recovery.
test/CMakeLists.txt Registers the new tiles_writer and tiles_entries tests with CTest.
merklecpp_tiles.h Introduces perfect_root, TileWriterT, and EntryBundleWriterT APIs and implementations.
doc/design/tlog-tiles.md Documents the new writer APIs and their incremental/resume semantics within the tlog-tiles design.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread merklecpp_tiles.h Outdated
Comment thread merklecpp_tiles.h Outdated
Comment thread test/tiles_writer.cpp Outdated
Comment thread test/tiles_entries.cpp Outdated
Comment thread test/tiles_writer.cpp
Comment thread test/tiles_entries.cpp
@achamayou
achamayou changed the base branch from achamayou/tiles-store to main July 20, 2026 19:47
achamayou and others added 4 commits July 20, 2026 20:49
Add TileWriterT (progressive full-tile writes with all-level rollups
and bounded contiguous-prefix recovery) and EntryBundleWriterT (mirrors
the writer for optional entry bundles). Both are incremental: existing
full tiles/bundles are immutable and never rewritten, resume is
idempotent, and recovery is caller-validated (the caller supplies the
tree/entry state and re-derives it consistently across restarts).

Add tiles_writer and tiles_entries tests covering exact full-tile/bundle
counts at several sizes, incremental/idempotent writes, interior-hole
repair on resume, and bounded recovery against sparse indices. Move the
store durability-retry tests that exercise TileWriter into tiles_store,
now that the writer exists.

Document the write path, compaction integration, and entry-bundle
writer in the design doc, and add the size-256/70000 worked examples.

Co-authored-by: Copilot <[email protected]>
Co-authored-by: Copilot App <[email protected]>
Copilot-Session: 8d3e4958-68c2-4b72-88ae-bba0b15052ae
Co-authored-by: Copilot App <[email protected]>
Copilot-Session: 8d3e4958-68c2-4b72-88ae-bba0b15052ae
Co-authored-by: Copilot App <[email protected]>
Copilot-Session: 8d3e4958-68c2-4b72-88ae-bba0b15052ae
@achamayou
achamayou force-pushed the achamayou/tiles-writers branch from eca982d to 6e2c46d Compare July 20, 2026 19:51

@cjen1-msft cjen1-msft left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is good.

Agent comments, but no blockers here.

  • The validation is noted as being structural only ie not checking for disk corruption
    • This should be alright.
  • Test for a level 2 roll-up possibly useful?
  • sha-384 or sha-512 could be nice to also test

Co-authored-by: Copilot App <[email protected]>
Copilot-Session: 8d3e4958-68c2-4b72-88ae-bba0b15052ae
@achamayou

Copy link
Copy Markdown
Member Author

Test for a level 2 roll-up possibly useful?
sha-384 or sha-512 could be nice to also test

Added both cases to tiles_writer: a full level-2 roll-up over 256 level-1 tiles, and an OpenSSL-gated SHA-384 writer test that verifies the level-1 tile and tiled root against Tree384.

@achamayou
achamayou merged commit 6d70d02 into main Jul 25, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants