From 8b22219f31aa3f7531a6d6d856717ff634fecd9f Mon Sep 17 00:00:00 2001 From: Amaury Chamayou Date: Sat, 13 Jun 2026 15:06:44 +0000 Subject: [PATCH 01/30] docs: add tlog-tiles tiled-storage and proof design Design for persisting the merkle tree as tlog-tiles tile files progressively (on compaction via flush_to) and serving inclusion and consistency proofs from tiles, the in-memory tree, or a combination. Hashing is unchanged: tiles and proofs are templated on the tree's existing HASH_FUNCTION, so tile-derived proofs are byte-identical to Tree::path()/past_path() and verify with PathT::verify(). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/tlog-tiles.md | 713 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 713 insertions(+) create mode 100644 doc/design/tlog-tiles.md diff --git a/doc/design/tlog-tiles.md b/doc/design/tlog-tiles.md new file mode 100644 index 0000000..af93558 --- /dev/null +++ b/doc/design/tlog-tiles.md @@ -0,0 +1,713 @@ +# Design: Tiled storage and tile-backed proofs for merklecpp + +Status: Proposal / Draft +Audience: merklecpp maintainers and contributors +Scope: Extend the header-only library to (a) persist the Merkle tree as +[tlog-tiles](https://c2sp.org/tlog-tiles) tile files progressively (on +compaction), and (b) serve **inclusion** and **consistency** proofs from those +tiles, from the in-memory tree, or from a combination of the two. + +--- + +## 1. Goals and non-goals + +### Goals + +1. Write the tree to disk as a set of immutable, cacheable **tile files** using + the [tlog-tiles](https://c2sp.org/tlog-tiles) file/directory layout and tile + geometry (256-wide tiles, 8 tree levels per tile, partial-tile rules, path + encoding). +2. Produce tiles **progressively**: each batch/checkpoint writes only the newly + completed tiles, and the write integrates with the existing `flush_to()` + "compaction" so that flushed (evicted) subtrees are durably persisted before + their nodes are freed from memory. +3. Provide an API to retrieve, **after compaction**, an **inclusion path** and a + **consistency path**: + - purely from tiles, + - purely from the in-memory tree (existing behaviour), or + - from a combination of tiles (old, flushed part) and the in-memory tree + (recent frontier). +4. Keep everything **header-only** and templated, matching the existing + `TreeT` style. + +### Hard constraints (from the request) + +- **Hashing logic is unchanged.** No modification to `HASH_FUNCTION`, to node + hashing, or to leaf handling. +- **Proofs remain compatible with the existing library.** A proof produced from + tiles MUST be identical to (and verify against the same root as) the proof the + existing library would produce via `Tree::path()` / `Tree::past_path()` / + `Tree::root()` / `Tree::past_root()`. + +### Non-goals + +- Byte-level interoperability with *external* RFC 6962 / Go-ecosystem tlog + clients. See [§2.3](#23-compatibility-statement): such interop is only + possible if the consumer instantiates `TreeT` with an RFC 6962 combiner, which + is **out of scope** here and not required by this design. +- Checkpoint **signing** (signed notes / Ed25519). We define a minimal, + unsigned checkpoint artifact and leave signing to the application. +- An HTTP server. We define the on-disk layout and the read/write/proof APIs; + serving the static files is an application concern. + +--- + +## 2. Background + +### 2.1 tlog-tiles, the parts we adopt + +A tiled log exposes the Merkle tree as a set of static resources: + +- **Tiles** at `/tile//[.p/]`, `application/octet-stream`. + - `` = level, decimal `0..63`, no leading zeros. + - `` = tile index within the level, encoded as zero-padded 3-digit path + elements where **all but the last element are prefixed with `x`**. Example: + `1234067` → `x001/x234/067`. + - `.p/` is present only for **partial** tiles; `` is the width `1..255`. +- A **full tile** is exactly **256 hashes** wide (`256 * HASH_SIZE` bytes). The + spec fixes `HASH_SIZE = 32` (SHA-256); here it is the template parameter. +- Level 0 hashes are **leaf hashes**. At level `L ≥ 1`, each hash is the Merkle + Tree Hash of a *full* tile at level `L-1`. A tile spans **8 tree levels**: its + 256 entries are the leaves of a height-8 perfect subtree, and intra-tile + internal nodes are reconstructed by hashing entries. +- The `n`-th tile at level `l` contains, for `i = 0..255`: + `MTH(D[(n*256+i)*256^l : (n*256+i+1)*256^l])`. + Its **start index** is `n*256^(l+1)`, its **end index** `(n+1)*256^(l+1)`. +- Some rightmost tiles are **partial**. The partial tile at level `l` for tree + size `s` has `floor(s / 256^l) mod 256` entries. **Empty tiles MUST NOT be + served.** Partial-tile entries are still complete `256^l`-leaf subtree roots; + only the *count* is partial. +- **Entry bundles** at `/tile/entries/[.p/]`: big-endian `uint16` + length-prefixed raw entries. (See [§6.8](#68-entry-bundles-optional).) +- A **checkpoint** (signed tree head) at `/checkpoint`. +- **Pruning**: a log keeps a *minimum index*; tiles/bundles whose end index is + `≤ minimum index` may be denied. This maps cleanly onto merklecpp's + `flush_to()` / `min_index()`. + +### 2.2 The merklecpp model + +- `TreeT` is a **left-balanced binary Merkle tree**. + Its shape is identical to RFC 6962: a node's left child is the largest perfect + subtree of `2^k` leaves with `2^k < n`. A *full* node of height `h` covers + exactly `2^(h-1)` aligned leaves, and its hash is the root of that perfect + subtree (verified from `walk_to`, `is_full`, `update_sizes`). +- Internal nodes are combined with `HASH_FUNCTION(left, right, out)` over exactly + two child hashes (`merklecpp.h` node hashing). There is **no** domain + separation; the combiner is the only cryptographic operation. +- `flush_to(index)` is the existing **compaction** primitive: it conflates the + left part of the tree into single hash-only nodes and drops `num_flushed` + leaves from memory, raising `min_index()` to `index`. After a flush, paths for + leaves `< index` can no longer be produced from memory. +- Existing proof APIs we must stay compatible with: + - `root()` / `past_root(i)` — current / historical root. + - `path(i)` — inclusion path for leaf `i` in the current tree. + - `past_path(i, as_of)` — inclusion path for leaf `i` as of size `as_of+1`. + - `PathT::verify(root)` / `PathT::root()` — verification via `HASH_FUNCTION`. + - merklecpp has **no** classic two-size consistency proof today; this design + adds one (built from the same combiner, so it reconciles `past_root`s). + +### 2.3 Compatibility statement + +The tile geometry, path encoding, partial-tile rules, and the inclusion / +consistency proof **algorithms** are exactly those of tlog-tiles / RFC 6962. +The **hash values stored in tiles** are produced by the tree's existing +`HASH_FUNCTION`. Because + +1. merklecpp's tree shape equals the RFC 6962 left-balanced shape, and +2. tile entries and proof building blocks are *perfect-subtree roots* combined + with the **same** `HASH_FUNCTION` the tree already uses, + +a tile entry at level `L`, index `g` equals the in-memory full node of height +`8L+1` over leaves `[g·256^L, (g+1)·256^L)`, and a proof assembled from tiles is +**byte-identical** to the one `Tree::path()` / `Tree::past_path()` would emit. +Tile-derived proofs therefore verify with the unchanged `PathT::verify()`. + +> External RFC 6962 interop would additionally require RFC 6962 domain +> separation (`SHA256(0x00‖entry)` leaves, `SHA256(0x01‖l‖r)` nodes). merklecpp +> already lets a consumer pass such a `HASH_FUNCTION`; doing so is **optional and +> out of scope**. This design never assumes it. + +--- + +## 3. Tile ↔ merklecpp mapping (the math) + +Let `TILE_HEIGHT = 8` and `TILE_WIDTH = 256 = 2^8`. All combiners below are the +tree's `HASH_FUNCTION`; `MTH` denotes the Merkle Tree Hash computed with it. + +- **Tile entry = perfect subtree root.** Entry `i` of tile `(L, N)` (global + entry index `g = N·256 + i`) is `MTH(D[g·256^L : (g+1)·256^L])`, i.e. the root + of a perfect subtree of `2^(8L)` leaves — the in-memory full node of height + `8L + 1`. +- **Level roll-up.** Because a full level-`L` entry is the root of 256 full + level-`(L-1)` entries, a level-`L` tile can be computed from level-`(L-1)` + tiles alone: `tile(L,N)[i] = perfect_root( tile(L-1, N·256+i)[0..255] )`. So + the **write path needs no tree internals above level 0** — only leaf hashes + plus roll-ups. +- **Intra-tile internal nodes.** A tile's 256 entries are the leaves of a + height-8 perfect subtree. The root of `2^r` (`0 ≤ r ≤ 8`) consecutive, + aligned entries is `perfect_root` of that run. Hence any RFC 6962 subtree root + at tree level `k = 8L + r` is obtained from **one** level-`L` tile by hashing + `2^r` consecutive entries (a single entry when `r = 0`). +- **Partial tiles.** Number of complete entries available at level `L` for size + `s` is `entries_L = floor(s / 256^L)`. Tile `N` holds entries + `[N·256, (N+1)·256)` clamped to `entries_L`. Width of tile `N`: + `clamp(entries_L − N·256, 0, 256)`. The partial (rightmost) width is + `entries_L mod 256`; width 0 ⇒ not served. + +This yields a single read primitive that both proof types are built on: + +``` +subtree_root(level k, index j) = MTH(D[j·2^k : (j+1)·2^k]) + L = k / 8 ; r = k % 8 + first = j << r // first level-L entry covered + N = first / 256 ; off = first % 256 // tile and offset within tile + return perfect_root( tile(L, N)[off : off + 2^r] ) // single entry if r==0 +``` + +and a range primitive for non-perfect (right-frontier) subtrees: + +``` +mth_range(a, b): // MTH(D[a:b]); a aligned to 2^ceil(log2(b-a)) + w = b - a + if w == 2^k and source can resolve subtree_root(k, a>>k): return it + k = largest power of two < w // RFC 6962 split + return HASH_FUNCTION( mth_range(a, a+k), mth_range(a+k, b) ) +``` + +`mth_range` falls back to splitting when a perfect subtree is not directly +resolvable from the chosen source; the recursion always bottoms out at resolvable +pieces (see the [combined-source invariant](#7-progressive-production--compaction)). + +--- + +## 4. File and directory layout + +Rooted at a configurable `prefix` directory on local disk: + +``` +/ + checkpoint # mutable: latest (unsigned) + tile/ + 0/ # level 0 (leaf hashes) + 000 001 ... 255 # full tiles (8192 B each for HASH_SIZE=32) + x001/ + 000 ... 255 + .p/ # partial tile, e.g. 273.p/112 + 1/ # level 1 (roll-ups of level-0 full tiles) + 000 ... + .../ + entries/ # optional raw entry bundles + 000 001 ... + .p/ +``` + +### Index encoding (`encode_tile_index`) + +``` +encode_tile_index(N): + parts = [] + do { parts.push_front(printf("%03d", N % 1000)); N /= 1000 } while (N > 0) + for each part except the last: prepend 'x' + return join(parts, "/") +``` + +Examples: `5 → "005"`, `255 → "255"`, `1000 → "x001/000"`, +`1234067 → "x001/x234/067"`. + +Resource paths: + +- Full tile: `tile//` +- Partial tile: `tile//.p/` +- Entry bundle: `tile/entries/[.p/]` + +Tile byte format: the `W` entries concatenated, each `HASH_SIZE` raw bytes +(`HashT::bytes`); a full tile is `256 * HASH_SIZE` bytes. + +--- + +## 5. Architecture overview + +``` + append(leaf hash) checkpoint()/flush + caller ───────────────▶ TiledTreeT ───────────────▶ TileStoreT (disk I/O) + │ owns Tree ▲ + │ │ read tiles + inclusion/consistency proof requests │ + ▼ │ + ProofEngine ──▶ HashSource ◀────────┘ + ├─ MemoryHashSource (in-memory Tree) + ├─ TileHashSource (TileStore + size) + └─ CombinedHashSource +``` + +New, all in a companion header `merklecpp_tiles.h` (includes `merklecpp.h`), +namespace `merkle::tiles`, every type templated on `` +with default aliases mirroring the bottom of `merklecpp.h`: + +| Component | Responsibility | +|----------------------|-----------------------------------------------------------------------| +| `TileCoord` | pure index math + path encoding (no I/O) | +| `TileStoreT` | read/write tile & checkpoint files on a local filesystem | +| `TileWriterT` | compute & persist newly-complete full/partial tiles (write path) | +| `HashSource` impls | resolve `subtree_root` from tiles, memory, or both | +| `ProofEngineT` | `mth_range`, `inclusion_proof`, `consistency_proof`, verifiers | +| `TiledTreeT` | convenience wrapper: `append` / `checkpoint` / `prove*` / compaction | + +The **only** (optional) change to the core header is a small, read-only, +**non-hashing** accessor (`subtree_root`) used by `MemoryHashSource`; see +[§6.2](#62-optional-core-accessor-non-hashing). + +--- + +## 6. Public API design + +### 6.1 Header, namespace, aliases + +```cpp +// merklecpp_tiles.h +#include "merklecpp.h" +#include +#include + +namespace merkle { +namespace tiles { + +static constexpr uint16_t TILE_HEIGHT = 8; +static constexpr uint16_t TILE_WIDTH = 256; // 2^TILE_HEIGHT + +template &, + const HashT&, + HashT&)> +class TileStoreT { /* ... */ }; +// ... TileWriterT, ProofEngineT, TiledTreeT ... + +} // namespace tiles + +// Convenience aliases (mirror merklecpp.h) +using TileStore = tiles::TileStoreT<32, sha256_compress>; +using TiledTree = tiles::TiledTreeT<32, sha256_compress>; +// + 384/512 variants +} // namespace merkle +``` + +Note the roll-up/proof combiner is the **same** `HASH_FUNCTION` the tree uses, +and it only ever combines two `HASH_SIZE`-byte hashes — so the default +`sha256_compress` single-block path is sufficient and **no OpenSSL dependency is +introduced**. + +### 6.2 Optional core accessor (non-hashing) + +To let proofs be served partly from the resident tree, add one small read-only +method to `TreeT`. It performs **no hashing changes**: it navigates to an +existing node and returns its already-computed hash. + +```cpp +// In TreeT (merklecpp.h). Returns true and sets `out` to MTH(D[j< read_tile(const TileRef&) const; // width entries + void write_tile(const TileRef&, const std::vector&); // atomic (tmp+rename) + + // Single entry convenience (used heavily by TileHashSource) + bool read_entry(uint8_t level, uint64_t global_entry, Hash& out, uint64_t size) const; + + // Checkpoint (unsigned: size + root) + void write_checkpoint(uint64_t size, const Hash& root); + bool read_checkpoint(uint64_t& size, Hash& root) const; +}; +``` + +- Writes are **atomic** (write to a temp file, `rename`) so a crash never leaves + a half-written immutable tile. +- Full tiles are written once and never rewritten (immutability). Partial tiles + are overwritten as the frontier advances and removed once the covering full + tile exists (spec-permitted; configurable). +- An in-process LRU cache of recently read tiles avoids repeated I/O during + proof generation. + +### 6.4 Write path — `TileWriterT` (progressive, on compaction) + +```cpp +class TileWriterT { +public: + struct Options { + bool write_partial_tiles = true; // required for checkpoint sizes + bool write_higher_levels = true; // roll up L>=1 each checkpoint + bool remove_superseded_partials = true; + }; + + TileWriterT(TileStoreT& store, Options = {}); + + // Persist all newly-complete full tiles (all levels) and, for `size`, + // the partial tiles. Incremental: only writes tiles not already present. + // `leaf_at` supplies level-0 leaf hashes for [0, size) (e.g. Tree::leaf). + void write_up_to(uint64_t size, + const std::function& leaf_at); +}; +``` + +Algorithm (`write_up_to`): + +``` +for L = 0, 1, 2, ... while entries_L = size >> (8*L) is > 0: + full_L = entries_L / 256 + for N in [first_unwritten_full(L) .. full_L): # new full tiles only + entries = [ entry(L, N*256 + i) for i in 0..255 ] + store.write_tile({L, N, width=0}, entries) + if write_partial_tiles: + w = entries_L % 256 + if w > 0: + entries = [ entry(L, full_L*256 + i) for i in 0..w-1 ] + store.write_tile({L, full_L, width=w}, entries) + if remove_superseded_partials: remove stale partials < full_L + +entry(L, g): + if L == 0: return leaf_at(g) + else: return perfect_root( store.read_tile({L-1, g, width=0}) ) # 256→1 +``` + +- Level 0 reads leaf hashes via the supplied `leaf_at` (e.g. `Tree::leaf(i)`), + which exist **before** flushing. Higher levels roll up from the level-0 tiles + just written — independent of tree internals and robust to prior flushes. +- `first_unwritten_full(L)` is tracked in the store (probe `has_full_tile`, or a + small persisted cursor) to keep each checkpoint O(new tiles). + +Compaction integration (the "on compaction" story): + +```cpp +// In TiledTreeT::checkpoint(): write durable tiles, THEN free memory. +writer.write_up_to(size, [&](uint64_t i) -> const Hash& { return tree.leaf(i); }); +store.write_checkpoint(size, tree.root()); +uint64_t covered = (size / TILE_WIDTH) * TILE_WIDTH; // full level-0 coverage +tree.flush_to(covered - retention_margin); // never past `covered` +``` + +### 6.5 `HashSource` — tiles, memory, or both + +```cpp +struct HashSource { // concept (duck-typed or virtual) + // MTH(D[index< inclusion_proof(uint64_t index, uint64_t size) const; + + // RFC 6962 consistency proof that size `m` is a prefix of size `n` (m<=n). + std::vector consistency_proof(uint64_t m, uint64_t n) const; + + // Verifier (consistency is new to merklecpp; inclusion reuses PathT::verify). + static bool verify_consistency(uint64_t m, uint64_t n, + const Hash& old_root, const Hash& new_root, + const std::vector& proof); +}; +``` + +Inclusion (top-down; element order/`direction` chosen to match `Tree::path`): + +``` +elements = [] # leaf→root order via push_front +lo = 0, hi = size, idx = index +while hi - lo > 1: + k = largest_pow2_lt(hi - lo) # split at lo+k + if idx - lo < k: # target in left ⇒ sibling on the RIGHT + sib = mth_range(lo+k, hi); dir = PATH_RIGHT; hi = lo + k + else: # target in right ⇒ sibling on the LEFT + sib = mth_range(lo, lo+k); dir = PATH_LEFT; lo = lo + k + elements.push_front({sib, dir}) +leaf = src.leaf(index) +return Path(leaf, index, elements, max_index = size - 1) +``` + +Consistency (RFC 6962 `SUBPROOF`): + +``` +consistency_proof(m, n): # 0 < m <= n + if m == n: return [] + subproof(m, lo=0, hi=n, complete=true) + +subproof(m, lo, hi, complete): + if m == hi - lo: + if not complete: proof.push_back(mth_range(lo, hi)) + return + k = largest_pow2_lt(hi - lo) + if m <= k: + subproof(m, lo, lo+k, complete) + proof.push_back(mth_range(lo+k, hi)) + else: + subproof(m-k, lo+k, hi, false) + proof.push_back(mth_range(lo, lo+k)) +``` + +Because every emitted hash is an `mth_range` computed with `HASH_FUNCTION`, the +consistency proof reconciles `Tree::past_root(m-1)` with +`Tree::past_root(n-1)` — i.e. it is consistent with the existing library. + +### 6.7 `TiledTreeT` — convenience wrapper + +```cpp +class TiledTreeT { +public: + struct Config { + std::filesystem::path prefix; + uint64_t retention_margin = 0; // keep this many recent leaves resident + TileWriterT::Options writer = {}; + }; + explicit TiledTreeT(Config); + + void append(const Hash& leaf_hash); // tree.insert + uint64_t size() const; // tree.num_leaves + Hash root(); // tree.root + + // Write newly-complete tiles + partials + checkpoint, then flush_to(coverage). + void checkpoint(); + + // Proofs over tiles ∪ resident tree (works for flushed indices). + std::shared_ptr inclusion_proof(uint64_t index, uint64_t size); + std::vector consistency_proof(uint64_t m, uint64_t n); + + Tree& tree(); // escape hatch +}; +``` + +Stateless alternative for callers with their own storage: free functions +`write_tiles(tree, store, size, opts)` and a `ProofEngineT` built on a +`CombinedHashSource`, so the wrapper is optional sugar. + +### 6.8 Entry bundles (optional) + +merklecpp never sees raw entries (callers insert pre-computed leaf hashes), so +entry bundles are an **application-owned** add-on, included for completeness: + +- `tile/entries/[.p/]` stores big-endian `uint16` length-prefixed entries, + 256 per full bundle. +- The application is responsible for the leaf-hash derivation it uses (e.g. + `leaf_hash = H(entry)`); merklecpp stores whatever leaf hash is inserted. +- A `BundleWriter` mirrors `TileWriterT` (write full bundles on 256 boundaries, + partial bundle for the checkpoint size). Marked optional/secondary. + +--- + +## 7. Progressive production & compaction + +The pairing of tile writing with `flush_to` gives the central correctness +invariant: + +> **Coverage invariant.** Never flush past the durably-written full-level-0 +> coverage: `min_index() ≤ covered`, where +> `covered = floor(size / 256) * 256` after `write_up_to(size, …)`. + +Per checkpoint: + +1. `append(...)` new leaf hashes; compute `root()`. +2. `write_up_to(size, leaf_at)` — persist newly-complete **full** tiles at all + levels (incremental) and the **partial** tiles for `size`. +3. `write_checkpoint(size, root)`. +4. `flush_to(covered − retention_margin)` — reclaim memory. + +Given the invariant, every leaf and every perfect subtree is resolvable: + +- leaf `i < covered` ⇒ in a full level-0 tile; leaf `i ≥ min_index` ⇒ resident. + Since `min_index ≤ covered`, **every** leaf is in tiles ∪ memory. +- `mth_range` resolves a perfect subtree directly when it lies wholly in tiles + (`end ≤ covered`) or wholly in memory (`start ≥ min_index`); otherwise it + splits and recurses, terminating at resolvable pieces (leaves at worst). + +Hence inclusion and consistency proofs are always producible after compaction, +from tiles alone, from memory alone (when nothing relevant was flushed), or from +the combination — satisfying the request. + +Cost per checkpoint is `O(new full tiles + partial tiles)`; higher-level tiles +are cheap roll-ups of 256 child hashes. Proof generation is +`O(log(size))` `mth_range` calls, each at most one tile read plus a `≤ 256`-leaf +roll-up, served from cache in the common case. + +--- + +## 8. Pruning / minimum index + +tlog-tiles pruning maps directly onto merklecpp: + +- The log's *minimum index* is `tree.min_index()` (== `num_flushed`). +- "Deny tiles/bundles whose end index ≤ minimum index" is implemented by the + serving layer consulting `min_index()`; on-disk tiles may be retained + (recommended by the spec) so historical proofs remain producible. +- The unpruned default is `min_index() == 0` (no `flush_to`). + +`flush_to` is the mechanism; *retention policy* (when/whether to prune) is left +to the application, exactly as the spec leaves it to log ecosystems. + +--- + +## 9. Worked examples (used as test vectors) + +**Size 256:** one full level-0 tile (`tile/0/000`) and one partial level-1 tile +of width 1 (`tile/1/000.p/1`). No partial level-0 tile (width `256 mod 256 = 0`, +not served). + +**Size 70 000:** 273 full level-0 tiles + one partial level-0 of width 112 +(`70000 mod 256`); one full level-1 tile + one partial level-1 of width 17 +(`273 mod 256`); one partial level-2 of width 1 (`floor(70000/65536) = 1`). These +exact counts are asserted in tests. + +**Index encoding:** `1234067 → x001/x234/067`; `1000 → x001/000`; `255 → 255`. + +--- + +## 10. Implementation plan + +Each phase is independently testable; phases 1–4 require **no** core changes. + +**Phase 0 — Scaffolding.** Add `merklecpp_tiles.h` (includes `merklecpp.h`), +`merkle::tiles` namespace, geometry constants, default aliases. Wire a new test +group in `test/CMakeLists.txt` following `add_merklecpp_test`. + +**Phase 1 — Coordinates & store.** `TileCoord`/`encode_index`, `TileRef`, +`TileStoreT` (path building, atomic `write_tile`, `read_tile`, checkpoint +read/write). *Tests:* index-encoding vectors; tile byte round-trip. + +**Phase 2 — Write path.** `TileWriterT::write_up_to` (level-0 from `leaf_at`, +roll-ups, partials, incremental cursor; superseded-partial removal). *Tests:* +size 256 & 70 000 produce exactly the expected tile set/widths; full-tile +immutability (re-running writes nothing new). + +**Phase 3 — Hash sources & proof engine.** `TileHashSource`, `mth_range`, +`ProofEngineT::root/inclusion_proof/consistency_proof/verify_consistency`. +*Tests:* `root(size)` from tiles == `tree.root()`; `inclusion_proof(i,size)` +**equals** `tree.path(i)` (operator==) and verifies; `inclusion_proof(i, m)` +equals `tree.past_path(i, m-1)`; consistency proof reconciles +`tree.past_root(m-1)`/`tree.past_root(n-1)`. + +**Phase 4 — Combination & wrapper.** Optional core `subtree_root` accessor; +`MemoryHashSource`, `CombinedHashSource`, `TiledTreeT` (append/checkpoint/prove +with `flush_to`). *Tests:* build N leaves, checkpoint+flush, then prove +inclusion for a **flushed** index and a **resident** index against a non-flushed +reference tree's root; consistency across a flush boundary. + +**Phase 5 — Optional entry bundles** (`BundleWriter`) and docs (README "Usage" +snippet, link this design doc, add `merklecpp_tiles.h` to Doxygen inputs). + +Deliverables: `merklecpp_tiles.h`; `test/tiles_*.cpp`; CMake wiring; optional +one-method core addition; README/docs updates. + +--- + +## 11. Testing strategy + +The existing library is the **oracle** — this is how we guarantee "proofs remain +compatible": + +- **Equality, not just verification.** Assert `*inclusion_proof(i,size) == + *tree.path(i)` using the existing `PathT::operator==`, and likewise against + `past_path`. This is stronger than `verify()` and pins byte-compatibility. +- **Root agreement.** `ProofEngineT::root(size)` == `tree.root()` / + `tree.past_root(size-1)` for many random sizes. +- **Spec vectors.** Exact tile counts/widths for sizes 256 and 70 000; the empty + partial-tile rule; index-encoding strings (incl. `x001/x234/067`). +- **Flush/compaction.** Cross-check proofs for flushed indices against a twin + tree that was never flushed (same inserts) — roots must match and proofs must + verify. +- **Consistency proofs.** Standard RFC 6962 `verify_consistency` plus the + reconciliation-of-`past_root`s check; randomized `m < n`. +- **Templating.** Run a subset under `Tree`, `Tree384`, `Tree512` to confirm + hash-size independence. +- **Style.** Mirror existing standalone-`main()` tests (`test/util.h`, + `make_hashes`, `get_timeout`) and register via `add_merklecpp_test`. + +--- + +## 12. Risks, edge cases, open questions + +- **External interop (by design, no).** With the default combiner the tiles are + *not* byte-compatible with RFC 6962 tooling. Documented in + [§2.3](#23-compatibility-statement); opt-in via an RFC 6962 `HASH_FUNCTION` is + the consumer's choice and out of scope. +- **Filesystem dependency.** Tile I/O needs ``/``; isolated + in the companion header so the core stays dependency-free. +- **Partial-tile lifecycle.** Overwriting/removing partials must never race a + reader; atomic rename + "fetch full tile as fallback" (spec-sanctioned) + mitigate this. +- **`flush_to` alignment.** Must flush only to a 256-multiple (minus retention) + to uphold the coverage invariant; enforced inside `TiledTreeT::checkpoint`. +- **Very large indices.** Index math uses `uint64_t`; encoding handles + multi-group indices. Level bound `≤ 63` per spec (8 suffices for `2^64`). +- **Open question — `subtree_root` in core vs. `past_path`-derived memory + source.** Recommend the tiny non-hashing accessor; falls back to zero-core- + change if maintainers prefer. Either keeps hashing untouched. +- **Open question — checkpoint format.** We store unsigned `size + root`. + Aligning the text format with `tlog-checkpoint`/`signed-note` (origin line, + base64 root, signatures) can be layered on without affecting tiles or proofs. + +--- + +## 13. Build & backwards-compatibility impact + +- **Additive only.** New header + tests; existing `merklecpp.h` behaviour and + API are unchanged. Default builds (no tiles) are unaffected. +- **No new mandatory dependencies.** Roll-ups/proofs use the existing + `HASH_FUNCTION` (two-hash combiner), so `sha256_compress` suffices; OpenSSL + stays optional. +- **Optional core change** is a single read-only accessor that performs no + hashing and does not alter any existing code path. +- **CMake.** Add a `tiles` test group guarded like the others; no changes to the + `merklecpp` INTERFACE target are required for consumers that don't use tiles. +``` From 2a979655048950354208d9fd93fadad95bcc9d7d Mon Sep 17 00:00:00 2001 From: Amaury Chamayou Date: Sat, 13 Jun 2026 15:11:29 +0000 Subject: [PATCH 02/30] feat(tiles): scaffold merklecpp_tiles.h and TileStore (Phase 1) Add a header-only companion (merklecpp_tiles.h) implementing the tlog-tiles file/directory layout, templated on the tree's existing so no hashing logic changes: - TILE_HEIGHT/TILE_WIDTH geometry constants - encode_tile_index(): x-prefixed 3-digit-group path encoding - TileRef: level/index/width tile identifier - TileStoreT: tile_path/entries_path/checkpoint_path, atomic write_tile, read_tile, has_full_tile, and unsigned checkpoint (size + root) read/write - TileStore / TileStore384 / TileStore512 aliases (384/512 guarded by HAVE_OPENSSL, mirroring merklecpp.h) Add test/tiles_store.cpp covering index-encoding vectors, resource path layout, full/partial tile byte round-trips, width-mismatch rejection, and checkpoint round-trip against a real Tree root; wire it into test/CMakeLists.txt. All 12 tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- merklecpp_tiles.h | 320 +++++++++++++++++++++++++++++++++++++++++++ test/CMakeLists.txt | 1 + test/tiles_store.cpp | 179 ++++++++++++++++++++++++ 3 files changed, 500 insertions(+) create mode 100644 merklecpp_tiles.h create mode 100644 test/tiles_store.cpp diff --git a/merklecpp_tiles.h b/merklecpp_tiles.h new file mode 100644 index 0000000..71de86f --- /dev/null +++ b/merklecpp_tiles.h @@ -0,0 +1,320 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. + +#pragma once + +#include "merklecpp.h" + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +// Tiled storage for merklecpp trees, following the C2SP tlog-tiles layout +// (https://c2sp.org/tlog-tiles). The tile geometry, path encoding and partial +// tile rules match the specification; the hash values stored in tiles are +// produced by the tree's existing HASH_FUNCTION, so tile-derived proofs are +// byte-identical to those produced by merkle::TreeT (see +// doc/design/tlog-tiles.md). + +namespace merkle +{ + namespace tiles + { + /// @brief Number of tree levels spanned by a single tile. + static constexpr uint16_t TILE_HEIGHT = 8; + + /// @brief Number of hashes in a full tile (2**TILE_HEIGHT). + static constexpr uint16_t TILE_WIDTH = 256; + + /// @brief Encodes a tile index as tlog-tiles path elements. + /// @param n The tile index + /// @return The index as zero-padded, '/'-separated 3-digit groups, where + /// all but the last group are prefixed with 'x'. For example, 1234067 is + /// encoded as "x001/x234/067" and 5 as "005". + static inline std::string encode_tile_index(uint64_t n) + { + std::vector parts; + char buf[8]; + do + { + std::snprintf(buf, sizeof(buf), "%03u", (unsigned)(n % 1000)); + parts.emplace_back(buf); + n /= 1000; + } while (n > 0); + + std::string r; + for (size_t i = 0; i < parts.size(); i++) + { + const std::string& part = parts[parts.size() - 1 - i]; + if (i != 0) + { + r += "/"; + } + if (i + 1 < parts.size()) + { + r += "x"; + } + r += part; + } + return r; + } + + /// @brief Identifies a single tile within a tiled log. + struct TileRef + { + /// @brief The level of the tile (0 == leaf hashes). + uint8_t level = 0; + + /// @brief The index of the tile within its level. + uint64_t index = 0; + + /// @brief The width of the tile; 0 denotes a full tile (TILE_WIDTH). + uint16_t width = 0; + + /// @brief The number of hashes the tile holds. + [[nodiscard]] uint16_t num_hashes() const + { + return width == 0 ? TILE_WIDTH : width; + } + + /// @brief Whether the tile is partial. + [[nodiscard]] bool is_partial() const + { + return width != 0; + } + }; + + /// @brief Reads and writes tlog-tiles tile and checkpoint files on a local + /// filesystem. + /// @tparam HASH_SIZE Size of each hash in bytes + /// @tparam HASH_FUNCTION The tree's node hash function (carried for use by + /// later components; tile I/O itself does not hash). + template < + size_t HASH_SIZE, + void HASH_FUNCTION( + const HashT&, const HashT&, HashT&)> + class TileStoreT + { + public: + /// @brief The type of hashes stored in tiles. + using Hash = HashT; + + /// @brief Constructs a tile store rooted at @p prefix. + /// @param prefix The directory under which tiles and the checkpoint live. + explicit TileStoreT(std::filesystem::path prefix) : + prefix(std::move(prefix)) + {} + + /// @brief The root directory of the store. + [[nodiscard]] const std::filesystem::path& root() const + { + return prefix; + } + + /// @brief Encodes a tile index (see encode_tile_index). + static std::string encode_index(uint64_t n) + { + return encode_tile_index(n); + } + + /// @brief The filesystem path of a tile. + [[nodiscard]] std::filesystem::path tile_path(const TileRef& ref) const + { + std::string rel = "tile/" + std::to_string((unsigned)ref.level) + "/" + + encode_tile_index(ref.index); + if (ref.is_partial()) + { + rel += ".p/" + std::to_string((unsigned)ref.width); + } + return prefix / rel; + } + + /// @brief The filesystem path of an entry bundle. + [[nodiscard]] std::filesystem::path entries_path( + uint64_t index, uint16_t width = 0) const + { + std::string rel = "tile/entries/" + encode_tile_index(index); + if (width != 0) + { + rel += ".p/" + std::to_string((unsigned)width); + } + return prefix / rel; + } + + /// @brief The filesystem path of the checkpoint. + [[nodiscard]] std::filesystem::path checkpoint_path() const + { + return prefix / "checkpoint"; + } + + /// @brief Whether a full tile exists on disk. + [[nodiscard]] bool has_full_tile(uint8_t level, uint64_t index) const + { + return std::filesystem::exists(tile_path(TileRef{level, index, 0})); + } + + /// @brief Whether a specific tile exists on disk. + [[nodiscard]] bool has_tile(const TileRef& ref) const + { + return std::filesystem::exists(tile_path(ref)); + } + + /// @brief Writes a tile to disk atomically. + /// @param ref The tile to write + /// @param hashes The tile's hashes (exactly ref.num_hashes() of them) + void write_tile(const TileRef& ref, const std::vector& hashes) + { + if (hashes.size() != ref.num_hashes()) + { + throw std::runtime_error("tile width mismatch"); + } + + std::vector bytes; + bytes.reserve(hashes.size() * HASH_SIZE); + for (const auto& h : hashes) + { + h.serialise(bytes); + } + + write_file_atomically(tile_path(ref), bytes); + } + + /// @brief Reads a tile from disk. + /// @param ref The tile to read + /// @return The tile's hashes (ref.num_hashes() of them) + [[nodiscard]] std::vector read_tile(const TileRef& ref) const + { + std::vector bytes = read_file(tile_path(ref)); + const size_t expected = (size_t)ref.num_hashes() * HASH_SIZE; + if (bytes.size() != expected) + { + throw std::runtime_error("unexpected tile size"); + } + + std::vector hashes; + hashes.reserve(ref.num_hashes()); + size_t position = 0; + for (uint16_t i = 0; i < ref.num_hashes(); i++) + { + hashes.emplace_back(bytes, position); + } + return hashes; + } + + /// @brief Writes the (unsigned) checkpoint: tree size and root hash. + void write_checkpoint(uint64_t size, const Hash& root_hash) + { + std::string text = + std::to_string(size) + "\n" + root_hash.to_string() + "\n"; + std::vector bytes(text.begin(), text.end()); + write_file_atomically(checkpoint_path(), bytes); + } + + /// @brief Reads the checkpoint, if present. + /// @param size Set to the checkpoint tree size on success + /// @param root_hash Set to the checkpoint root hash on success + /// @return Whether a checkpoint was read + [[nodiscard]] bool read_checkpoint(uint64_t& size, Hash& root_hash) const + { + std::ifstream f(checkpoint_path(), std::ios::binary); + if (!f.good()) + { + return false; + } + std::string size_line; + std::string root_line; + if (!std::getline(f, size_line) || !std::getline(f, root_line)) + { + throw std::runtime_error("malformed checkpoint"); + } + trim(size_line); + trim(root_line); + size = std::stoull(size_line); + root_hash = Hash(root_line); + return true; + } + + protected: + /// @brief The root directory of the store. + std::filesystem::path prefix; + + /// @brief Removes trailing carriage-return/whitespace from a line. + static void trim(std::string& s) + { + while (!s.empty() && + (s.back() == '\r' || s.back() == '\n' || s.back() == ' ' || + s.back() == '\t')) + { + s.pop_back(); + } + } + + /// @brief Reads an entire file into a byte vector. + static std::vector read_file(const std::filesystem::path& path) + { + std::ifstream f(path, std::ios::binary); + if (!f.good()) + { + throw std::runtime_error("cannot open file: " + path.string()); + } + return std::vector( + (std::istreambuf_iterator(f)), + std::istreambuf_iterator()); + } + + /// @brief Writes a file atomically via a temporary file and rename. + /// @note Immutable tiles are never left half-written after a crash. + static void write_file_atomically( + const std::filesystem::path& path, const std::vector& bytes) + { + std::error_code ec; + std::filesystem::create_directories(path.parent_path(), ec); + if (ec) + { + throw std::runtime_error( + "cannot create directory " + path.parent_path().string() + ": " + + ec.message()); + } + + std::filesystem::path tmp = path; + tmp += ".tmp"; + { + std::ofstream f(tmp, std::ios::binary | std::ios::trunc); + if (!f.good()) + { + throw std::runtime_error("cannot open file: " + tmp.string()); + } + if (!bytes.empty()) + { + f.write( + reinterpret_cast(bytes.data()), + (std::streamsize)bytes.size()); + } + f.flush(); + if (!f.good()) + { + throw std::runtime_error("error writing file: " + tmp.string()); + } + } + std::filesystem::rename(tmp, path); + } + }; + + /// @brief Default tile store (SHA256, default hash function). + using TileStore = TileStoreT<32, sha256_compress>; + +#ifdef HAVE_OPENSSL + /// @brief SHA384 tile store. + using TileStore384 = TileStoreT<48, sha384_openssl>; + + /// @brief SHA512 tile store. + using TileStore512 = TileStoreT<64, sha512_openssl>; +#endif + } +} diff --git a/test/CMakeLists.txt b/test/CMakeLists.txt index e90a87c..a47f8a9 100644 --- a/test/CMakeLists.txt +++ b/test/CMakeLists.txt @@ -29,6 +29,7 @@ add_merklecpp_test(serialisation serialisation.cpp) add_merklecpp_test(partial_serialisation partial_serialisation.cpp) add_merklecpp_test(serialise_to_file serialise_to_file.cpp) add_merklecpp_test(coverage coverage.cpp) +add_merklecpp_test(tiles_store tiles_store.cpp) if(TARGET evercrypt.host) add_merklecpp_test(compare_evercrypt compare_evercrypt.cpp) diff --git a/test/tiles_store.cpp b/test/tiles_store.cpp new file mode 100644 index 0000000..388ec88 --- /dev/null +++ b/test/tiles_store.cpp @@ -0,0 +1,179 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. + +#include "util.h" + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace fs = std::filesystem; +using merkle::Hash; +using merkle::tiles::TileRef; + +static void expect(bool cond, const std::string& what) +{ + if (!cond) + { + throw std::runtime_error("check failed: " + what); + } +} + +static void expect_eq( + const std::string& got, const std::string& expected, const std::string& what) +{ + if (got != expected) + { + throw std::runtime_error( + what + ": got '" + got + "', expected '" + expected + "'"); + } +} + +static std::string rel(const merkle::tiles::TileStore& store, const fs::path& p) +{ + return p.lexically_relative(store.root()).generic_string(); +} + +int main() +{ + const auto seed = std::time(nullptr); + std::srand((unsigned)seed); + std::cout << "seed=" << seed << '\n'; + + const fs::path dir = fs::temp_directory_path() / + ("merklecpp_tiles_" + std::to_string((unsigned long long)seed) + "_" + + std::to_string(std::rand())); + + try + { + // 1. Tile index encoding vectors (tlog-tiles examples). + expect_eq(merkle::tiles::encode_tile_index(0), "000", "encode 0"); + expect_eq(merkle::tiles::encode_tile_index(5), "005", "encode 5"); + expect_eq(merkle::tiles::encode_tile_index(255), "255", "encode 255"); + expect_eq(merkle::tiles::encode_tile_index(999), "999", "encode 999"); + expect_eq( + merkle::tiles::encode_tile_index(1000), "x001/000", "encode 1000"); + expect_eq( + merkle::tiles::encode_tile_index(1234067), + "x001/x234/067", + "encode 1234067"); + + merkle::tiles::TileStore store(dir); + + // 2. Resource path layout. + expect_eq( + rel(store, store.tile_path(TileRef{0, 0, 0})), + "tile/0/000", + "tile_path L0 N0 full"); + expect_eq( + rel(store, store.tile_path(TileRef{0, 273, 112})), + "tile/0/273.p/112", + "tile_path L0 N273 partial"); + expect_eq( + rel(store, store.tile_path(TileRef{1, 1234067, 0})), + "tile/1/x001/x234/067", + "tile_path L1 big index"); + expect_eq( + rel(store, store.entries_path(5)), "tile/entries/005", "entries full"); + expect_eq( + rel(store, store.entries_path(5, 3)), + "tile/entries/005.p/3", + "entries partial"); + expect_eq( + rel(store, store.checkpoint_path()), "checkpoint", "checkpoint path"); + + const size_t hsz = Hash().size(); + + // 3a. Full tile byte round-trip. + const auto full = make_hashes(merkle::tiles::TILE_WIDTH); + const TileRef full_ref{0, 0, 0}; + store.write_tile(full_ref, full); + expect(store.has_full_tile(0, 0), "has_full_tile after write"); + expect(!store.has_full_tile(0, 5), "missing full tile"); + expect( + fs::file_size(store.tile_path(full_ref)) == + (uintmax_t)merkle::tiles::TILE_WIDTH * hsz, + "full tile file size"); + const auto full_rt = store.read_tile(full_ref); + expect(full_rt.size() == full.size(), "full tile width round-trip"); + for (size_t i = 0; i < full.size(); i++) + { + expect(full_rt[i] == full[i], "full tile hash round-trip"); + } + + // 3b. Partial tile byte round-trip. + const std::vector partial(full.begin(), full.begin() + 3); + const TileRef partial_ref{0, 1, 3}; + store.write_tile(partial_ref, partial); + expect( + fs::file_size(store.tile_path(partial_ref)) == (uintmax_t)3 * hsz, + "partial tile file size"); + const auto partial_rt = store.read_tile(partial_ref); + expect(partial_rt.size() == 3, "partial tile width round-trip"); + for (size_t i = 0; i < partial.size(); i++) + { + expect(partial_rt[i] == partial[i], "partial tile hash round-trip"); + } + + // 3c. Width mismatches are rejected. + bool threw = false; + try + { + store.write_tile(TileRef{0, 2, 0}, partial); // 3 hashes for a full tile + } + catch (const std::exception&) + { + threw = true; + } + expect(threw, "width mismatch rejected"); + + // 4. Checkpoint round-trip against a real tree root. + merkle::Tree tree; + for (const auto& h : full) + { + tree.insert(h); + } + const Hash root = tree.root(); + store.write_checkpoint(tree.num_leaves(), root); + + uint64_t rd_size = 0; + Hash rd_root; + expect(store.read_checkpoint(rd_size, rd_root), "checkpoint present"); + expect(rd_size == tree.num_leaves(), "checkpoint size round-trip"); + expect(rd_root == root, "checkpoint root round-trip"); + + // Missing checkpoint reports absence rather than throwing. + const merkle::tiles::TileStore empty_store(dir / "does_not_exist"); + uint64_t s2 = 0; + Hash r2; + expect( + !empty_store.read_checkpoint(s2, r2), "missing checkpoint returns false"); + + std::cout << "tiles_store: OK" << '\n'; + + std::error_code ec; + fs::remove_all(dir, ec); + } + catch (std::exception& ex) + { + std::cout << "Error: " << ex.what() << '\n'; + std::error_code ec; + fs::remove_all(dir, ec); + return 1; + } + catch (...) + { + std::cout << "Error" << '\n'; + std::error_code ec; + fs::remove_all(dir, ec); + return 1; + } + + return 0; +} From 2c2a329f0878cf79f6ef72957c342bc808964375 Mon Sep 17 00:00:00 2001 From: Amaury Chamayou Date: Sat, 13 Jun 2026 15:24:31 +0000 Subject: [PATCH 03/30] feat(tiles): add TileWriterT progressive write path (Phase 2) Add perfect_root (balanced power-of-two roll-up via the tree's HASH_FUNCTION) and TileWriterT to merklecpp_tiles.h: - write_up_to(size, leaf_at) writes every newly-complete full tile at all levels and the partial tiles for the current size, returning WriteStats (full/partial written, partials removed). - Level 0 comes from leaf_at; levels >= 1 roll up the complete child full tiles read from disk. - Balanced/immutable invariant enforced: only complete 2^(8L)-leaf subtree roots are tiled, full tiles are write-once (per-level cursor plus has_full_tile guard never rewrites one), and only partial tiles are grown or removed once superseded. - TileWriter[/384/512] aliases (384/512 guarded by HAVE_OPENSSL). test/tiles_writer.cpp covers: size 256 (+ cross-check that the L1 entry and roll-up equal the merkle::Tree root), the size-70000 spec example (exact tile set/widths), incremental immutability/idempotency, partial growth, and partial supersession. All 13 tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- merklecpp_tiles.h | 325 ++++++++++++++++++++++++++++++++++++++++++ test/CMakeLists.txt | 1 + test/tiles_writer.cpp | 220 ++++++++++++++++++++++++++++ 3 files changed, 546 insertions(+) create mode 100644 test/tiles_writer.cpp diff --git a/merklecpp_tiles.h b/merklecpp_tiles.h index 71de86f..0fb7b85 100644 --- a/merklecpp_tiles.h +++ b/merklecpp_tiles.h @@ -306,15 +306,340 @@ namespace merkle } }; + /// @brief Computes the Merkle Tree Hash of a perfect (balanced) subtree. + /// @param leaves The subtree's leaves; the count MUST be a power of two. + /// @return The subtree root, computed with the tree's HASH_FUNCTION. + /// @note This is exactly a merkle::TreeT full-node hash, which is why tile + /// entries (such roots) are immutable: an unbalanced subtree would still + /// change as leaves are added and must therefore never be tiled. + template < + size_t HASH_SIZE, + void HASH_FUNCTION( + const HashT&, const HashT&, HashT&)> + inline HashT perfect_root( + const std::vector>& leaves) + { + if (leaves.empty()) + { + throw std::runtime_error("perfect_root requires at least one leaf"); + } + if ((leaves.size() & (leaves.size() - 1)) != 0) + { + throw std::runtime_error( + "perfect_root requires a power-of-two number of leaves"); + } + + std::vector> level = leaves; + while (level.size() > 1) + { + std::vector> next; + next.reserve(level.size() / 2); + for (size_t i = 0; i + 1 < level.size(); i += 2) + { + HashT h; + HASH_FUNCTION(level[i], level[i + 1], h); + next.push_back(h); + } + level.swap(next); + } + return level.front(); + } + + /// @brief Computes and persists tlog-tiles tiles for a growing tree. + /// @tparam HASH_SIZE Size of each hash in bytes + /// @tparam HASH_FUNCTION The tree's node hash function + /// @note Only balanced subtrees are tiled: a level-L entry is the root of a + /// complete 2**(8L)-leaf subtree. Full tiles (256 such entries) are + /// therefore immutable and written exactly once; the incomplete frontier is + /// never tiled, and partial tiles (complete entries, partial count) are the + /// only resources that may be re-written (grown) or removed once superseded + /// by a full tile. + template < + size_t HASH_SIZE, + void HASH_FUNCTION( + const HashT&, const HashT&, HashT&)> + class TileWriterT + { + public: + /// @brief The type of hashes stored in tiles. + using Hash = HashT; + + /// @brief The associated tile store type. + using Store = TileStoreT; + + /// @brief Supplies the level-0 leaf hash for a given leaf index. + using LeafFn = std::function; + + /// @brief Write-path options. + struct Options + { + /// @brief Write the partial (rightmost) tiles for the current size. + bool write_partial_tiles = true; + + /// @brief Roll up and write tiles at levels >= 1. + bool write_higher_levels = true; + + /// @brief Remove partial tiles once superseded by a full tile (or a + /// wider partial at the same index). + bool remove_superseded_partials = true; + }; + + /// @brief Counts of work performed by a write_up_to call. + struct Stats + { + /// @brief Number of full tiles written. + uint64_t full_written = 0; + + /// @brief Number of partial tiles written. + uint64_t partial_written = 0; + + /// @brief Number of superseded partial tiles removed. + uint64_t partial_removed = 0; + }; + + /// @brief Constructs a writer over @p store. + explicit TileWriterT(Store& store, Options options = {}) : + store(store), options(options) + {} + + /// @brief Writes all newly-complete full tiles and the partial tiles for + /// a tree of @p size leaves. + /// @param size The current tree size + /// @param leaf_at Returns the level-0 leaf hash for a leaf index in + /// [0, size); only ever queried for leaves of complete subtrees. + /// @return Counts of tiles written/removed + /// @note Incremental: full tiles already on disk are immutable and are + /// never rewritten. + Stats write_up_to(uint64_t size, const LeafFn& leaf_at) + { + Stats stats; + const uint8_t max_level = options.write_higher_levels ? 63 : 0; + + for (uint8_t level = 0; level <= max_level; level++) + { + // Number of complete (balanced) level-L entries available; this + // deliberately excludes the incomplete frontier subtree. + const uint64_t entries = entries_at_level(size, level); + if (entries == 0) + { + break; + } + ensure_level(level); + + const uint64_t full_tiles = entries / TILE_WIDTH; + + if (!cursor_inited[level]) + { + next_full[level] = full_prefix_length(level); + cursor_inited[level] = true; + } + + for (uint64_t n = next_full[level]; n < full_tiles; n++) + { + if (store.has_full_tile(level, n)) + { + continue; // immutable: never rewrite an existing full tile + } + store.write_tile( + TileRef{level, n, 0}, + collect(level, n * TILE_WIDTH, TILE_WIDTH, leaf_at)); + stats.full_written++; + } + if (full_tiles > next_full[level]) + { + next_full[level] = full_tiles; + } + + if (options.write_partial_tiles) + { + write_partial( + level, + full_tiles, + (uint16_t)(entries % TILE_WIDTH), + leaf_at, + stats); + } + } + + return stats; + } + + protected: + /// @brief The tile store written to. + Store& store; + + /// @brief Write-path options. + Options options; + + /// @brief Per-level index of the next full tile to write. + std::vector next_full; + + /// @brief Per-level flag indicating next_full has been initialised. + std::vector cursor_inited; + + /// @brief Per-level last partial tile written by this writer. + std::vector last_partial; + + /// @brief Per-level flag indicating last_partial is valid. + std::vector has_last_partial; + + /// @brief Number of complete level-@p level entries for a tree of @p + /// size. + static uint64_t entries_at_level(uint64_t size, uint8_t level) + { + const unsigned shift = 8u * (unsigned)level; + return shift >= 64 ? 0 : (size >> shift); + } + + /// @brief Ensures per-level bookkeeping vectors cover @p level. + void ensure_level(uint8_t level) + { + const size_t needed = (size_t)level + 1; + if (next_full.size() < needed) + { + next_full.resize(needed, 0); + cursor_inited.resize(needed, false); + last_partial.resize(needed, TileRef{}); + has_last_partial.resize(needed, false); + } + } + + /// @brief Length of the contiguous prefix of full tiles already on disk. + [[nodiscard]] uint64_t full_prefix_length(uint8_t level) const + { + if (!store.has_full_tile(level, 0)) + { + return 0; + } + uint64_t lo = 0; // present + uint64_t hi = 1; + while (store.has_full_tile(level, hi)) + { + lo = hi; + hi <<= 1; + } + while (hi - lo > 1) + { + const uint64_t mid = lo + (hi - lo) / 2; + if (store.has_full_tile(level, mid)) + { + lo = mid; + } + else + { + hi = mid; + } + } + return lo + 1; + } + + /// @brief Collects @p count consecutive level-@p level entries, each the + /// root of a complete (balanced) subtree. + std::vector collect( + uint8_t level, + uint64_t first_entry, + uint64_t count, + const LeafFn& leaf_at) + { + std::vector out; + out.reserve(count); + for (uint64_t i = 0; i < count; i++) + { + const uint64_t g = first_entry + i; + if (level == 0) + { + out.push_back(leaf_at(g)); + } + else + { + // Roll up the complete child full tile (256 complete entries). + out.push_back(perfect_root( + store.read_tile(TileRef{(uint8_t)(level - 1), g, 0}))); + } + } + return out; + } + + /// @brief Writes (or removes) the partial tile at the rightmost index. + void write_partial( + uint8_t level, + uint64_t index, + uint16_t width, + const LeafFn& leaf_at, + Stats& stats) + { + if (width == 0) + { + // No partial at this size; a previous partial (if any) is now covered + // by a full tile and may be removed. + if (options.remove_superseded_partials && has_last_partial[level]) + { + remove_partial_dir(level, last_partial[level].index); + has_last_partial[level] = false; + stats.partial_removed++; + } + return; + } + + const TileRef ref{level, index, width}; + + if ( + has_last_partial[level] && last_partial[level].index == index && + last_partial[level].width == width) + { + return; // identical partial already written by this writer + } + + if (options.remove_superseded_partials && has_last_partial[level]) + { + if (last_partial[level].index != index) + { + // Previous partial's index is now covered by a full tile. + remove_partial_dir(level, last_partial[level].index); + stats.partial_removed++; + } + else + { + // Same index, narrower width: drop the stale width file(s). + remove_partial_dir(level, index); + } + } + + store.write_tile( + ref, collect(level, index * TILE_WIDTH, width, leaf_at)); + stats.partial_written++; + last_partial[level] = ref; + has_last_partial[level] = true; + } + + /// @brief Removes the partial-tile directory tile//.p. + void remove_partial_dir(uint8_t level, uint64_t index) + { + const std::filesystem::path dir = + store.tile_path(TileRef{level, index, 1}).parent_path(); + std::error_code ec; + std::filesystem::remove_all(dir, ec); + } + }; + /// @brief Default tile store (SHA256, default hash function). using TileStore = TileStoreT<32, sha256_compress>; + /// @brief Default tile writer (SHA256, default hash function). + using TileWriter = TileWriterT<32, sha256_compress>; + #ifdef HAVE_OPENSSL /// @brief SHA384 tile store. using TileStore384 = TileStoreT<48, sha384_openssl>; /// @brief SHA512 tile store. using TileStore512 = TileStoreT<64, sha512_openssl>; + + /// @brief SHA384 tile writer. + using TileWriter384 = TileWriterT<48, sha384_openssl>; + + /// @brief SHA512 tile writer. + using TileWriter512 = TileWriterT<64, sha512_openssl>; #endif } } diff --git a/test/CMakeLists.txt b/test/CMakeLists.txt index a47f8a9..c14c58b 100644 --- a/test/CMakeLists.txt +++ b/test/CMakeLists.txt @@ -30,6 +30,7 @@ add_merklecpp_test(partial_serialisation partial_serialisation.cpp) add_merklecpp_test(serialise_to_file serialise_to_file.cpp) add_merklecpp_test(coverage coverage.cpp) add_merklecpp_test(tiles_store tiles_store.cpp) +add_merklecpp_test(tiles_writer tiles_writer.cpp) if(TARGET evercrypt.host) add_merklecpp_test(compare_evercrypt compare_evercrypt.cpp) diff --git a/test/tiles_writer.cpp b/test/tiles_writer.cpp new file mode 100644 index 0000000..74bfd54 --- /dev/null +++ b/test/tiles_writer.cpp @@ -0,0 +1,220 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. + +#include "util.h" + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace fs = std::filesystem; +using merkle::Hash; +using merkle::tiles::TileRef; +using merkle::tiles::TileStore; +using merkle::tiles::TileWriter; + +static void expect(bool cond, const std::string& what) +{ + if (!cond) + { + throw std::runtime_error("check failed: " + what); + } +} + +static bool partial_dir_exists(const TileStore& store, uint8_t L, uint64_t idx) +{ + return fs::exists(store.tile_path(TileRef{L, idx, 1}).parent_path()); +} + +// Roll up a full level-0 tile and compare with a level-1 tile entry. +static Hash rollup(const std::vector& leaves) +{ + return merkle::tiles::perfect_root<32, merkle::sha256_compress>(leaves); +} + +int main() +{ + const auto seed = std::time(nullptr); + std::srand((unsigned)seed); + std::cout << "seed=" << seed << '\n'; + + const fs::path base = fs::temp_directory_path() / + ("merklecpp_tiles_writer_" + std::to_string((unsigned long long)seed) + + "_" + std::to_string(std::rand())); + + try + { + // ---- A. size 256: one full L0 tile + one width-1 L1 partial, and the L1 + // entry equals the real merkle::Tree root (roll-up == tree hash). + { + const auto hashes = make_hashes(256); + const auto leaf_at = [&](uint64_t i) -> const Hash& { return hashes[i]; }; + + TileStore store(base / "a"); + TileWriter writer(store); + const auto s = writer.write_up_to(256, leaf_at); + + expect(s.full_written == 1, "A full_written"); + expect(s.partial_written == 1, "A partial_written"); + expect(s.partial_removed == 0, "A partial_removed"); + + expect(store.has_full_tile(0, 0), "A L0 full tile"); + expect( + fs::file_size(store.tile_path(TileRef{0, 0, 0})) == + 256u * Hash().size(), + "A L0 full tile size"); + expect(!partial_dir_exists(store, 0, 0), "A no L0 partial"); + expect(!store.has_full_tile(1, 0), "A no L1 full tile"); + expect(store.has_tile(TileRef{1, 0, 1}), "A L1 partial width 1"); + + // Level-0 tile is the leaf hashes verbatim. + const auto l0 = store.read_tile(TileRef{0, 0, 0}); + for (size_t i = 0; i < hashes.size(); i++) + { + expect(l0[i] == hashes[i], "A L0 entry == leaf"); + } + + // Level-1 entry == root of the equivalent merkle::Tree. + merkle::Tree tree; + for (const auto& h : hashes) + { + tree.insert(h); + } + const Hash root = tree.root(); + const auto l1 = store.read_tile(TileRef{1, 0, 1}); + expect(l1.size() == 1, "A L1 partial width"); + expect(l1[0] == root, "A L1 entry == tree root"); + expect(rollup(l0) == root, "A rollup(L0) == tree root"); + + std::cout << "A (size 256): OK" << '\n'; + } + + // ---- B. size 70000: the spec's worked example, exact tile set/widths. + { + const auto hashes = make_hashes(70000); + const auto leaf_at = [&](uint64_t i) -> const Hash& { return hashes[i]; }; + + TileStore store(base / "b"); + TileWriter writer(store); + const auto s = writer.write_up_to(70000, leaf_at); + + // 273 full L0 + 1 full L1 = 274 full; partials at L0, L1, L2. + expect(s.full_written == 274, "B full_written"); + expect(s.partial_written == 3, "B partial_written"); + expect(s.partial_removed == 0, "B partial_removed"); + + expect(store.has_full_tile(0, 0), "B L0 tile 0"); + expect(store.has_full_tile(0, 272), "B L0 tile 272"); + expect(!store.has_full_tile(0, 273), "B no L0 tile 273"); + expect(store.has_tile(TileRef{0, 273, 112}), "B L0 partial width 112"); + expect( + fs::file_size(store.tile_path(TileRef{0, 273, 112})) == + 112u * Hash().size(), + "B L0 partial size"); + + expect(store.has_full_tile(1, 0), "B L1 tile 0"); + expect(!store.has_full_tile(1, 1), "B no L1 tile 1"); + expect(store.has_tile(TileRef{1, 1, 17}), "B L1 partial width 17"); + + expect(store.has_tile(TileRef{2, 0, 1}), "B L2 partial width 1"); + expect(!fs::exists(store.root() / "tile" / "3"), "B no level 3"); + + // Higher-level entries are roll-ups of the complete child tiles. + const auto l1 = store.read_tile(TileRef{1, 0, 0}); + expect(l1.size() == 256, "B L1 full width"); + expect(l1[0] == rollup(store.read_tile(TileRef{0, 0, 0})), "B L1[0]"); + expect( + l1[255] == rollup(store.read_tile(TileRef{0, 255, 0})), "B L1[255]"); + + std::cout << "B (size 70000): OK" << '\n'; + } + + // ---- C. incremental writes: immutability, idempotency, partial growth. + { + const auto hashes = make_hashes(1024); + const auto leaf_at = [&](uint64_t i) -> const Hash& { return hashes[i]; }; + + TileStore store(base / "c"); + TileWriter writer(store); + + const auto s1 = writer.write_up_to(256, leaf_at); + expect(s1.full_written == 1, "C s1 full"); + + // Re-running at the same size writes nothing (full tiles are immutable). + const auto s2 = writer.write_up_to(256, leaf_at); + expect(s2.full_written == 0, "C s2 full immutable"); + expect(s2.partial_written == 0, "C s2 partial idempotent"); + expect(s2.partial_removed == 0, "C s2 nothing removed"); + + // Grow to 600: one new full L0 tile, new L0 partial, L1 partial widens. + const auto s3 = writer.write_up_to(600, leaf_at); + expect(s3.full_written == 1, "C s3 full"); + expect(s3.partial_written == 2, "C s3 partial"); + + expect(store.has_full_tile(0, 0), "C L0 tile 0"); + expect(store.has_full_tile(0, 1), "C L0 tile 1"); + expect(store.has_tile(TileRef{0, 2, 88}), "C L0 partial width 88"); + expect(store.has_tile(TileRef{1, 0, 2}), "C L1 partial width 2"); + expect(!store.has_tile(TileRef{1, 0, 1}), "C old L1 partial removed"); + expect(!store.has_full_tile(1, 0), "C no L1 full tile"); + + std::cout << "C (incremental): OK" << '\n'; + } + + // ---- D. supersession: a partial whose index becomes a full tile is + // removed when it is covered. + { + const auto hashes = make_hashes(1024); + const auto leaf_at = [&](uint64_t i) -> const Hash& { return hashes[i]; }; + + TileStore store(base / "d"); + TileWriter writer(store); + + writer.write_up_to(300, leaf_at); + expect(store.has_full_tile(0, 0), "D L0 tile 0"); + expect(store.has_tile(TileRef{0, 1, 44}), "D L0 partial width 44"); + expect(store.has_tile(TileRef{1, 0, 1}), "D L1 partial width 1"); + + const auto s = writer.write_up_to(512, leaf_at); + expect(s.full_written == 1, "D full"); + expect(s.partial_removed == 1, "D one partial superseded"); + + expect(store.has_full_tile(0, 1), "D L0 tile 1"); + expect( + !partial_dir_exists(store, 0, 1), "D superseded L0 partial removed"); + expect(!store.has_tile(TileRef{0, 1, 44}), "D old L0 partial gone"); + expect(store.has_tile(TileRef{1, 0, 2}), "D L1 partial width 2"); + expect(!store.has_tile(TileRef{1, 0, 1}), "D old L1 partial gone"); + + std::cout << "D (supersession): OK" << '\n'; + } + + std::cout << "tiles_writer: OK" << '\n'; + + std::error_code ec; + fs::remove_all(base, ec); + } + catch (std::exception& ex) + { + std::cout << "Error: " << ex.what() << '\n'; + std::error_code ec; + fs::remove_all(base, ec); + return 1; + } + catch (...) + { + std::cout << "Error" << '\n'; + std::error_code ec; + fs::remove_all(base, ec); + return 1; + } + + return 0; +} From 1bff8dae1d7786fb0d88d4805efa3310fca6346a Mon Sep 17 00:00:00 2001 From: Amaury Chamayou Date: Sat, 13 Jun 2026 15:34:36 +0000 Subject: [PATCH 04/30] feat(tiles): add tile-backed proof engine (Phase 3) Add HashSourceT, TileHashSourceT and ProofEngineT to merklecpp_tiles.h, all templated on the tree's HASH_FUNCTION so results match the existing library exactly: - HashSourceT: abstract resolver of complete (balanced) subtree roots. - TileHashSourceT: resolves subtree roots from tile files for a given available size; returns false (rather than throwing) on a missing tile so a builder can fall back to another source. - ProofEngineT: mth_range (perfect-subtree resolution with split fallback), root(size), inclusion_proof (byte-identical to TreeT::path/past_path, returns PathT), consistency_proof and verify_consistency (RFC 6962, new to merklecpp). - HashSource/TileHashSource/ProofEngine aliases (+384/512 guarded). test/tiles_proofs.cpp uses the existing library as the oracle: tile root == Tree::root/past_root; inclusion_proof == Tree::path/past_path via PathT::operator==; consistency proofs verify and reconcile past_roots, with tamper/wrong-root rejection; exhaustive consistency for n<=16 and a size-70000 tree (full L1 + L2 partial). All 14 tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- merklecpp_tiles.h | 387 ++++++++++++++++++++++++++++++++++++++++++ test/CMakeLists.txt | 1 + test/tiles_proofs.cpp | 221 ++++++++++++++++++++++++ 3 files changed, 609 insertions(+) create mode 100644 test/tiles_proofs.cpp diff --git a/merklecpp_tiles.h b/merklecpp_tiles.h index 0fb7b85..daa6d46 100644 --- a/merklecpp_tiles.h +++ b/merklecpp_tiles.h @@ -622,12 +622,389 @@ namespace merkle } }; + /// @brief Abstract source of Merkle subtree roots for proof generation. + /// @note Implementations resolve the root of a complete (balanced) subtree + /// from tiles, from an in-memory tree, or from a combination of the two. + template < + size_t HASH_SIZE, + void HASH_FUNCTION( + const HashT&, const HashT&, HashT&)> + struct HashSourceT + { + /// @brief The type of hashes resolved. + using Hash = HashT; + + virtual ~HashSourceT() = default; + + /// @brief Resolves MTH(D[index << level : (index + 1) << level]). + /// @param level The subtree height (the subtree spans 2**level leaves) + /// @param index The subtree index at that height + /// @param out Set to the subtree root on success + /// @return Whether the complete, balanced subtree could be resolved + virtual bool subtree_root( + uint8_t level, uint64_t index, Hash& out) const = 0; + + /// @brief Resolves the level-0 leaf hash at @p index. + virtual bool leaf(uint64_t index, Hash& out) const + { + return subtree_root(0, index, out); + } + }; + + /// @brief Resolves subtree roots from tlog-tiles tile files. + /// @note @p available_size is the largest tree size whose tiles are durably + /// written; any complete subtree within it is resolvable. A missing tile + /// (e.g. a frontier partial that was not written) yields false so that a + /// proof builder can fall back to another source. + template < + size_t HASH_SIZE, + void HASH_FUNCTION( + const HashT&, const HashT&, HashT&)> + class TileHashSourceT : public HashSourceT + { + public: + using Hash = HashT; + using Store = TileStoreT; + + /// @brief Constructs a source over @p store for trees up to + /// @p available_size leaves. + TileHashSourceT(const Store& store, uint64_t available_size) : + store(store), available_size(available_size) + {} + + bool subtree_root(uint8_t level, uint64_t index, Hash& out) const override + { + const uint8_t L = level / TILE_HEIGHT; + const uint8_t r = level % TILE_HEIGHT; + const uint64_t span = (uint64_t)1 << r; // entries spanned in level L + const uint64_t first = index << r; // first level-L entry index + + const uint64_t entries = entries_at_level(available_size, L); + if (first + span > entries) + { + return false; // extends into the incomplete frontier + } + + const uint64_t n = first / TILE_WIDTH; + const uint64_t off = first % TILE_WIDTH; + + // Choose the full or current partial tile holding these entries. + TileRef ref{L, n, 0}; + if ((n + 1) * TILE_WIDTH > entries) + { + ref.width = (uint16_t)(entries - n * TILE_WIDTH); + } + if (!store.has_tile(ref)) + { + return false; + } + + const std::vector tile = store.read_tile(ref); + if (span == 1) + { + out = tile.at(off); + return true; + } + const std::vector sub( + tile.begin() + off, tile.begin() + off + span); + out = perfect_root(sub); + return true; + } + + protected: + const Store& store; + uint64_t available_size; + + static uint64_t entries_at_level(uint64_t size, uint8_t level) + { + const unsigned shift = 8u * (unsigned)level; + return shift >= 64 ? 0 : (size >> shift); + } + }; + + /// @brief Builds and verifies inclusion and consistency proofs. + /// @note Proofs are assembled from a HashSource using the tree's + /// HASH_FUNCTION, so an inclusion proof is byte-identical to the one + /// produced by merkle::TreeT::path()/past_path() and verifies with + /// PathT::verify(). + template < + size_t HASH_SIZE, + void HASH_FUNCTION( + const HashT&, const HashT&, HashT&)> + class ProofEngineT + { + public: + using Hash = HashT; + using Path = PathT; + using Source = HashSourceT; + + explicit ProofEngineT(const Source& source) : source(source) {} + + /// @brief The Merkle root of a tree of @p size leaves. + Hash root(uint64_t size) const + { + if (size == 0) + { + throw std::runtime_error("empty tree has no root"); + } + Hash out; + if (!mth_range(0, size, out)) + { + throw std::runtime_error("unresolved subtree while computing root"); + } + return out; + } + + /// @brief Inclusion proof for leaf @p index in a tree of @p size leaves. + /// @note Equivalent to TreeT::path(index) when size == num_leaves(), and + /// to TreeT::past_path(index, size - 1) otherwise. + std::shared_ptr inclusion_proof(uint64_t index, uint64_t size) const + { + if (index >= size) + { + throw std::runtime_error("leaf index out of bounds"); + } + + std::list elements; // leaf -> root order + uint64_t lo = 0; + uint64_t hi = size; + while (hi - lo > 1) + { + const uint64_t k = largest_pow2_lt(hi - lo); + typename Path::Element e; + if (index - lo < k) + { + if (!mth_range(lo + k, hi, e.hash)) + { + throw std::runtime_error("unresolved subtree in inclusion proof"); + } + e.direction = Path::PATH_RIGHT; + hi = lo + k; + } + else + { + if (!mth_range(lo, lo + k, e.hash)) + { + throw std::runtime_error("unresolved subtree in inclusion proof"); + } + e.direction = Path::PATH_LEFT; + lo = lo + k; + } + elements.push_front(std::move(e)); + } + + Hash leaf; + if (!source.leaf(index, leaf)) + { + throw std::runtime_error("unresolved leaf in inclusion proof"); + } + return std::make_shared( + leaf, index, std::move(elements), size - 1); + } + + /// @brief Consistency proof that a tree of @p m leaves is a prefix of a + /// tree of @p n leaves (RFC 6962). + std::vector consistency_proof(uint64_t m, uint64_t n) const + { + if (m == 0 || m > n) + { + throw std::runtime_error("invalid consistency proof sizes"); + } + std::vector proof; + if (m == n) + { + return proof; + } + subproof(m, 0, n, true, proof); + return proof; + } + + /// @brief Verifies an RFC 6962 consistency proof reconciling the roots of + /// trees of @p m and @p n leaves. + static bool verify_consistency( + uint64_t m, + uint64_t n, + const Hash& first_hash, + const Hash& second_hash, + std::vector proof) + { + if (m > n) + { + return false; + } + if (m == n) + { + return proof.empty() && first_hash == second_hash; + } + if (m == 0) + { + return proof.empty(); + } + + if (is_pow2(m)) + { + proof.insert(proof.begin(), first_hash); + } + if (proof.empty()) + { + return false; + } + + uint64_t fn = m - 1; + uint64_t sn = n - 1; + while ((fn & 1) != 0) + { + fn >>= 1; + sn >>= 1; + } + + Hash fr = proof[0]; + Hash sr = proof[0]; + for (size_t i = 1; i < proof.size(); i++) + { + if (sn == 0) + { + return false; + } + const Hash& c = proof[i]; + if ((fn & 1) != 0 || fn == sn) + { + HASH_FUNCTION(c, fr, fr); + HASH_FUNCTION(c, sr, sr); + if ((fn & 1) == 0) + { + while ((fn & 1) == 0 && fn != 0) + { + fn >>= 1; + sn >>= 1; + } + } + } + else + { + HASH_FUNCTION(sr, c, sr); + } + fn >>= 1; + sn >>= 1; + } + + return fr == first_hash && sr == second_hash && sn == 0; + } + + protected: + const Source& source; + + static bool is_pow2(uint64_t n) + { + return n != 0 && (n & (n - 1)) == 0; + } + + static uint64_t largest_pow2_lt(uint64_t n) + { + uint64_t k = 1; + while ((k << 1) < n) + { + k <<= 1; + } + return k; + } + + static uint8_t log2_exact(uint64_t n) + { + uint8_t r = 0; + while (n > 1) + { + n >>= 1; + r++; + } + return r; + } + + /// @brief MTH(D[a:b]) via the source; falls back to splitting when a + /// perfect subtree cannot be resolved directly. + bool mth_range(uint64_t a, uint64_t b, Hash& out) const + { + const uint64_t w = b - a; + if (w == 1) + { + return source.leaf(a, out); + } + if (is_pow2(w) && (a % w == 0)) + { + if (source.subtree_root(log2_exact(w), a / w, out)) + { + return true; + } + } + const uint64_t k = largest_pow2_lt(w); + Hash left; + Hash right; + if (!mth_range(a, a + k, left) || !mth_range(a + k, b, right)) + { + return false; + } + HASH_FUNCTION(left, right, out); + return true; + } + + void subproof( + uint64_t m, + uint64_t lo, + uint64_t hi, + bool complete, + std::vector& proof) const + { + if (m == hi - lo) + { + if (!complete) + { + Hash h; + if (!mth_range(lo, hi, h)) + { + throw std::runtime_error( + "unresolved subtree in consistency proof"); + } + proof.push_back(h); + } + return; + } + const uint64_t k = largest_pow2_lt(hi - lo); + Hash h; + if (m <= k) + { + subproof(m, lo, lo + k, complete, proof); + if (!mth_range(lo + k, hi, h)) + { + throw std::runtime_error("unresolved subtree in consistency proof"); + } + } + else + { + subproof(m - k, lo + k, hi, false, proof); + if (!mth_range(lo, lo + k, h)) + { + throw std::runtime_error("unresolved subtree in consistency proof"); + } + } + proof.push_back(h); + } + }; + /// @brief Default tile store (SHA256, default hash function). using TileStore = TileStoreT<32, sha256_compress>; /// @brief Default tile writer (SHA256, default hash function). using TileWriter = TileWriterT<32, sha256_compress>; + /// @brief Default abstract hash source (SHA256, default hash function). + using HashSource = HashSourceT<32, sha256_compress>; + + /// @brief Default tile-backed hash source (SHA256, default hash function). + using TileHashSource = TileHashSourceT<32, sha256_compress>; + + /// @brief Default proof engine (SHA256, default hash function). + using ProofEngine = ProofEngineT<32, sha256_compress>; + #ifdef HAVE_OPENSSL /// @brief SHA384 tile store. using TileStore384 = TileStoreT<48, sha384_openssl>; @@ -640,6 +1017,16 @@ namespace merkle /// @brief SHA512 tile writer. using TileWriter512 = TileWriterT<64, sha512_openssl>; + + /// @brief SHA384 hash source, tile-backed source and proof engine. + using HashSource384 = HashSourceT<48, sha384_openssl>; + using TileHashSource384 = TileHashSourceT<48, sha384_openssl>; + using ProofEngine384 = ProofEngineT<48, sha384_openssl>; + + /// @brief SHA512 hash source, tile-backed source and proof engine. + using HashSource512 = HashSourceT<64, sha512_openssl>; + using TileHashSource512 = TileHashSourceT<64, sha512_openssl>; + using ProofEngine512 = ProofEngineT<64, sha512_openssl>; #endif } } diff --git a/test/CMakeLists.txt b/test/CMakeLists.txt index c14c58b..0e976d3 100644 --- a/test/CMakeLists.txt +++ b/test/CMakeLists.txt @@ -31,6 +31,7 @@ add_merklecpp_test(serialise_to_file serialise_to_file.cpp) add_merklecpp_test(coverage coverage.cpp) add_merklecpp_test(tiles_store tiles_store.cpp) add_merklecpp_test(tiles_writer tiles_writer.cpp) +add_merklecpp_test(tiles_proofs tiles_proofs.cpp) if(TARGET evercrypt.host) add_merklecpp_test(compare_evercrypt compare_evercrypt.cpp) diff --git a/test/tiles_proofs.cpp b/test/tiles_proofs.cpp new file mode 100644 index 0000000..5be4294 --- /dev/null +++ b/test/tiles_proofs.cpp @@ -0,0 +1,221 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. + +#include "util.h" + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace fs = std::filesystem; +using merkle::Hash; +using merkle::tiles::ProofEngine; +using merkle::tiles::TileHashSource; +using merkle::tiles::TileStore; +using merkle::tiles::TileWriter; + +static void expect(bool cond, const std::string& what) +{ + if (!cond) + { + throw std::runtime_error("check failed: " + what); + } +} + +// Exercises tile-derived proofs for a tree of `n` leaves against the existing +// library (which acts as the oracle: proofs must be byte-identical). +static void check_size( + const fs::path& dir, uint64_t n, const std::vector& hashes) +{ + const std::string at = " @n=" + std::to_string(n); + + TileStore store(dir); + TileWriter writer(store); + const auto leaf_at = [&](uint64_t i) -> const Hash& { return hashes[i]; }; + writer.write_up_to(n, leaf_at); + + merkle::Tree tree; + for (uint64_t i = 0; i < n; i++) + { + tree.insert(hashes[i]); + } + const Hash root = tree.root(); + + TileHashSource source(store, n); + ProofEngine engine(source); + + // Root recomputed from tiles equals the library root. + expect(engine.root(n) == root, "root" + at); + + // Indices to probe: all of them for small trees, else a spread. + std::vector indices; + if (n <= 16) + { + for (uint64_t i = 0; i < n; i++) + { + indices.push_back(i); + } + } + else + { + for (uint64_t i : {(uint64_t)0, (uint64_t)1, n / 3, n / 2, n - 2, n - 1}) + { + indices.push_back(i); + } + } + + // Inclusion proofs are identical to TreeT::path and verify. + for (uint64_t i : indices) + { + if (i >= n) + { + continue; + } + const auto p = engine.inclusion_proof(i, n); + expect(*p == *tree.path(i), "inclusion==path i=" + std::to_string(i) + at); + expect(p->verify(root), "inclusion verify i=" + std::to_string(i) + at); + } + + // Consistency pairs: exhaustive for small trees, else a fixed spread. + std::vector> pairs; + if (n <= 16) + { + for (uint64_t m = 1; m < n; m++) + { + for (uint64_t k = m + 1; k <= n; k++) + { + pairs.emplace_back(m, k); + } + } + } + else + { + for (auto pr : + {std::make_pair(1, (uint64_t)n), + std::make_pair(n / 2, n), + std::make_pair(n - 1, n), + std::make_pair((uint64_t)1, (uint64_t)2)}) + { + pairs.push_back(pr); + } + // Tile-boundary crossings. + if (n > 256) + { + pairs.emplace_back(256, n); + pairs.emplace_back(257, n); + } + } + + for (const auto& pr : pairs) + { + const uint64_t m = pr.first; + const uint64_t k = pr.second; + if (!(m >= 1 && m < k && k <= n)) + { + continue; + } + + const Hash rm = engine.root(m); + const Hash rk = engine.root(k); + expect(rm == *tree.past_root(m - 1), "past_root m" + at); + expect(rk == *tree.past_root(k - 1), "past_root k" + at); + + // Past inclusion proof matches TreeT::past_path. + const uint64_t i = m / 2; + const auto pp = engine.inclusion_proof(i, m); + expect( + *pp == *tree.past_path(i, m - 1), + "inclusion(m)==past_path i=" + std::to_string(i) + at); + expect(pp->verify(rm), "inclusion(m) verify" + at); + + // Consistency proof reconciles the two roots. + const auto cp = engine.consistency_proof(m, k); + expect( + ProofEngine::verify_consistency(m, k, rm, rk, cp), + "consistency " + std::to_string(m) + "->" + std::to_string(k) + at); + + // Tampering with a proof element or a root is rejected. + auto bad = cp; + bad[0].bytes[0] ^= 0xFFu; + expect( + !ProofEngine::verify_consistency(m, k, rm, rk, bad), + "consistency tamper rejected" + at); + + Hash wrong = rk; + wrong.bytes[0] ^= 0xFFu; + expect( + !ProofEngine::verify_consistency(m, k, rm, wrong, cp), + "consistency wrong root rejected" + at); + } + + std::error_code ec; + fs::remove_all(dir, ec); +} + +int main() +{ + const auto seed = std::time(nullptr); + std::srand((unsigned)seed); + std::cout << "seed=" << seed << '\n'; + + const fs::path base = fs::temp_directory_path() / + ("merklecpp_tiles_proofs_" + std::to_string((unsigned long long)seed) + + "_" + std::to_string(std::rand())); + + try + { + const auto hashes = make_hashes(70000); + + for (uint64_t n : + {(uint64_t)1, + (uint64_t)2, + (uint64_t)3, + (uint64_t)4, + (uint64_t)5, + (uint64_t)7, + (uint64_t)8, + (uint64_t)13, + (uint64_t)16, + (uint64_t)255, + (uint64_t)256, + (uint64_t)257, + (uint64_t)1000}) + { + check_size(base / ("n" + std::to_string(n)), n, hashes); + } + std::cout << "small/medium sizes: OK" << '\n'; + + // Large tree: exercises full L1 tiles and an L2 partial. + check_size(base / "big", 70000, hashes); + std::cout << "size 70000: OK" << '\n'; + + std::cout << "tiles_proofs: OK" << '\n'; + + std::error_code ec; + fs::remove_all(base, ec); + } + catch (std::exception& ex) + { + std::cout << "Error: " << ex.what() << '\n'; + std::error_code ec; + fs::remove_all(base, ec); + return 1; + } + catch (...) + { + std::cout << "Error" << '\n'; + std::error_code ec; + fs::remove_all(base, ec); + return 1; + } + + return 0; +} From 237f3dd82ea773ff198708c628ab15ec091a3ff9 Mon Sep 17 00:00:00 2001 From: Amaury Chamayou Date: Sat, 13 Jun 2026 15:46:13 +0000 Subject: [PATCH 05/30] feat(tiles): combine tiles with in-memory tree; TiledTree (Phase 4) Add the proof combination and a tiled-tree wrapper so proofs remain available after compaction (flushing). merklecpp.h (additive, read-only, no hashing change): - TreeT::subtree_root(level, index, out): returns the hash of a complete subtree that is fully resident in memory, or false if any of its leaves were flushed / it is not a full subtree. Navigation mirrors walk_to; it only returns an already-defined node hash (computed on demand exactly as root()/path() do). merklecpp_tiles.h: - MemoryHashSourceT: resolves resident subtree roots via the accessor. - CombinedHashSourceT: tries a primary source (memory: resident frontier) then a secondary (tiles: flushed past). - TiledTreeT: append / checkpoint (write tiles + checkpoint, then flush_to the full-tile coverage minus a retention margin) / inclusion_proof / consistency_proof over the combined source. - Memory/Combined/TiledTree aliases (+384/512 guarded by HAVE_OPENSSL). test/tiles_tree.cpp: memory-source proofs match the library (validates subtree_root); a TiledTree that checkpoints and flushes still proves flushed, overlap and frontier indices against a non-flushed reference root (where the resident tree alone throws), proof_size may exceed the checkpoint size, consistency proofs hold across the flush boundary, and a second checkpoint flushes further without ever reading a flushed leaf. All 15 tests pass; OpenSSL-guarded aliases type-check. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- merklecpp.h | 66 +++++++++++ merklecpp_tiles.h | 211 +++++++++++++++++++++++++++++++++++ test/CMakeLists.txt | 1 + test/tiles_tree.cpp | 262 ++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 540 insertions(+) create mode 100644 test/tiles_tree.cpp diff --git a/merklecpp.h b/merklecpp.h index 1170b1c..a4cee98 100644 --- a/merklecpp.h +++ b/merklecpp.h @@ -1316,6 +1316,72 @@ namespace merkle leaf_node(index)->hash, index, std::move(path), as_of); } + /// @brief Extracts the root hash of a complete subtree resident in memory + /// @param level The height of the subtree (it spans 2**level leaves) + /// @param index The index of the subtree at that height + /// @param out Set to the subtree root hash on success + /// @return Whether the subtree is a complete (balanced) subtree fully + /// resident in memory + /// @note This is read-only and does not change the hashing of the tree: it + /// returns an existing node hash (computing it on demand exactly as root() + /// and path() do). It returns false if any leaf of the subtree has been + /// flushed, if the subtree extends past the last leaf, or if the node at + /// that position is not a full subtree. The subtree spans leaf indices + /// [index << level, (index + 1) << level). + bool subtree_root(uint8_t level, size_t index, Hash& out) + { + const size_t lo = index << level; + const size_t count = (size_t)1 << level; + + if (num_leaves() == 0 || lo < min_index() || lo + count > num_leaves()) + { + return false; + } + + if (level == 0) + { + out = leaf(lo); + return true; + } + + compute_root(); + + const uint8_t target_height = level + 1; + if (!_root || _root->height < target_height) + { + return false; + } + + Node* cur = _root; + size_t it = lo << (sizeof(lo) * 8 - _root->height + 1); + for (uint8_t height = _root->height; height > target_height;) + { + const bool go_right = ((it >> (8 * sizeof(it) - 1)) & 0x01) != 0U; + if (cur->height == height) + { + Node* next = go_right ? cur->right : cur->left; + if (!next) + { + return false; // conflated/flushed: not resident + } + cur = next; + } + it <<= 1; + height--; + } + + if (cur->height != target_height || !cur->is_full()) + { + return false; + } + if (cur->dirty) + { + hash(cur); + } + out = cur->hash; + return true; + } + /// @brief Serialises the tree /// @param bytes The vector of bytes to serialise to void serialise(std::vector& bytes) diff --git a/merklecpp_tiles.h b/merklecpp_tiles.h index daa6d46..a50b205 100644 --- a/merklecpp_tiles.h +++ b/merklecpp_tiles.h @@ -990,6 +990,198 @@ namespace merkle } }; + /// @brief Resolves subtree roots from an in-memory merkle::TreeT. + /// @note Resolves only complete subtrees that are fully resident (not + /// flushed), returning false otherwise so that a builder can fall back to + /// another source. Performs no hashing changes (see TreeT::subtree_root). + template < + size_t HASH_SIZE, + void HASH_FUNCTION( + const HashT&, const HashT&, HashT&)> + class MemoryHashSourceT : public HashSourceT + { + public: + using Hash = HashT; + using Tree = TreeT; + + explicit MemoryHashSourceT(Tree& tree) : tree(tree) {} + + bool subtree_root(uint8_t level, uint64_t index, Hash& out) const override + { + return tree.subtree_root(level, (size_t)index, out); + } + + protected: + Tree& tree; + }; + + /// @brief Resolves subtree roots from a primary source, falling back to a + /// secondary source. + /// @note Used to combine an in-memory tree (primary: no I/O, serves the + /// resident frontier) with tile files (secondary: serve the flushed past). + template < + size_t HASH_SIZE, + void HASH_FUNCTION( + const HashT&, const HashT&, HashT&)> + class CombinedHashSourceT : public HashSourceT + { + public: + using Hash = HashT; + using Source = HashSourceT; + + CombinedHashSourceT(const Source& primary, const Source& secondary) : + primary(primary), secondary(secondary) + {} + + bool subtree_root(uint8_t level, uint64_t index, Hash& out) const override + { + return primary.subtree_root(level, index, out) || + secondary.subtree_root(level, index, out); + } + + protected: + const Source& primary; + const Source& secondary; + }; + + /// @brief A merkle tree backed by tlog-tiles storage. + /// @note Appends grow an in-memory tree; checkpoint() durably writes tiles + /// and a checkpoint, then compacts memory by flushing the leaves now + /// covered by full tiles. Proofs are served from the combination of the + /// resident tree and the tiles, so they remain available for flushed + /// (compacted) leaves. + template < + size_t HASH_SIZE, + void HASH_FUNCTION( + const HashT&, const HashT&, HashT&)> + class TiledTreeT + { + public: + using Hash = HashT; + using Tree = TreeT; + using Path = PathT; + using Store = TileStoreT; + using Writer = TileWriterT; + using Stats = typename Writer::Stats; + + /// @brief Configuration for a tiled tree. + struct Config + { + /// @brief Root directory for tiles and the checkpoint. + std::filesystem::path prefix; + + /// @brief Number of most-recent leaves to keep resident across a + /// checkpoint (i.e. not flushed). + uint64_t retention_margin = 0; + + /// @brief Tile writer options. + typename Writer::Options writer = {}; + }; + + explicit TiledTreeT(Config config) : + config(std::move(config)), + store(this->config.prefix), + writer(store, this->config.writer) + {} + + /// @brief Appends a leaf hash. + void append(const Hash& leaf_hash) + { + tree.insert(leaf_hash); + } + + /// @brief The number of leaves (including flushed ones). + [[nodiscard]] uint64_t size() const + { + return tree.num_leaves(); + } + + /// @brief The current Merkle root. + Hash root() + { + return tree.root(); + } + + /// @brief The tree size of the last checkpoint (the durable tile size). + [[nodiscard]] uint64_t checkpoint_size() const + { + return tiles_size; + } + + /// @brief Access to the underlying tree. + Tree& tree_ref() + { + return tree; + } + + /// @brief Access to the underlying tile store. + Store& store_ref() + { + return store; + } + + /// @brief Writes tiles and a checkpoint, then compacts memory. + /// @return Counts of the tiles written/removed by this checkpoint + Stats checkpoint() + { + Stats stats; + const uint64_t n = tree.num_leaves(); + if (n == 0) + { + return stats; + } + + stats = writer.write_up_to(n, [this](uint64_t i) -> const Hash& { + return tree.leaf((size_t)i); + }); + store.write_checkpoint(n, tree.root()); + tiles_size = n; + + // Compaction: flush to the durable full-tile coverage, keeping the + // retention margin resident. The flush target is a multiple of + // TILE_WIDTH so the coverage invariant (min_index <= covered) holds. + const uint64_t covered = (n / TILE_WIDTH) * TILE_WIDTH; + uint64_t target = covered > config.retention_margin ? + covered - config.retention_margin : + 0; + target = (target / TILE_WIDTH) * TILE_WIDTH; + if (target > tree.min_index()) + { + tree.flush_to((size_t)target); + } + return stats; + } + + /// @brief Inclusion proof for @p index in a tree of @p proof_size leaves. + /// @note Served from tiles (flushed past) combined with the resident tree + /// (recent frontier); @p proof_size may exceed checkpoint_size(). + std::shared_ptr inclusion_proof(uint64_t index, uint64_t proof_size) + { + MemoryHashSourceT mem(tree); + TileHashSourceT tile_src(store, tiles_size); + CombinedHashSourceT combined(mem, tile_src); + ProofEngineT engine(combined); + return engine.inclusion_proof(index, proof_size); + } + + /// @brief Consistency proof between tree sizes @p m and @p n. + std::vector consistency_proof(uint64_t m, uint64_t n) + { + MemoryHashSourceT mem(tree); + TileHashSourceT tile_src(store, tiles_size); + CombinedHashSourceT combined(mem, tile_src); + ProofEngineT engine(combined); + return engine.consistency_proof(m, n); + } + + protected: + Config config; + Store store; + Writer writer; + Tree tree; + uint64_t tiles_size = 0; + }; + /// @brief Default tile store (SHA256, default hash function). using TileStore = TileStoreT<32, sha256_compress>; @@ -1005,6 +1197,15 @@ namespace merkle /// @brief Default proof engine (SHA256, default hash function). using ProofEngine = ProofEngineT<32, sha256_compress>; + /// @brief Default in-memory hash source (SHA256, default hash function). + using MemoryHashSource = MemoryHashSourceT<32, sha256_compress>; + + /// @brief Default combined hash source (SHA256, default hash function). + using CombinedHashSource = CombinedHashSourceT<32, sha256_compress>; + + /// @brief Default tiled tree (SHA256, default hash function). + using TiledTree = TiledTreeT<32, sha256_compress>; + #ifdef HAVE_OPENSSL /// @brief SHA384 tile store. using TileStore384 = TileStoreT<48, sha384_openssl>; @@ -1027,6 +1228,16 @@ namespace merkle using HashSource512 = HashSourceT<64, sha512_openssl>; using TileHashSource512 = TileHashSourceT<64, sha512_openssl>; using ProofEngine512 = ProofEngineT<64, sha512_openssl>; + + /// @brief SHA384 memory/combined sources and tiled tree. + using MemoryHashSource384 = MemoryHashSourceT<48, sha384_openssl>; + using CombinedHashSource384 = CombinedHashSourceT<48, sha384_openssl>; + using TiledTree384 = TiledTreeT<48, sha384_openssl>; + + /// @brief SHA512 memory/combined sources and tiled tree. + using MemoryHashSource512 = MemoryHashSourceT<64, sha512_openssl>; + using CombinedHashSource512 = CombinedHashSourceT<64, sha512_openssl>; + using TiledTree512 = TiledTreeT<64, sha512_openssl>; #endif } } diff --git a/test/CMakeLists.txt b/test/CMakeLists.txt index 0e976d3..cb1fd36 100644 --- a/test/CMakeLists.txt +++ b/test/CMakeLists.txt @@ -32,6 +32,7 @@ add_merklecpp_test(coverage coverage.cpp) add_merklecpp_test(tiles_store tiles_store.cpp) add_merklecpp_test(tiles_writer tiles_writer.cpp) add_merklecpp_test(tiles_proofs tiles_proofs.cpp) +add_merklecpp_test(tiles_tree tiles_tree.cpp) if(TARGET evercrypt.host) add_merklecpp_test(compare_evercrypt compare_evercrypt.cpp) diff --git a/test/tiles_tree.cpp b/test/tiles_tree.cpp new file mode 100644 index 0000000..c4b7ef0 --- /dev/null +++ b/test/tiles_tree.cpp @@ -0,0 +1,262 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. + +#include "util.h" + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace fs = std::filesystem; +using merkle::Hash; +using merkle::tiles::MemoryHashSource; +using merkle::tiles::ProofEngine; +using merkle::tiles::TiledTree; + +static void expect(bool cond, const std::string& what) +{ + if (!cond) + { + throw std::runtime_error("check failed: " + what); + } +} + +// Validates the TreeT::subtree_root accessor via memory-only proofs: they must +// match the library exactly. +static void check_memory_source(uint64_t n, const std::vector& hashes) +{ + const std::string at = " @n=" + std::to_string(n); + + merkle::Tree tree; + for (uint64_t i = 0; i < n; i++) + { + tree.insert(hashes[i]); + } + const Hash root = tree.root(); + + MemoryHashSource source(tree); + ProofEngine engine(source); + + expect(engine.root(n) == root, "mem root" + at); + + std::vector indices; + if (n <= 16) + { + for (uint64_t i = 0; i < n; i++) + { + indices.push_back(i); + } + } + else + { + for (uint64_t i : {(uint64_t)0, (uint64_t)1, n / 2, n - 1}) + { + indices.push_back(i); + } + } + + for (uint64_t i : indices) + { + const auto p = engine.inclusion_proof(i, n); + expect( + *p == *tree.path(i), "mem inclusion==path i=" + std::to_string(i) + at); + expect(p->verify(root), "mem inclusion verify i=" + std::to_string(i) + at); + } + + std::vector> pairs; + if (n <= 16) + { + for (uint64_t m = 1; m < n; m++) + { + for (uint64_t k = m + 1; k <= n; k++) + { + pairs.emplace_back(m, k); + } + } + } + else + { + pairs = {{1, n}, {n / 2, n}, {n - 1, n}}; + } + + for (const auto& pr : pairs) + { + const uint64_t m = pr.first; + const uint64_t k = pr.second; + const Hash rm = engine.root(m); + const Hash rk = engine.root(k); + expect(rm == *tree.past_root(m - 1), "mem past_root m" + at); + + const auto pp = engine.inclusion_proof(m / 2, m); + expect( + *pp == *tree.past_path(m / 2, m - 1), "mem inclusion(m)==past_path" + at); + + const auto cp = engine.consistency_proof(m, k); + expect( + ProofEngine::verify_consistency(m, k, rm, rk, cp), + "mem consistency" + at); + } +} + +int main() +{ + const auto seed = std::time(nullptr); + std::srand((unsigned)seed); + std::cout << "seed=" << seed << '\n'; + + const fs::path base = fs::temp_directory_path() / + ("merklecpp_tiles_tree_" + std::to_string((unsigned long long)seed) + "_" + + std::to_string(std::rand())); + + try + { + const auto hashes = make_hashes(1500); + + // ---- Part 1: memory-source proofs (exercises TreeT::subtree_root). + for (uint64_t n : + {(uint64_t)1, + (uint64_t)2, + (uint64_t)3, + (uint64_t)5, + (uint64_t)8, + (uint64_t)13, + (uint64_t)16, + (uint64_t)256, + (uint64_t)257, + (uint64_t)1000}) + { + check_memory_source(n, hashes); + } + std::cout << "memory source: OK" << '\n'; + + // ---- Part 2: TiledTree checkpoint + flush, proofs over tiles + memory. + const uint64_t n1 = 1000; // first checkpoint size + const uint64_t N = 1500; // final size + + // Reference: a plain (never flushed) tree with the same leaves. + merkle::Tree ref; + for (uint64_t i = 0; i < N; i++) + { + ref.insert(hashes[i]); + } + const Hash ref_root = ref.root(); + + TiledTree::Config cfg; + cfg.prefix = base / "tt"; + cfg.retention_margin = 0; + TiledTree tt(cfg); + + for (uint64_t i = 0; i < n1; i++) + { + tt.append(hashes[i]); + } + tt.checkpoint(); // tiles_size = 1000; flush to covered = 768 + expect(tt.checkpoint_size() == n1, "checkpoint size"); + expect(tt.tree_ref().min_index() == 768, "flushed to 768"); + + for (uint64_t i = n1; i < N; i++) + { + tt.append(hashes[i]); + } + expect(tt.size() == N, "size after appends"); + expect(tt.root() == ref_root, "tiled root == reference root"); + + // Indices that are: flushed (tiles only), in the checkpointed-but-resident + // overlap, and on the un-checkpointed resident frontier. + for (uint64_t i : + {(uint64_t)0, + (uint64_t)767, + (uint64_t)800, + (uint64_t)999, + (uint64_t)1000, + (uint64_t)1499}) + { + const auto p = tt.inclusion_proof(i, N); + expect( + *p == *ref.path(i), "combined inclusion==ref i=" + std::to_string(i)); + expect( + p->verify(ref_root), + "combined inclusion verify i=" + std::to_string(i)); + } + + // The resident tree alone cannot prove a flushed index. + bool threw = false; + try + { + (void)tt.tree_ref().path(0); + } + catch (const std::exception&) + { + threw = true; + } + expect(threw, "memory-only path throws for flushed index"); + + // Consistency across the flush boundary: checkpoint size -> current size, + // and a flushed-era size -> current size. + { + const auto cp = tt.consistency_proof(n1, N); + expect( + ProofEngine::verify_consistency( + n1, N, *ref.past_root(n1 - 1), ref_root, cp), + "consistency n1->N across flush"); + + const uint64_t m = 500; // flushed era + const auto cp2 = tt.consistency_proof(m, N); + expect( + ProofEngine::verify_consistency( + m, N, *ref.past_root(m - 1), ref_root, cp2), + "consistency m->N across flush"); + } + + // A second checkpoint flushes further; proofs for now-flushed indices still + // work (this also confirms the writer never reads a flushed leaf). + tt.checkpoint(); // tiles_size = 1500; flush to covered = 1280 + expect(tt.checkpoint_size() == N, "second checkpoint size"); + expect(tt.tree_ref().min_index() == 1280, "flushed to 1280"); + + for (uint64_t i : + {(uint64_t)0, + (uint64_t)1000, + (uint64_t)1279, + (uint64_t)1280, + (uint64_t)1499}) + { + const auto p = tt.inclusion_proof(i, N); + expect( + *p == *ref.path(i), "post-2nd inclusion==ref i=" + std::to_string(i)); + expect( + p->verify(ref_root), + "post-2nd inclusion verify i=" + std::to_string(i)); + } + + std::cout << "tiled tree (flush + combination): OK" << '\n'; + + std::cout << "tiles_tree: OK" << '\n'; + + std::error_code ec; + fs::remove_all(base, ec); + } + catch (std::exception& ex) + { + std::cout << "Error: " << ex.what() << '\n'; + std::error_code ec; + fs::remove_all(base, ec); + return 1; + } + catch (...) + { + std::cout << "Error" << '\n'; + std::error_code ec; + fs::remove_all(base, ec); + return 1; + } + + return 0; +} From 4f4aea62a50cd6ecf0d5cdcde23294624862d36c Mon Sep 17 00:00:00 2001 From: Amaury Chamayou Date: Sat, 13 Jun 2026 15:54:11 +0000 Subject: [PATCH 06/30] feat(tiles): entry bundles + docs (Phase 5) merklecpp_tiles.h: - TileStoreT: entry-bundle format (big-endian uint16 length-prefixed entries) via encode_entries/decode_entries plus write_entry_bundle/read_entry_bundle/has_entry_bundle and the tile/entries/[.p/] path layout. - EntryBundleWriterT: progressive, application-owned writer of raw entry bundles (incremental immutable full bundles + a growable/removable partial), mirroring TileWriterT's level-0 logic. - EntryBundleWriter[/384/512] aliases. Entry bundles are application-owned: merklecpp stores leaf hashes, while the raw entries and the leaf-hash derivation linking them to level-0 tiles are the application's responsibility. Docs: - README: a 'Tiled storage (tlog-tiles)' section with a TiledTree usage snippet and a link to doc/design/tlog-tiles.md. - Doxyfile: add merklecpp_tiles.h to INPUT. test/tiles_entries.cpp: encode/decode round-trip, the size-256 and size-70000 bundle sets/widths, content round-trip, tile<->bundle correspondence under an identity leaf-hash, incremental immutability and partial supersession. All 16 tests pass; OpenSSL aliases type-check. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- Doxyfile | 3 +- README.md | 34 ++++++ merklecpp_tiles.h | 270 +++++++++++++++++++++++++++++++++++++++++ test/CMakeLists.txt | 1 + test/tiles_entries.cpp | 159 ++++++++++++++++++++++++ 5 files changed, 466 insertions(+), 1 deletion(-) create mode 100644 test/tiles_entries.cpp diff --git a/Doxyfile b/Doxyfile index 33229bb..d045b46 100644 --- a/Doxyfile +++ b/Doxyfile @@ -790,7 +790,8 @@ WARN_LOGFILE = # spaces. See also FILE_PATTERNS and EXTENSION_MAPPING # Note: If this tag is empty the current directory is searched. -INPUT = merklecpp.h +INPUT = merklecpp.h \ + merklecpp_tiles.h # This tag can be used to specify the character encoding of the source files # that doxygen parses. Internally doxygen uses the UTF-8 encoding. Doxygen uses diff --git a/README.md b/README.md index 904c442..4db5cae 100644 --- a/README.md +++ b/README.md @@ -21,6 +21,40 @@ unusual features like flushing, retracting, and tree segment serialisation. assert(path->verify(root)); +## Tiled storage (tlog-tiles) + +The companion header `merklecpp_tiles.h` adds optional, header-only support for +persisting a tree as [tlog-tiles](https://c2sp.org/tlog-tiles) tile files +*progressively* (on compaction) and for retrieving inclusion and consistency +proofs from those tiles, from the in-memory tree, or from a combination of the +two. The hashing is unchanged: tiles and tile-derived proofs are templated on +the tree's existing hash function, so a tile-derived inclusion proof is +byte-identical to one from `merkle::Tree::path()` and verifies with the same +`merkle::Path::verify()`. + + #include + + merkle::tiles::TiledTree::Config cfg; + cfg.prefix = "/var/log/mylog"; // tile files and checkpoint live here + cfg.retention_margin = 1024; // keep the most recent leaves in memory + + merkle::tiles::TiledTree log(cfg); + for (const auto& leaf_hash : batch) + log.append(leaf_hash); + + // Write newly-complete tiles and a checkpoint, then compact (flush) memory. + log.checkpoint(); + + // Proofs are served from tiles + the resident tree, even for flushed leaves. + auto inclusion = log.inclusion_proof(/*index=*/0, log.size()); + assert(inclusion->verify(log.root())); + + auto consistency = log.consistency_proof(/*m=*/100, /*n=*/log.size()); + +See [doc/design/tlog-tiles.md](doc/design/tlog-tiles.md) for the full design, +file/directory layout, and the proof algorithms. + + ## Contributing This project welcomes contributions and suggestions. Most contributions require you to agree to a diff --git a/merklecpp_tiles.h b/merklecpp_tiles.h index a50b205..d5e7d83 100644 --- a/merklecpp_tiles.h +++ b/merklecpp_tiles.h @@ -240,6 +240,88 @@ namespace merkle return true; } + /// @brief Whether a full entry bundle exists on disk. + [[nodiscard]] bool has_entry_bundle(uint64_t index) const + { + return std::filesystem::exists(entries_path(index, 0)); + } + + /// @brief Writes an entry bundle to disk atomically. + /// @param index The bundle index + /// @param width The bundle width; 0 denotes a full bundle (TILE_WIDTH) + /// @param entries The raw log entries (exactly width, or TILE_WIDTH) + /// @note Entries are stored in the tlog-tiles entry-bundle format: a + /// sequence of big-endian uint16 length-prefixed byte strings. + void write_entry_bundle( + uint64_t index, + uint16_t width, + const std::vector>& entries) + { + const uint16_t expected = width == 0 ? TILE_WIDTH : width; + if (entries.size() != expected) + { + throw std::runtime_error("entry bundle width mismatch"); + } + write_file_atomically( + entries_path(index, width), encode_entries(entries)); + } + + /// @brief Reads an entry bundle from disk. + /// @param index The bundle index + /// @param width The bundle width; 0 denotes a full bundle (TILE_WIDTH) + /// @return The raw log entries + [[nodiscard]] std::vector> read_entry_bundle( + uint64_t index, uint16_t width = 0) const + { + const uint16_t count = width == 0 ? TILE_WIDTH : width; + return decode_entries(read_file(entries_path(index, width)), count); + } + + /// @brief Encodes log entries into the tlog-tiles entry-bundle format. + static std::vector encode_entries( + const std::vector>& entries) + { + std::vector bytes; + for (const auto& e : entries) + { + if (e.size() > 0xFFFF) + { + throw std::runtime_error( + "entry too large for uint16 length prefix"); + } + bytes.push_back((uint8_t)((e.size() >> 8) & 0xFF)); + bytes.push_back((uint8_t)(e.size() & 0xFF)); + bytes.insert(bytes.end(), e.begin(), e.end()); + } + return bytes; + } + + /// @brief Decodes @p count entries from the entry-bundle format. + static std::vector> decode_entries( + const std::vector& bytes, size_t count) + { + std::vector> out; + out.reserve(count); + size_t pos = 0; + for (size_t i = 0; i < count; i++) + { + if (pos + 2 > bytes.size()) + { + throw std::runtime_error("truncated entry bundle"); + } + const uint16_t len = + (uint16_t)(((uint16_t)bytes[pos] << 8) | bytes[pos + 1]); + pos += 2; + if (pos + len > bytes.size()) + { + throw std::runtime_error("truncated entry bundle"); + } + out.emplace_back(bytes.begin() + pos, bytes.begin() + pos + len); + pos += len; + } + return out; + } + protected: /// @brief The root directory of the store. std::filesystem::path prefix; @@ -1182,6 +1264,187 @@ namespace merkle uint64_t tiles_size = 0; }; + /// @brief Writes tlog-tiles entry bundles (raw log entries) for a growing + /// log. + /// @note Entry bundles are level-0 only and application-owned: merklecpp + /// stores leaf hashes, while the raw entries (and the leaf-hash derivation + /// linking each entry to its level-0 tile hash) are the application's + /// responsibility. Full bundles are immutable and written once; only the + /// rightmost partial bundle may grow or be removed once superseded. + template < + size_t HASH_SIZE, + void HASH_FUNCTION( + const HashT&, const HashT&, HashT&)> + class EntryBundleWriterT + { + public: + using Store = TileStoreT; + + /// @brief Supplies the raw bytes of the log entry at a given index. + using EntryFn = std::function(uint64_t)>; + + /// @brief Write-path options. + struct Options + { + /// @brief Write the partial (rightmost) bundle for the current size. + bool write_partial = true; + + /// @brief Remove a partial bundle once superseded by a full bundle (or + /// a wider partial at the same index). + bool remove_superseded_partials = true; + }; + + /// @brief Counts of work performed by a write_up_to call. + struct Stats + { + uint64_t full_written = 0; + uint64_t partial_written = 0; + uint64_t partial_removed = 0; + }; + + explicit EntryBundleWriterT(Store& store, Options options = {}) : + store(store), options(options) + {} + + /// @brief Writes all newly-complete full bundles and the partial bundle + /// for a log of @p size entries. + /// @param size The current number of entries + /// @param entry_at Returns the raw bytes of the entry at an index in + /// [0, size) + /// @return Counts of bundles written/removed + Stats write_up_to(uint64_t size, const EntryFn& entry_at) + { + Stats stats; + const uint64_t full = size / TILE_WIDTH; + + if (!cursor_inited) + { + next_full = full_prefix_length(); + cursor_inited = true; + } + + for (uint64_t n = next_full; n < full; n++) + { + if (store.has_entry_bundle(n)) + { + continue; // immutable: never rewrite an existing full bundle + } + store.write_entry_bundle( + n, 0, collect(n * TILE_WIDTH, TILE_WIDTH, entry_at)); + stats.full_written++; + } + if (full > next_full) + { + next_full = full; + } + + if (options.write_partial) + { + write_partial(full, (uint16_t)(size % TILE_WIDTH), entry_at, stats); + } + return stats; + } + + protected: + Store& store; + Options options; + uint64_t next_full = 0; + bool cursor_inited = false; + uint64_t last_partial_index = 0; + uint16_t last_partial_width = 0; + bool has_last_partial = false; + + std::vector> collect( + uint64_t first, uint64_t count, const EntryFn& entry_at) + { + std::vector> out; + out.reserve(count); + for (uint64_t i = 0; i < count; i++) + { + out.push_back(entry_at(first + i)); + } + return out; + } + + [[nodiscard]] uint64_t full_prefix_length() const + { + if (!store.has_entry_bundle(0)) + { + return 0; + } + uint64_t lo = 0; + uint64_t hi = 1; + while (store.has_entry_bundle(hi)) + { + lo = hi; + hi <<= 1; + } + while (hi - lo > 1) + { + const uint64_t mid = lo + (hi - lo) / 2; + if (store.has_entry_bundle(mid)) + { + lo = mid; + } + else + { + hi = mid; + } + } + return lo + 1; + } + + void write_partial( + uint64_t index, uint16_t width, const EntryFn& entry_at, Stats& stats) + { + if (width == 0) + { + if (options.remove_superseded_partials && has_last_partial) + { + remove_partial(last_partial_index); + has_last_partial = false; + stats.partial_removed++; + } + return; + } + + if ( + has_last_partial && last_partial_index == index && + last_partial_width == width) + { + return; + } + + if (options.remove_superseded_partials && has_last_partial) + { + if (last_partial_index != index) + { + remove_partial(last_partial_index); + stats.partial_removed++; + } + else + { + remove_partial(index); + } + } + + store.write_entry_bundle( + index, width, collect(index * TILE_WIDTH, width, entry_at)); + stats.partial_written++; + last_partial_index = index; + last_partial_width = width; + has_last_partial = true; + } + + void remove_partial(uint64_t index) + { + const std::filesystem::path dir = + store.entries_path(index, 1).parent_path(); + std::error_code ec; + std::filesystem::remove_all(dir, ec); + } + }; + /// @brief Default tile store (SHA256, default hash function). using TileStore = TileStoreT<32, sha256_compress>; @@ -1206,6 +1469,9 @@ namespace merkle /// @brief Default tiled tree (SHA256, default hash function). using TiledTree = TiledTreeT<32, sha256_compress>; + /// @brief Default entry-bundle writer (SHA256, default hash function). + using EntryBundleWriter = EntryBundleWriterT<32, sha256_compress>; + #ifdef HAVE_OPENSSL /// @brief SHA384 tile store. using TileStore384 = TileStoreT<48, sha384_openssl>; @@ -1238,6 +1504,10 @@ namespace merkle using MemoryHashSource512 = MemoryHashSourceT<64, sha512_openssl>; using CombinedHashSource512 = CombinedHashSourceT<64, sha512_openssl>; using TiledTree512 = TiledTreeT<64, sha512_openssl>; + + /// @brief SHA384/512 entry-bundle writers. + using EntryBundleWriter384 = EntryBundleWriterT<48, sha384_openssl>; + using EntryBundleWriter512 = EntryBundleWriterT<64, sha512_openssl>; #endif } } diff --git a/test/CMakeLists.txt b/test/CMakeLists.txt index cb1fd36..0e889f9 100644 --- a/test/CMakeLists.txt +++ b/test/CMakeLists.txt @@ -33,6 +33,7 @@ add_merklecpp_test(tiles_store tiles_store.cpp) add_merklecpp_test(tiles_writer tiles_writer.cpp) add_merklecpp_test(tiles_proofs tiles_proofs.cpp) add_merklecpp_test(tiles_tree tiles_tree.cpp) +add_merklecpp_test(tiles_entries tiles_entries.cpp) if(TARGET evercrypt.host) add_merklecpp_test(compare_evercrypt compare_evercrypt.cpp) diff --git a/test/tiles_entries.cpp b/test/tiles_entries.cpp new file mode 100644 index 0000000..8fa32bc --- /dev/null +++ b/test/tiles_entries.cpp @@ -0,0 +1,159 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. + +#include "util.h" + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace fs = std::filesystem; +using merkle::Hash; +using merkle::tiles::EntryBundleWriter; +using merkle::tiles::TileRef; +using merkle::tiles::TileStore; +using merkle::tiles::TileWriter; + +static void expect(bool cond, const std::string& what) +{ + if (!cond) + { + throw std::runtime_error("check failed: " + what); + } +} + +int main() +{ + const auto seed = std::time(nullptr); + std::srand((unsigned)seed); + std::cout << "seed=" << seed << '\n'; + + const fs::path base = fs::temp_directory_path() / + ("merklecpp_tiles_entries_" + std::to_string((unsigned long long)seed) + + "_" + std::to_string(std::rand())); + + try + { + // 1. Encode/decode round-trip with variable-length entries. + { + const std::vector> entries = { + {}, {0x01}, {0x02, 0x03, 0x04}, std::vector(300, 0xAB)}; + const auto enc = TileStore::encode_entries(entries); + // 4 length prefixes (8 bytes) + 0 + 1 + 3 + 300 payload bytes. + expect(enc.size() == 8 + 0 + 1 + 3 + 300, "encoded size"); + const auto dec = TileStore::decode_entries(enc, entries.size()); + expect(dec == entries, "encode/decode round-trip"); + } + + const auto hashes = make_hashes(70000); + const auto entry_at = [&](uint64_t i) -> std::vector { + return hashes[i]; // HashT -> vector + }; + + // 2. size 256: a single full bundle, no partial. + { + TileStore store(base / "e256"); + EntryBundleWriter writer(store); + const auto s = writer.write_up_to(256, entry_at); + expect(s.full_written == 1 && s.partial_written == 0, "256 counts"); + expect(store.has_entry_bundle(0), "256 full bundle"); + expect( + !fs::exists(store.entries_path(1, 1).parent_path()), "256 no partial"); + + const auto b0 = store.read_entry_bundle(0); + expect(b0.size() == 256, "256 bundle width"); + for (size_t i = 0; i < 256; i++) + { + expect(Hash(b0[i]) == hashes[i], "256 entry round-trip"); + } + } + + // 3. size 70000: 273 full bundles + a width-112 partial, plus tile linkage. + { + TileStore store(base / "e70k"); + EntryBundleWriter writer(store); + const auto s = writer.write_up_to(70000, entry_at); + expect(s.full_written == 273 && s.partial_written == 1, "70000 counts"); + expect(store.has_entry_bundle(0), "70000 bundle 0"); + expect(store.has_entry_bundle(272), "70000 bundle 272"); + expect(!store.has_entry_bundle(273), "70000 no bundle 273"); + expect(fs::exists(store.entries_path(273, 112)), "70000 partial 112"); + + const auto bp = store.read_entry_bundle(273, 112); + expect(bp.size() == 112, "70000 partial width"); + for (size_t i = 0; i < 112; i++) + { + expect(Hash(bp[i]) == hashes[69888 + i], "70000 partial round-trip"); + } + + // Level-0 tile entries (leaf hashes) correspond to the bundle entries + // under the identity leaf-hash used here. + TileWriter tw(store); + const auto leaf_at = [&](uint64_t i) -> const Hash& { return hashes[i]; }; + tw.write_up_to(70000, leaf_at); + const auto tile0 = store.read_tile(TileRef{0, 0, 0}); + const auto bundle0 = store.read_entry_bundle(0); + for (size_t i = 0; i < 256; i++) + { + expect(Hash(bundle0[i]) == tile0[i], "bundle<->tile linkage"); + } + } + + // 4. Incremental writes: full bundles immutable, partial grows. + { + TileStore store(base / "einc"); + EntryBundleWriter writer(store); + expect(writer.write_up_to(256, entry_at).full_written == 1, "inc s1"); + const auto s2 = writer.write_up_to(256, entry_at); + expect( + s2.full_written == 0 && s2.partial_written == 0, "inc immutable rerun"); + const auto s3 = writer.write_up_to(600, entry_at); + expect(s3.full_written == 1, "inc s3 full"); + expect(store.has_entry_bundle(1), "inc bundle 1"); + expect(fs::exists(store.entries_path(2, 88)), "inc partial 88"); + } + + // 5. Supersession: a partial whose index becomes a full bundle is removed. + { + TileStore store(base / "esup"); + EntryBundleWriter writer(store); + writer.write_up_to(300, entry_at); + expect(fs::exists(store.entries_path(1, 44)), "sup partial 44 before"); + const auto s = writer.write_up_to(512, entry_at); + expect(s.partial_removed == 1, "sup one removed"); + expect(store.has_entry_bundle(1), "sup full bundle 1"); + expect(!fs::exists(store.entries_path(1, 44)), "sup partial 44 gone"); + expect( + !fs::exists(store.entries_path(1, 1).parent_path()), + "sup partial dir gone"); + } + + std::cout << "tiles_entries: OK" << '\n'; + + std::error_code ec; + fs::remove_all(base, ec); + } + catch (std::exception& ex) + { + std::cout << "Error: " << ex.what() << '\n'; + std::error_code ec; + fs::remove_all(base, ec); + return 1; + } + catch (...) + { + std::cout << "Error" << '\n'; + std::error_code ec; + fs::remove_all(base, ec); + return 1; + } + + return 0; +} From 384c4985adc8739d3bbf317f904c6eb7f8bab7cf Mon Sep 17 00:00:00 2001 From: Amaury Chamayou Date: Sat, 13 Jun 2026 16:13:44 +0000 Subject: [PATCH 07/30] feat(tiles): make in-memory compaction opt-in Dropping leaves from the in-memory tree is now opt-in, and only ever drops leaves already covered by a durably-written full tile: - TiledTreeT::Config gains compact_on_checkpoint (default false). checkpoint() always writes tiles but only compacts when the flag is set, so by default the tree keeps every leaf resident. - New TiledTreeT::compact(): flushes to the largest tile-aligned boundary covered by written full tiles (minus retention_margin), callable explicitly. Proofs for dropped leaves remain available from the tiles. This keeps memory compaction a deliberate, tiling-gated choice rather than a side effect of checkpointing. test/tiles_tree.cpp: assert the default writes tiles without dropping (min_index stays 0), that an explicit compact() then drops to the tile boundary, and exercise compact_on_checkpoint=true for the existing flush/combination scenarios. README and design doc updated. 16/16 tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- README.md | 15 ++++++++----- doc/design/tlog-tiles.md | 16 ++++++++++---- merklecpp_tiles.h | 47 +++++++++++++++++++++++++++++----------- test/tiles_tree.cpp | 21 ++++++++++++++++-- 4 files changed, 74 insertions(+), 25 deletions(-) diff --git a/README.md b/README.md index 4db5cae..4376196 100644 --- a/README.md +++ b/README.md @@ -25,9 +25,9 @@ unusual features like flushing, retracting, and tree segment serialisation. The companion header `merklecpp_tiles.h` adds optional, header-only support for persisting a tree as [tlog-tiles](https://c2sp.org/tlog-tiles) tile files -*progressively* (on compaction) and for retrieving inclusion and consistency -proofs from those tiles, from the in-memory tree, or from a combination of the -two. The hashing is unchanged: tiles and tile-derived proofs are templated on +*progressively* (optionally dropping already-tiled leaves from memory) and for +retrieving inclusion and consistency proofs from those tiles, from the in-memory +tree, or from a combination of the two. The hashing is unchanged: tiles and tile-derived proofs are templated on the tree's existing hash function, so a tile-derived inclusion proof is byte-identical to one from `merkle::Tree::path()` and verifies with the same `merkle::Path::verify()`. @@ -35,14 +35,17 @@ byte-identical to one from `merkle::Tree::path()` and verifies with the same #include merkle::tiles::TiledTree::Config cfg; - cfg.prefix = "/var/log/mylog"; // tile files and checkpoint live here - cfg.retention_margin = 1024; // keep the most recent leaves in memory + cfg.prefix = "/var/log/mylog"; // tile files and checkpoint live here + cfg.retention_margin = 1024; // keep the most recent leaves in memory + cfg.compact_on_checkpoint = true; // opt in to dropping already-tiled leaves merkle::tiles::TiledTree log(cfg); for (const auto& leaf_hash : batch) log.append(leaf_hash); - // Write newly-complete tiles and a checkpoint, then compact (flush) memory. + // Write newly-complete tiles (and a checkpoint). With compaction enabled + // this also drops from memory the leaves already covered by a full tile; + // otherwise the tree keeps every leaf and you can call log.compact() later. log.checkpoint(); // Proofs are served from tiles + the resident tree, even for flushed leaves. diff --git a/doc/design/tlog-tiles.md b/doc/design/tlog-tiles.md index af93558..c8d23cb 100644 --- a/doc/design/tlog-tiles.md +++ b/doc/design/tlog-tiles.md @@ -509,7 +509,8 @@ class TiledTreeT { public: struct Config { std::filesystem::path prefix; - uint64_t retention_margin = 0; // keep this many recent leaves resident + uint64_t retention_margin = 0; // keep this many recent leaves resident + bool compact_on_checkpoint = false; // opt in to dropping tiled leaves TileWriterT::Options writer = {}; }; explicit TiledTreeT(Config); @@ -518,8 +519,13 @@ public: uint64_t size() const; // tree.num_leaves Hash root(); // tree.root - // Write newly-complete tiles + partials + checkpoint, then flush_to(coverage). - void checkpoint(); + // Write newly-complete tiles + partials + checkpoint. Compaction (dropping + // already-tiled leaves from memory) happens only if compact_on_checkpoint. + Stats checkpoint(); + + // Drop from memory the leaves already covered by a full tile (opt-in); + // proofs for dropped leaves remain available from the tiles. + uint64_t compact(); // Proofs over tiles ∪ resident tree (works for flushed indices). std::shared_ptr inclusion_proof(uint64_t index, uint64_t size); @@ -562,7 +568,9 @@ Per checkpoint: 2. `write_up_to(size, leaf_at)` — persist newly-complete **full** tiles at all levels (incremental) and the **partial** tiles for `size`. 3. `write_checkpoint(size, root)`. -4. `flush_to(covered − retention_margin)` — reclaim memory. +4. *(optional)* `compact()` → `flush_to(covered − retention_margin)` — reclaim + memory, only when `compact_on_checkpoint` is set (or `compact()` is called + explicitly). By default nothing is dropped and the tree stays whole. Given the invariant, every leaf and every perfect subtree is resolvable: diff --git a/merklecpp_tiles.h b/merklecpp_tiles.h index d5e7d83..d98867f 100644 --- a/merklecpp_tiles.h +++ b/merklecpp_tiles.h @@ -1127,11 +1127,12 @@ namespace merkle }; /// @brief A merkle tree backed by tlog-tiles storage. - /// @note Appends grow an in-memory tree; checkpoint() durably writes tiles - /// and a checkpoint, then compacts memory by flushing the leaves now - /// covered by full tiles. Proofs are served from the combination of the - /// resident tree and the tiles, so they remain available for flushed - /// (compacted) leaves. + /// @note Appends grow an in-memory tree; checkpoint() durably writes the + /// tiles. Compaction (dropping from memory the leaves already covered by a + /// durably-written full tile) is optional: enable it per checkpoint with + /// Config::compact_on_checkpoint, or call compact() explicitly. Proofs are + /// served from the combination of the resident tree and the tiles, so they + /// remain available for compacted (dropped) leaves. template < size_t HASH_SIZE, void HASH_FUNCTION( @@ -1152,10 +1153,15 @@ namespace merkle /// @brief Root directory for tiles and the checkpoint. std::filesystem::path prefix; - /// @brief Number of most-recent leaves to keep resident across a - /// checkpoint (i.e. not flushed). + /// @brief Number of most-recent leaves to keep resident when + /// compacting (i.e. never dropped from memory). uint64_t retention_margin = 0; + /// @brief If set, checkpoint() compacts after writing tiles, dropping + /// from memory the leaves already covered by a full tile. Off by + /// default: tiles are written but the tree keeps every leaf resident. + bool compact_on_checkpoint = false; + /// @brief Tile writer options. typename Writer::Options writer = {}; }; @@ -1202,7 +1208,8 @@ namespace merkle return store; } - /// @brief Writes tiles and a checkpoint, then compacts memory. + /// @brief Writes tiles and a checkpoint; compacts only if + /// Config::compact_on_checkpoint is set. /// @return Counts of the tiles written/removed by this checkpoint Stats checkpoint() { @@ -1219,10 +1226,24 @@ namespace merkle store.write_checkpoint(n, tree.root()); tiles_size = n; - // Compaction: flush to the durable full-tile coverage, keeping the - // retention margin resident. The flush target is a multiple of - // TILE_WIDTH so the coverage invariant (min_index <= covered) holds. - const uint64_t covered = (n / TILE_WIDTH) * TILE_WIDTH; + if (config.compact_on_checkpoint) + { + compact(); + } + return stats; + } + + /// @brief Drops from the in-memory tree every leaf already covered by a + /// durably-written full tile, keeping retention_margin recent leaves. + /// @return The new minimum (smallest still-resident) leaf index + /// @note Only leaves whose full tile already exists are dropped, so + /// inclusion and consistency proofs remain available (served from the + /// tiles). Has no effect until tiling has produced full tiles. The flush + /// target is a multiple of TILE_WIDTH, so the coverage invariant + /// (min_index <= durable full-tile coverage) always holds. + uint64_t compact() + { + const uint64_t covered = (tiles_size / TILE_WIDTH) * TILE_WIDTH; uint64_t target = covered > config.retention_margin ? covered - config.retention_margin : 0; @@ -1231,7 +1252,7 @@ namespace merkle { tree.flush_to((size_t)target); } - return stats; + return tree.min_index(); } /// @brief Inclusion proof for @p index in a tree of @p proof_size leaves. diff --git a/test/tiles_tree.cpp b/test/tiles_tree.cpp index c4b7ef0..554b45e 100644 --- a/test/tiles_tree.cpp +++ b/test/tiles_tree.cpp @@ -148,18 +148,35 @@ int main() } const Hash ref_root = ref.root(); + // Default mode: checkpoint() writes tiles but drops nothing from memory; + // an explicit compact() drops only the leaves already covered by a tile. + { + TiledTree::Config dcfg; + dcfg.prefix = base / "tt_default"; + TiledTree dtt(dcfg); + for (uint64_t i = 0; i < n1; i++) + { + dtt.append(hashes[i]); + } + dtt.checkpoint(); + expect(dtt.tree_ref().min_index() == 0, "default: nothing dropped"); + dtt.compact(); + expect(dtt.tree_ref().min_index() == 768, "compact() drops to 768"); + } + TiledTree::Config cfg; cfg.prefix = base / "tt"; cfg.retention_margin = 0; + cfg.compact_on_checkpoint = true; TiledTree tt(cfg); for (uint64_t i = 0; i < n1; i++) { tt.append(hashes[i]); } - tt.checkpoint(); // tiles_size = 1000; flush to covered = 768 + tt.checkpoint(); // tiles_size = 1000; compact to covered = 768 expect(tt.checkpoint_size() == n1, "checkpoint size"); - expect(tt.tree_ref().min_index() == 768, "flushed to 768"); + expect(tt.tree_ref().min_index() == 768, "compacted to 768"); for (uint64_t i = n1; i < N; i++) { From 4e3f11b7fa9c6ef81a80d656208c4a4f7fe14b23 Mon Sep 17 00:00:00 2001 From: Amaury Chamayou Date: Sat, 13 Jun 2026 16:24:17 +0000 Subject: [PATCH 08/30] feat(tiles): guard rollback against immutable tiles Rolling the tree back over already-tiled entries left the immutable full tiles stale (never rewritten), so the tile-derived root diverged from the tree and proofs failed to verify. TiledTree now refuses this: - TiledTreeT::retract_to(index) throws if the resulting size would be below checkpoint_size() (entries committed to immutable tiles cannot be rolled back); it permits rolling back only the un-tiled, post-checkpoint frontier, after which tiles stay consistent. - Retracting the underlying tree directly via tree_ref() still bypasses the guard and is documented as unsafe. test/tiles_tree.cpp Part 3: a frontier rollback followed by re-append and checkpoint keeps the tile-derived root equal to the tree root; rolling back below the tiled size (and the exact boundary) throws. Design doc risks updated. 16/16 tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/tlog-tiles.md | 6 ++++ merklecpp_tiles.h | 19 +++++++++++++ test/tiles_tree.cpp | 60 ++++++++++++++++++++++++++++++++++++++++ 3 files changed, 85 insertions(+) diff --git a/doc/design/tlog-tiles.md b/doc/design/tlog-tiles.md index c8d23cb..cb3422f 100644 --- a/doc/design/tlog-tiles.md +++ b/doc/design/tlog-tiles.md @@ -696,6 +696,12 @@ compatible": mitigate this. - **`flush_to` alignment.** Must flush only to a 256-multiple (minus retention) to uphold the coverage invariant; enforced inside `TiledTreeT::checkpoint`. +- **Rollback vs. immutable tiles.** Tiles are write-once, so rolling the tree + back (`retract_to`) over already-tiled entries would leave stale, never- + rewritten tiles and is forbidden: `TiledTreeT::retract_to` throws if the + resulting size is below `checkpoint_size()`, permitting only rollback of the + un-tiled (post-checkpoint) frontier. Retracting the underlying tree directly + via `tree_ref()` bypasses this guard and must be avoided. - **Very large indices.** Index math uses `uint64_t`; encoding handles multi-group indices. Level bound `≤ 63` per spec (8 suffices for `2^64`). - **Open question — `subtree_root` in core vs. `past_path`-derived memory diff --git a/merklecpp_tiles.h b/merklecpp_tiles.h index d98867f..ab4f9e4 100644 --- a/merklecpp_tiles.h +++ b/merklecpp_tiles.h @@ -1255,6 +1255,25 @@ namespace merkle return tree.min_index(); } + /// @brief Rolls the tree back so that @p index becomes the last leaf, + /// removing all leaves after it (same semantics as TreeT::retract_to). + /// @note Tiles are immutable, so entries already committed to tiles + /// cannot be rolled back: this throws if the resulting size would be + /// smaller than checkpoint_size(). Only un-checkpointed (not-yet-tiled) + /// entries may be rolled back. (Retracting the underlying tree directly + /// via tree_ref() bypasses this guard and can leave stale tiles -- do not + /// do that.) + void retract_to(size_t index) + { + if ((uint64_t)index + 1 < tiles_size) + { + throw std::runtime_error( + "TiledTree::retract_to: cannot roll back entries already committed " + "to immutable tiles (resulting size < checkpoint size)"); + } + tree.retract_to(index); + } + /// @brief Inclusion proof for @p index in a tree of @p proof_size leaves. /// @note Served from tiles (flushed past) combined with the resident tree /// (recent frontier); @p proof_size may exceed checkpoint_size(). diff --git a/test/tiles_tree.cpp b/test/tiles_tree.cpp index 554b45e..59965e3 100644 --- a/test/tiles_tree.cpp +++ b/test/tiles_tree.cpp @@ -255,6 +255,66 @@ int main() std::cout << "tiled tree (flush + combination): OK" << '\n'; + // ---- Part 3: rollback. Only un-tiled (post-checkpoint) entries may be + // rolled back; rolling back committed entries is refused. + { + TiledTree::Config cfg; + cfg.prefix = base / "rb"; + TiledTree rb(cfg); + for (uint64_t i = 0; i < 300; i++) + { + rb.append(hashes[i]); + } + rb.checkpoint(); // tiles_size = 300 + expect(rb.checkpoint_size() == 300, "rb checkpoint size"); + + // Append an un-tiled frontier, roll part of it back, then re-append + // DIFFERENT leaves. The tiled region [0,300) is never touched. + for (uint64_t i = 300; i < 400; i++) + { + rb.append(hashes[i]); + } + rb.retract_to(349); // keep [0,349], size 350 + expect(rb.size() == 350, "rb retracted to 350"); + for (uint64_t i = 350; i < 400; i++) + { + rb.append(hashes[1000 + i]); // different leaves + } + rb.checkpoint(); // tiles_size = 400 + + // Tile-derived root equals the tree root: no corruption. + merkle::tiles::TileHashSource src(rb.store_ref(), rb.checkpoint_size()); + ProofEngine eng(src); + expect( + eng.root(rb.checkpoint_size()) == rb.tree_ref().root(), + "rb frontier rollback keeps tiles consistent"); + + // Rolling back already-tiled entries is refused. + bool threw = false; + try + { + rb.retract_to(100); + } + catch (const std::exception&) + { + threw = true; + } + expect(threw, "rb retract below tiled size throws"); + + // Boundary: resulting size tiles_size-1 (399 < 400) throws. + threw = false; + try + { + rb.retract_to(398); + } + catch (const std::exception&) + { + threw = true; + } + expect(threw, "rb retract to size 399 throws"); + } + std::cout << "rollback: OK" << '\n'; + std::cout << "tiles_tree: OK" << '\n'; std::error_code ec; From f08b3b17bdbdd1e1d3c3d87908b753799f34ea08 Mon Sep 17 00:00:00 2001 From: Amaury Chamayou Date: Sat, 13 Jun 2026 16:33:43 +0000 Subject: [PATCH 09/30] docs+test: tiles API guide and expanded rollback coverage - Add doc/tiles-guide.md: a practical how-to for the tiles API (TiledTree quick start, checkpoints, opt-in compaction, rollback semantics, inclusion/consistency proofs, lower-level TileStore/ TileWriter/HashSource/ProofEngine building blocks, entry bundles, and the on-disk layout). Linked from the README. - Expand the rollback tests (test/tiles_tree.cpp Part 3): rollback is unrestricted before any checkpoint; after a checkpoint a frontier rollback + re-append + checkpoint matches a reference tree on root, inclusion proofs (tiled and frontier indices) and a consistency proof; the below-tiled and exact-boundary cases throw while retracting to the tiled size is a no-op; and rollback interacts correctly with compaction (a flushed+tiled leaf stays provable; rolling back into the flushed past throws). 16/16 tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- README.md | 4 +- doc/tiles-guide.md | 243 ++++++++++++++++++++++++++++++++++++++++++++ test/tiles_tree.cpp | 178 +++++++++++++++++++++++--------- 3 files changed, 376 insertions(+), 49 deletions(-) create mode 100644 doc/tiles-guide.md diff --git a/README.md b/README.md index 4376196..83c5ad6 100644 --- a/README.md +++ b/README.md @@ -54,7 +54,9 @@ byte-identical to one from `merkle::Tree::path()` and verifies with the same auto consistency = log.consistency_proof(/*m=*/100, /*n=*/log.size()); -See [doc/design/tlog-tiles.md](doc/design/tlog-tiles.md) for the full design, +See the [tiled storage guide](doc/tiles-guide.md) for a how-to covering +checkpoints, compaction, rollback, proofs, and the lower-level building blocks, +and [doc/design/tlog-tiles.md](doc/design/tlog-tiles.md) for the full design, file/directory layout, and the proof algorithms. diff --git a/doc/tiles-guide.md b/doc/tiles-guide.md new file mode 100644 index 0000000..b71ee23 --- /dev/null +++ b/doc/tiles-guide.md @@ -0,0 +1,243 @@ +# Tiled storage and proofs — a guide to `merklecpp_tiles.h` + +`merklecpp_tiles.h` is an optional, header-only companion to `merklecpp.h`. It +lets you persist a Merkle tree as a set of immutable **tile files** on disk and +serve **inclusion** and **consistency** proofs from those tiles, from the +in-memory tree, or from a combination of the two — so proofs stay available even +after old entries are dropped from memory. + +It builds on the [tlog-tiles](https://c2sp.org/tlog-tiles) file/directory layout +but is **not** trying to be wire-compatible with external tlog-tiles clients. See +[`design/tlog-tiles.md`](design/tlog-tiles.md) for the design and internals; this +page is a practical how-to. + +## Contents + +- [Requirements and a note on hashing](#requirements-and-a-note-on-hashing) +- [Quick start: `TiledTree`](#quick-start-tiledtree) +- [Checkpoints and compaction](#checkpoints-and-compaction) +- [Rollback](#rollback) +- [Proofs](#proofs) +- [Lower-level building blocks](#lower-level-building-blocks) +- [Entry bundles (optional)](#entry-bundles-optional) +- [On-disk layout](#on-disk-layout) + +## Requirements and a note on hashing + +- C++17 (the header uses `` and ``). +- Include the companion header; it pulls in `merklecpp.h` for you: + + ```cpp + #include + ``` + +- Everything lives in `namespace merkle::tiles` and is templated on the same + `` as your tree. The default aliases + (`merkle::tiles::TiledTree`, `TileStore`, `TileWriter`, `ProofEngine`, …) use + the **same** SHA-256 as `merkle::Tree`, so a tile-derived inclusion proof is + byte-identical to one from `merkle::Tree::path()` and verifies with the usual + `merkle::Path::verify()`. +- You insert **leaf hashes**, not raw entries — exactly like `merkle::Tree`. + Deriving a leaf hash from an entry (e.g. `leaf = H(entry)`) is your + application's job. The tile hash values are whatever your `HASH_FUNCTION` + produces; they are not RFC 6962 unless you instantiate your tree with an + RFC 6962 hash function (not required, and not the goal here). + +## Quick start: `TiledTree` + +`TiledTree` is the high-level wrapper: append leaf hashes, take checkpoints +(which write tiles), and ask for proofs. + +```cpp +#include + +merkle::tiles::TiledTree::Config cfg; +cfg.prefix = "/var/log/mylog"; // directory for tile files + checkpoint + +merkle::tiles::TiledTree log(cfg); + +// Append leaf hashes (compute these from your entries however you like). +for (const merkle::Hash& leaf : batch) + log.append(leaf); + +// Persist newly-complete tiles (and a small checkpoint file) to disk. +log.checkpoint(); + +merkle::Hash root = log.root(); // current Merkle root +uint64_t n = log.size(); // number of leaves + +// Inclusion proof for leaf 0 in the tree of `n` leaves. +auto inclusion = log.inclusion_proof(/*index=*/0, /*size=*/n); +assert(inclusion->verify(root)); + +// Consistency proof that size 100 is a prefix of size n. +auto consistency = log.consistency_proof(/*m=*/100, /*n=*/n); +``` + +`checkpoint()` is incremental: each call writes only the tiles that became +complete since the previous call, plus the partial tiles for the current size. +Full tiles are immutable — written once and never rewritten. + +## Checkpoints and compaction + +By default `checkpoint()` only *writes* tiles; it keeps every leaf resident in +memory. Dropping already-tiled leaves from memory ("compaction") is **opt-in**, +because once you drop them you can only prove them from the tiles. + +```cpp +merkle::tiles::TiledTree::Config cfg; +cfg.prefix = "/var/log/mylog"; +cfg.compact_on_checkpoint = true; // drop tiled leaves after each checkpoint +cfg.retention_margin = 4096; // ...but keep the most recent 4096 resident + +merkle::tiles::TiledTree log(cfg); +``` + +- `compact_on_checkpoint` (default `false`): when set, `checkpoint()` calls + `compact()` for you. +- `compact()` can also be called explicitly at any time. It drops from memory + only the leaves already covered by a **durably written full tile**, keeping + `retention_margin` recent leaves resident. It returns the new minimum + (smallest still-resident) leaf index. +- Proofs for dropped leaves are still produced — they are served from the tiles + and transparently combined with the resident frontier. + +```cpp +log.compact(); // free memory now +uint64_t resident_from = log.tree_ref().min_index(); +``` + +## Rollback + +Tiles are immutable, so you may only roll back entries that have **not yet been +committed to tiles** (i.e. appended since the last checkpoint). `retract_to` +enforces this: + +```cpp +log.retract_to(index); // keep leaves [0, index], drop the rest +``` + +- Allowed when the resulting size is `>= checkpoint_size()` (only the un-tiled + frontier is removed). +- Throws otherwise (rolling back committed entries would leave stale tiles). +- `retract_to` mirrors `merkle::Tree::retract_to`: `index` is the new *last* + leaf, so the resulting size is `index + 1`. + +> ⚠️ Retracting the underlying tree directly via `log.tree_ref().retract_to(...)` +> bypasses this guard and can leave the tiles inconsistent with the tree. Use +> `TiledTree::retract_to` instead. + +## Proofs + +Both proof types come from `TiledTree` (or, at a lower level, from +`ProofEngine`). They are produced with your tree's hash function, so they match +what `merkle::Tree` would produce. + +### Inclusion proofs + +```cpp +// Prove leaf `index` in a tree of `size` leaves. +std::shared_ptr p = log.inclusion_proof(index, size); +bool ok = p->verify(root_at_size); +``` + +`size` is the tree size you are proving against: + +- `size == log.size()` ⇒ equivalent to `merkle::Tree::path(index)`; verify + against `log.root()`. +- a past `size` ⇒ equivalent to `merkle::Tree::past_path(index, size - 1)`; + verify against the root at that size (e.g. the root from the checkpoint you are + auditing). + +`size` may even exceed `checkpoint_size()`: the recent, not-yet-tiled frontier is +taken from the resident tree while the older part comes from tiles. + +### Consistency proofs + +```cpp +std::vector proof = log.consistency_proof(m, n); // m <= n +bool ok = merkle::tiles::ProofEngine::verify_consistency( + m, n, old_root /* root at size m */, new_root /* root at size n */, proof); +``` + +`verify_consistency` is a static helper, so you can verify on a client that only +has the two roots and the proof. + +## Lower-level building blocks + +If you manage your own tree/storage you can use the pieces directly instead of +`TiledTree`. + +### Writing tiles from your own tree + +```cpp +merkle::Tree tree; +for (auto& leaf : batch) tree.insert(leaf); + +merkle::tiles::TileStore store("/var/log/mylog"); +merkle::tiles::TileWriter writer(store); + +// Write all newly-complete full tiles and the partial tiles for this size. +auto stats = writer.write_up_to( + tree.num_leaves(), + [&](uint64_t i) -> const merkle::Hash& { return tree.leaf(i); }); +// stats.full_written / stats.partial_written / stats.partial_removed +``` + +### Reading tiles and computing proofs + +A `HashSource` resolves the root of a complete subtree; pick where it reads from: + +- `TileHashSource(store, available_size)` — from tile files. +- `MemoryHashSource(tree)` — from a resident `merkle::Tree`. +- `CombinedHashSource(primary, secondary)` — try `primary` first, then + `secondary` (e.g. memory then tiles). + +```cpp +merkle::tiles::TileHashSource src(store, /*available_size=*/tree.num_leaves()); +merkle::tiles::ProofEngine engine(src); + +merkle::Hash root = engine.root(size); +auto inclusion = engine.inclusion_proof(index, size); +auto consistency = engine.consistency_proof(m, n); +``` + +`TiledTree` simply wires a `CombinedHashSource(MemoryHashSource, TileHashSource)` +into a `ProofEngine` for you. + +## Entry bundles (optional) + +If you also want to store the raw log entries (tlog-tiles "entry bundles"), use +`EntryBundleWriter`. Bundles are level-0 only and application-owned — merklecpp +stores leaf hashes; you supply the raw bytes and decide how an entry maps to its +leaf hash. + +```cpp +merkle::tiles::EntryBundleWriter bundles(store); +bundles.write_up_to(num_entries, + [&](uint64_t i) -> std::vector { return raw_entry_bytes(i); }); + +// Read them back (full bundle = 256 entries; pass the width for a partial one). +std::vector> e = store.read_entry_bundle(/*index=*/0); +``` + +Entries are encoded as big-endian `uint16` length-prefixed byte strings. + +## On-disk layout + +Under the configured `prefix`: + +``` +/ + checkpoint # latest + tile/0/000, tile/0/001 … # level-0 tiles (leaf hashes), 256 hashes each + tile/0/.p/ # partial tile of width W + tile/1/… # higher levels (roll-ups of full tiles below) + tile/entries/… # optional raw entry bundles +``` + +Tile indices use the tlog-tiles path encoding: zero-padded 3-digit groups with +all but the last prefixed by `x` (e.g. index `1234067` → `x001/x234/067`). Full +tiles are immutable; partial tiles grow and may be removed once superseded. See +[`design/tlog-tiles.md`](design/tlog-tiles.md) for the full specification of the +geometry and proof algorithms. diff --git a/test/tiles_tree.cpp b/test/tiles_tree.cpp index 59965e3..79e5958 100644 --- a/test/tiles_tree.cpp +++ b/test/tiles_tree.cpp @@ -255,63 +255,145 @@ int main() std::cout << "tiled tree (flush + combination): OK" << '\n'; - // ---- Part 3: rollback. Only un-tiled (post-checkpoint) entries may be - // rolled back; rolling back committed entries is refused. + // ---- Part 3: rollback. Tiles are immutable, so only un-tiled + // (post-checkpoint) entries may be rolled back. { - TiledTree::Config cfg; - cfg.prefix = base / "rb"; - TiledTree rb(cfg); - for (uint64_t i = 0; i < 300; i++) + // 3a. Before any checkpoint nothing is tiled, so rollback is + // unrestricted. { - rb.append(hashes[i]); + TiledTree::Config cfg; + cfg.prefix = base / "rb_pre"; + TiledTree rb(cfg); + for (uint64_t i = 0; i < 50; i++) + { + rb.append(hashes[i]); + } + rb.retract_to(29); // tiles_size == 0 -> allowed + expect(rb.size() == 30, "rb pre-checkpoint retract allowed"); } - rb.checkpoint(); // tiles_size = 300 - expect(rb.checkpoint_size() == 300, "rb checkpoint size"); - // Append an un-tiled frontier, roll part of it back, then re-append - // DIFFERENT leaves. The tiled region [0,300) is never touched. - for (uint64_t i = 300; i < 400; i++) + // 3b. After a checkpoint, the un-tiled frontier can be rolled back and + // the + // tiled region stays consistent and provable; committed entries + // can't. { - rb.append(hashes[i]); + TiledTree::Config cfg; + cfg.prefix = base / "rb"; + TiledTree rb(cfg); + for (uint64_t i = 0; i < 300; i++) + { + rb.append(hashes[i]); + } + rb.checkpoint(); // tiles_size = 300 + for (uint64_t i = 300; i < 400; i++) + { + rb.append(hashes[i]); + } + rb.retract_to(349); // keep [0,349] + expect(rb.size() == 350, "rb retracted to 350"); + for (uint64_t i = 350; i < 400; i++) + { + rb.append(hashes[1000 + i]); // re-append DIFFERENT leaves + } + rb.checkpoint(); // tiles_size = 400 + + // Reference tree of the exact post-rollback state. + merkle::Tree exp_tree; + for (uint64_t i = 0; i < 350; i++) + { + exp_tree.insert(hashes[i]); + } + for (uint64_t i = 350; i < 400; i++) + { + exp_tree.insert(hashes[1000 + i]); + } + const Hash exp_root = exp_tree.root(); + expect(rb.root() == exp_root, "rb root matches reference"); + + // Proofs for a tiled index and a frontier index match the reference. + for (uint64_t i : {(uint64_t)100, (uint64_t)299, (uint64_t)399}) + { + const auto p = rb.inclusion_proof(i, 400); + expect( + *p == *exp_tree.path(i), + "rb inclusion==ref i=" + std::to_string(i)); + expect( + p->verify(exp_root), "rb inclusion verify i=" + std::to_string(i)); + } + const auto cp = rb.consistency_proof(300, 400); + expect( + ProofEngine::verify_consistency( + 300, 400, *exp_tree.past_root(299), exp_root, cp), + "rb consistency 300->400"); + + // Rolling back committed entries is refused (below and at the + // boundary); rolling back to exactly the tiled size removes nothing and + // is allowed. + bool threw = false; + try + { + rb.retract_to(100); + } + catch (const std::exception&) + { + threw = true; + } + expect(threw, "rb retract below tiled size throws"); + threw = false; + try + { + rb.retract_to(398); + } + catch (const std::exception&) + { + threw = true; + } + expect(threw, "rb retract to size 399 throws"); + rb.retract_to(399); + expect(rb.size() == 400, "rb retract to tiled size is a no-op"); } - rb.retract_to(349); // keep [0,349], size 350 - expect(rb.size() == 350, "rb retracted to 350"); - for (uint64_t i = 350; i < 400; i++) - { - rb.append(hashes[1000 + i]); // different leaves - } - rb.checkpoint(); // tiles_size = 400 - - // Tile-derived root equals the tree root: no corruption. - merkle::tiles::TileHashSource src(rb.store_ref(), rb.checkpoint_size()); - ProofEngine eng(src); - expect( - eng.root(rb.checkpoint_size()) == rb.tree_ref().root(), - "rb frontier rollback keeps tiles consistent"); - // Rolling back already-tiled entries is refused. - bool threw = false; - try - { - rb.retract_to(100); - } - catch (const std::exception&) - { - threw = true; - } - expect(threw, "rb retract below tiled size throws"); - - // Boundary: resulting size tiles_size-1 (399 < 400) throws. - threw = false; - try - { - rb.retract_to(398); - } - catch (const std::exception&) + // 3c. Rollback interacts correctly with compaction (flushed + tiled + // past). { - threw = true; + TiledTree::Config cfg; + cfg.prefix = base / "rb_compact"; + cfg.compact_on_checkpoint = true; + TiledTree rb(cfg); + for (uint64_t i = 0; i < 1000; i++) + { + rb.append(hashes[i]); + } + rb.checkpoint(); // tiles_size = 1000; flush to 768 + expect(rb.tree_ref().min_index() == 768, "rb compact flushed to 768"); + for (uint64_t i = 1000; i < 1200; i++) + { + rb.append(hashes[i]); + } + rb.retract_to(1099); // frontier rollback (>= tiles_size) is allowed + expect(rb.size() == 1100, "rb compact frontier retract ok"); + + merkle::Tree exp_tree; + for (uint64_t i = 0; i < 1100; i++) + { + exp_tree.insert(hashes[i]); + } + const Hash exp_root = exp_tree.root(); + const auto p = rb.inclusion_proof(0, 1100); // leaf 0 is flushed + tiled + expect(*p == *exp_tree.path(0), "rb compact flushed inclusion==ref"); + expect(p->verify(exp_root), "rb compact flushed inclusion verify"); + + bool threw = false; + try + { + rb.retract_to(500); + } + catch (const std::exception&) + { + threw = true; + } + expect(threw, "rb compact retract below tiled throws"); } - expect(threw, "rb retract to size 399 throws"); } std::cout << "rollback: OK" << '\n'; From b859a043e083bba1a16c9fd6004357a823900a57 Mon Sep 17 00:00:00 2001 From: Amaury Chamayou Date: Sat, 13 Jun 2026 16:41:09 +0000 Subject: [PATCH 10/30] feat(tiles): add index-based consistency_proof variant Add consistency_proof_from_indices(i, j) to both ProofEngineT and TiledTreeT, mapping leaf index i to the tree of size i+1 (the same last-leaf convention as past_path/retract_to); it is exactly consistency_proof(i + 1, j + 1) and works transparently across tiles and the live tree. Tests assert the index variant equals the size variant (and verifies) in tiles_proofs.cpp and across the flush boundary in tiles_tree.cpp. Guide updated. 16/16 tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/tiles-guide.md | 12 ++++++++++++ merklecpp_tiles.h | 23 +++++++++++++++++++++++ test/tiles_proofs.cpp | 5 +++++ test/tiles_tree.cpp | 9 +++++++++ 4 files changed, 49 insertions(+) diff --git a/doc/tiles-guide.md b/doc/tiles-guide.md index b71ee23..a85e71c 100644 --- a/doc/tiles-guide.md +++ b/doc/tiles-guide.md @@ -163,6 +163,18 @@ bool ok = merkle::tiles::ProofEngine::verify_consistency( `verify_consistency` is a static helper, so you can verify on a client that only has the two roots and the proof. +The arguments are tree **sizes** (leaf counts). If you have leaf **indices** +instead, use the variant that maps index `i` to the tree of size `i + 1` (the +"last leaf" convention, matching `past_path`/`retract_to`): + +```cpp +// Equivalent to consistency_proof(i + 1, j + 1). +auto proof = log.consistency_proof_from_indices(i, j); // i <= j +``` + +Both `TiledTree` and the lower-level `ProofEngine` provide +`consistency_proof_from_indices`. + ## Lower-level building blocks If you manage your own tree/storage you can use the pieces directly instead of diff --git a/merklecpp_tiles.h b/merklecpp_tiles.h index ab4f9e4..14b771f 100644 --- a/merklecpp_tiles.h +++ b/merklecpp_tiles.h @@ -901,6 +901,18 @@ namespace merkle return proof; } + /// @brief Consistency proof between the trees whose last leaves are at + /// indices @p first_index and @p second_index (first_index <= + /// second_index). + /// @note Equivalent to consistency_proof(first_index + 1, + /// second_index + 1): it proves the tree of the first first_index + 1 + /// leaves is a prefix of the tree of the first second_index + 1 leaves. + std::vector consistency_proof_from_indices( + uint64_t first_index, uint64_t second_index) const + { + return consistency_proof(first_index + 1, second_index + 1); + } + /// @brief Verifies an RFC 6962 consistency proof reconciling the roots of /// trees of @p m and @p n leaves. static bool verify_consistency( @@ -1296,6 +1308,17 @@ namespace merkle return engine.consistency_proof(m, n); } + /// @brief Consistency proof between the trees whose last leaves are at + /// indices @p first_index and @p second_index (first_index <= + /// second_index). + /// @note Equivalent to consistency_proof(first_index + 1, + /// second_index + 1). + std::vector consistency_proof_from_indices( + uint64_t first_index, uint64_t second_index) + { + return consistency_proof(first_index + 1, second_index + 1); + } + protected: Config config; Store store; diff --git a/test/tiles_proofs.cpp b/test/tiles_proofs.cpp index 5be4294..37fdedc 100644 --- a/test/tiles_proofs.cpp +++ b/test/tiles_proofs.cpp @@ -142,6 +142,11 @@ static void check_size( ProofEngine::verify_consistency(m, k, rm, rk, cp), "consistency " + std::to_string(m) + "->" + std::to_string(k) + at); + // The index-based variant is consistency_proof(i+1, j+1). + expect( + engine.consistency_proof_from_indices(m - 1, k - 1) == cp, + "consistency index variant" + at); + // Tampering with a proof element or a root is rejected. auto bad = cp; bad[0].bytes[0] ^= 0xFFu; diff --git a/test/tiles_tree.cpp b/test/tiles_tree.cpp index 79e5958..1e44a0b 100644 --- a/test/tiles_tree.cpp +++ b/test/tiles_tree.cpp @@ -230,6 +230,15 @@ int main() ProofEngine::verify_consistency( m, N, *ref.past_root(m - 1), ref_root, cp2), "consistency m->N across flush"); + + // Index-based variant: consistency_proof_from_indices(i, j) == + // consistency_proof(i + 1, j + 1), spanning tiles and the live tree. + const auto cpi = tt.consistency_proof_from_indices(m - 1, N - 1); + expect(cpi == cp2, "consistency index variant across flush"); + expect( + ProofEngine::verify_consistency( + m, N, *ref.past_root(m - 1), ref_root, cpi), + "consistency index variant verifies"); } // A second checkpoint flushes further; proofs for now-flushed indices still From 90ff8987161b4be0b13c26fa7d9ae0936b1d70fe Mon Sep 17 00:00:00 2001 From: Amaury Chamayou Date: Sat, 13 Jun 2026 16:58:39 +0000 Subject: [PATCH 11/30] style(tiles): make merklecpp_tiles.h and tests clang-tidy clean Resolve all clang-tidy findings (config has WarningsAsErrors: '*') in the tiles header and tests, and wire the header into CI: merklecpp_tiles.h: - guard mth_range against a zero-width range (also clears a clang-analyzer-core.DivideZero false positive behind the is_pow2 short-circuit) - const-correctness, use-auto, braced-init return, explicit ptrdiff_t casts, uppercase literal suffixes - NOLINT the intentional reference data members (view/adapter types) and the nested-namespace style tests: const-correctness, uppercase suffixes, drop redundant casts, DeMorgan simplification, simpler pair initialisation. ci.yml: also run clang-tidy on merklecpp_tiles.h (like merklecpp.h). Also note proof error behaviour in the guide. clang-tidy now passes on the header and all tiles tests; clang-format clean; 16/16 tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/workflows/ci.yml | 4 +++- doc/tiles-guide.md | 4 +++- merklecpp_tiles.h | 30 +++++++++++++++++++++--------- test/tiles_proofs.cpp | 26 ++++++++++---------------- test/tiles_tree.cpp | 16 ++++++++-------- test/tiles_writer.cpp | 4 ++-- 6 files changed, 47 insertions(+), 37 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 4a7a12b..fe5ce76 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -64,7 +64,9 @@ jobs: - name: Clang-Tidy Header if: matrix.os == 'ubuntu-latest' shell: bash - run: clang-tidy merklecpp.h -p build/${{ matrix.build_type }} --warnings-as-errors='*' + run: | + clang-tidy merklecpp.h -p build/${{ matrix.build_type }} --warnings-as-errors='*' + clang-tidy merklecpp_tiles.h -p build/${{ matrix.build_type }} --warnings-as-errors='*' - name: Test working-directory: ${{github.workspace}}/build/${{ matrix.build_type }} diff --git a/doc/tiles-guide.md b/doc/tiles-guide.md index a85e71c..3c089c3 100644 --- a/doc/tiles-guide.md +++ b/doc/tiles-guide.md @@ -131,7 +131,9 @@ log.retract_to(index); // keep leaves [0, index], drop the rest Both proof types come from `TiledTree` (or, at a lower level, from `ProofEngine`). They are produced with your tree's hash function, so they match -what `merkle::Tree` would produce. +what `merkle::Tree` would produce. Requests outside the current tree (e.g. a +size greater than `size()`, or an out-of-range index) throw `std::runtime_error` +rather than returning an incorrect proof. ### Inclusion proofs diff --git a/merklecpp_tiles.h b/merklecpp_tiles.h index 14b771f..60c69ed 100644 --- a/merklecpp_tiles.h +++ b/merklecpp_tiles.h @@ -22,7 +22,7 @@ // byte-identical to those produced by merkle::TreeT (see // doc/design/tlog-tiles.md). -namespace merkle +namespace merkle // NOLINT(modernize-concat-nested-namespaces) { namespace tiles { @@ -212,7 +212,7 @@ namespace merkle { std::string text = std::to_string(size) + "\n" + root_hash.to_string() + "\n"; - std::vector bytes(text.begin(), text.end()); + const std::vector bytes(text.begin(), text.end()); write_file_atomically(checkpoint_path(), bytes); } @@ -309,14 +309,16 @@ namespace merkle { throw std::runtime_error("truncated entry bundle"); } - const uint16_t len = + const auto len = (uint16_t)(((uint16_t)bytes[pos] << 8) | bytes[pos + 1]); pos += 2; if (pos + len > bytes.size()) { throw std::runtime_error("truncated entry bundle"); } - out.emplace_back(bytes.begin() + pos, bytes.begin() + pos + len); + out.emplace_back( + bytes.begin() + static_cast(pos), + bytes.begin() + static_cast(pos + len)); pos += len; } return out; @@ -345,9 +347,8 @@ namespace merkle { throw std::runtime_error("cannot open file: " + path.string()); } - return std::vector( - (std::istreambuf_iterator(f)), - std::istreambuf_iterator()); + return { + std::istreambuf_iterator(f), std::istreambuf_iterator()}; } /// @brief Writes a file atomically via a temporary file and rename. @@ -548,6 +549,7 @@ namespace merkle protected: /// @brief The tile store written to. + // NOLINTNEXTLINE(cppcoreguidelines-avoid-const-or-ref-data-members) Store& store; /// @brief Write-path options. @@ -569,7 +571,7 @@ namespace merkle /// size. static uint64_t entries_at_level(uint64_t size, uint8_t level) { - const unsigned shift = 8u * (unsigned)level; + const unsigned shift = 8U * (unsigned)level; return shift >= 64 ? 0 : (size >> shift); } @@ -794,12 +796,13 @@ namespace merkle } protected: + // NOLINTNEXTLINE(cppcoreguidelines-avoid-const-or-ref-data-members) const Store& store; uint64_t available_size; static uint64_t entries_at_level(uint64_t size, uint8_t level) { - const unsigned shift = 8u * (unsigned)level; + const unsigned shift = 8U * (unsigned)level; return shift >= 64 ? 0 : (size >> shift); } }; @@ -986,6 +989,7 @@ namespace merkle } protected: + // NOLINTNEXTLINE(cppcoreguidelines-avoid-const-or-ref-data-members) const Source& source; static bool is_pow2(uint64_t n) @@ -1019,6 +1023,10 @@ namespace merkle bool mth_range(uint64_t a, uint64_t b, Hash& out) const { const uint64_t w = b - a; + if (w == 0) + { + return false; + } if (w == 1) { return source.leaf(a, out); @@ -1106,6 +1114,7 @@ namespace merkle } protected: + // NOLINTNEXTLINE(cppcoreguidelines-avoid-const-or-ref-data-members) Tree& tree; }; @@ -1134,7 +1143,9 @@ namespace merkle } protected: + // NOLINTNEXTLINE(cppcoreguidelines-avoid-const-or-ref-data-members) const Source& primary; + // NOLINTNEXTLINE(cppcoreguidelines-avoid-const-or-ref-data-members) const Source& secondary; }; @@ -1409,6 +1420,7 @@ namespace merkle } protected: + // NOLINTNEXTLINE(cppcoreguidelines-avoid-const-or-ref-data-members) Store& store; Options options; uint64_t next_full = 0; diff --git a/test/tiles_proofs.cpp b/test/tiles_proofs.cpp index 37fdedc..09666c9 100644 --- a/test/tiles_proofs.cpp +++ b/test/tiles_proofs.cpp @@ -49,8 +49,8 @@ static void check_size( } const Hash root = tree.root(); - TileHashSource source(store, n); - ProofEngine engine(source); + const TileHashSource source(store, n); + const ProofEngine engine(source); // Root recomputed from tiles equals the library root. expect(engine.root(n) == root, "root" + at); @@ -66,14 +66,15 @@ static void check_size( } else { - for (uint64_t i : {(uint64_t)0, (uint64_t)1, n / 3, n / 2, n - 2, n - 1}) + for (const uint64_t i : + {(uint64_t)0, (uint64_t)1, n / 3, n / 2, n - 2, n - 1}) { indices.push_back(i); } } // Inclusion proofs are identical to TreeT::path and verify. - for (uint64_t i : indices) + for (const uint64_t i : indices) { if (i >= n) { @@ -98,14 +99,7 @@ static void check_size( } else { - for (auto pr : - {std::make_pair(1, (uint64_t)n), - std::make_pair(n / 2, n), - std::make_pair(n - 1, n), - std::make_pair((uint64_t)1, (uint64_t)2)}) - { - pairs.push_back(pr); - } + pairs = {{1, n}, {n / 2, n}, {n - 1, n}, {1, 2}}; // Tile-boundary crossings. if (n > 256) { @@ -118,7 +112,7 @@ static void check_size( { const uint64_t m = pr.first; const uint64_t k = pr.second; - if (!(m >= 1 && m < k && k <= n)) + if (m == 0 || m >= k || k > n) { continue; } @@ -149,13 +143,13 @@ static void check_size( // Tampering with a proof element or a root is rejected. auto bad = cp; - bad[0].bytes[0] ^= 0xFFu; + bad[0].bytes[0] ^= 0xFFU; expect( !ProofEngine::verify_consistency(m, k, rm, rk, bad), "consistency tamper rejected" + at); Hash wrong = rk; - wrong.bytes[0] ^= 0xFFu; + wrong.bytes[0] ^= 0xFFU; expect( !ProofEngine::verify_consistency(m, k, rm, wrong, cp), "consistency wrong root rejected" + at); @@ -179,7 +173,7 @@ int main() { const auto hashes = make_hashes(70000); - for (uint64_t n : + for (const uint64_t n : {(uint64_t)1, (uint64_t)2, (uint64_t)3, diff --git a/test/tiles_tree.cpp b/test/tiles_tree.cpp index 1e44a0b..023040b 100644 --- a/test/tiles_tree.cpp +++ b/test/tiles_tree.cpp @@ -41,8 +41,8 @@ static void check_memory_source(uint64_t n, const std::vector& hashes) } const Hash root = tree.root(); - MemoryHashSource source(tree); - ProofEngine engine(source); + const MemoryHashSource source(tree); + const ProofEngine engine(source); expect(engine.root(n) == root, "mem root" + at); @@ -56,13 +56,13 @@ static void check_memory_source(uint64_t n, const std::vector& hashes) } else { - for (uint64_t i : {(uint64_t)0, (uint64_t)1, n / 2, n - 1}) + for (const uint64_t i : {(uint64_t)0, (uint64_t)1, n / 2, n - 1}) { indices.push_back(i); } } - for (uint64_t i : indices) + for (const uint64_t i : indices) { const auto p = engine.inclusion_proof(i, n); expect( @@ -120,7 +120,7 @@ int main() const auto hashes = make_hashes(1500); // ---- Part 1: memory-source proofs (exercises TreeT::subtree_root). - for (uint64_t n : + for (const uint64_t n : {(uint64_t)1, (uint64_t)2, (uint64_t)3, @@ -187,7 +187,7 @@ int main() // Indices that are: flushed (tiles only), in the checkpointed-but-resident // overlap, and on the un-checkpointed resident frontier. - for (uint64_t i : + for (const uint64_t i : {(uint64_t)0, (uint64_t)767, (uint64_t)800, @@ -247,7 +247,7 @@ int main() expect(tt.checkpoint_size() == N, "second checkpoint size"); expect(tt.tree_ref().min_index() == 1280, "flushed to 1280"); - for (uint64_t i : + for (const uint64_t i : {(uint64_t)0, (uint64_t)1000, (uint64_t)1279, @@ -320,7 +320,7 @@ int main() expect(rb.root() == exp_root, "rb root matches reference"); // Proofs for a tiled index and a frontier index match the reference. - for (uint64_t i : {(uint64_t)100, (uint64_t)299, (uint64_t)399}) + for (const uint64_t i : {(uint64_t)100, (uint64_t)299, (uint64_t)399}) { const auto p = rb.inclusion_proof(i, 400); expect( diff --git a/test/tiles_writer.cpp b/test/tiles_writer.cpp index 74bfd54..9a0f049 100644 --- a/test/tiles_writer.cpp +++ b/test/tiles_writer.cpp @@ -68,7 +68,7 @@ int main() expect(store.has_full_tile(0, 0), "A L0 full tile"); expect( fs::file_size(store.tile_path(TileRef{0, 0, 0})) == - 256u * Hash().size(), + 256U * Hash().size(), "A L0 full tile size"); expect(!partial_dir_exists(store, 0, 0), "A no L0 partial"); expect(!store.has_full_tile(1, 0), "A no L1 full tile"); @@ -116,7 +116,7 @@ int main() expect(store.has_tile(TileRef{0, 273, 112}), "B L0 partial width 112"); expect( fs::file_size(store.tile_path(TileRef{0, 273, 112})) == - 112u * Hash().size(), + 112U * Hash().size(), "B L0 partial size"); expect(store.has_full_tile(1, 0), "B L1 tile 0"); From 16c38e798cabf21d4fd2bc94ae0820bd7932c2c6 Mon Sep 17 00:00:00 2001 From: Amaury Chamayou Date: Sun, 14 Jun 2026 11:22:31 +0000 Subject: [PATCH 12/30] refactor(tiles): remove checkpoint concept, persist tiles via flush() The tiles feature carried a "checkpoint" concept (a /checkpoint file holding , plus checkpoint()/checkpoint_size()/ compact_on_checkpoint). That concept does not belong in this library, so remove it entirely: - Rename the durable-write operation: checkpoint() -> flush(), checkpoint_size() -> flushed_size(), compact_on_checkpoint -> compact_on_flush. - Remove TileStore::checkpoint_path/write_checkpoint/read_checkpoint, the checkpoint file written by flush(), and the now-unused trim() helper. - Update README, the tiles guide, and the tlog-tiles design doc to match. Tiles are still persisted, now triggered by flush(); 16/16 tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- README.md | 10 ++--- doc/design/tlog-tiles.md | 61 +++++++++++----------------- doc/tiles-guide.md | 35 ++++++++-------- merklecpp_tiles.h | 88 ++++++++-------------------------------- test/tiles_store.cpp | 24 ----------- test/tiles_tree.cpp | 42 +++++++++---------- 6 files changed, 85 insertions(+), 175 deletions(-) diff --git a/README.md b/README.md index 83c5ad6..04f93a2 100644 --- a/README.md +++ b/README.md @@ -35,18 +35,18 @@ byte-identical to one from `merkle::Tree::path()` and verifies with the same #include merkle::tiles::TiledTree::Config cfg; - cfg.prefix = "/var/log/mylog"; // tile files and checkpoint live here + cfg.prefix = "/var/log/mylog"; // tile files live here cfg.retention_margin = 1024; // keep the most recent leaves in memory - cfg.compact_on_checkpoint = true; // opt in to dropping already-tiled leaves + cfg.compact_on_flush = true; // opt in to dropping already-tiled leaves merkle::tiles::TiledTree log(cfg); for (const auto& leaf_hash : batch) log.append(leaf_hash); - // Write newly-complete tiles (and a checkpoint). With compaction enabled + // Write newly-complete tiles. With compaction enabled // this also drops from memory the leaves already covered by a full tile; // otherwise the tree keeps every leaf and you can call log.compact() later. - log.checkpoint(); + log.flush(); // Proofs are served from tiles + the resident tree, even for flushed leaves. auto inclusion = log.inclusion_proof(/*index=*/0, log.size()); @@ -55,7 +55,7 @@ byte-identical to one from `merkle::Tree::path()` and verifies with the same auto consistency = log.consistency_proof(/*m=*/100, /*n=*/log.size()); See the [tiled storage guide](doc/tiles-guide.md) for a how-to covering -checkpoints, compaction, rollback, proofs, and the lower-level building blocks, +flushing, compaction, rollback, proofs, and the lower-level building blocks, and [doc/design/tlog-tiles.md](doc/design/tlog-tiles.md) for the full design, file/directory layout, and the proof algorithms. diff --git a/doc/design/tlog-tiles.md b/doc/design/tlog-tiles.md index cb3422f..8f7180b 100644 --- a/doc/design/tlog-tiles.md +++ b/doc/design/tlog-tiles.md @@ -17,7 +17,7 @@ tiles, from the in-memory tree, or from a combination of the two. the [tlog-tiles](https://c2sp.org/tlog-tiles) file/directory layout and tile geometry (256-wide tiles, 8 tree levels per tile, partial-tile rules, path encoding). -2. Produce tiles **progressively**: each batch/checkpoint writes only the newly +2. Produce tiles **progressively**: each batch/flush writes only the newly completed tiles, and the write integrates with the existing `flush_to()` "compaction" so that flushed (evicted) subtrees are durably persisted before their nodes are freed from memory. @@ -45,8 +45,6 @@ tiles, from the in-memory tree, or from a combination of the two. clients. See [§2.3](#23-compatibility-statement): such interop is only possible if the consumer instantiates `TreeT` with an RFC 6962 combiner, which is **out of scope** here and not required by this design. -- Checkpoint **signing** (signed notes / Ed25519). We define a minimal, - unsigned checkpoint artifact and leave signing to the application. - An HTTP server. We define the on-disk layout and the read/write/proof APIs; serving the static files is an application concern. @@ -79,7 +77,6 @@ A tiled log exposes the Merkle tree as a set of static resources: only the *count* is partial. - **Entry bundles** at `/tile/entries/[.p/]`: big-endian `uint16` length-prefixed raw entries. (See [§6.8](#68-entry-bundles-optional).) -- A **checkpoint** (signed tree head) at `/checkpoint`. - **Pruning**: a log keeps a *minimum index*; tiles/bundles whose end index is `≤ minimum index` may be denied. This maps cleanly onto merklecpp's `flush_to()` / `min_index()`. @@ -186,7 +183,6 @@ Rooted at a configurable `prefix` directory on local disk: ``` / - checkpoint # mutable: latest (unsigned) tile/ 0/ # level 0 (leaf hashes) 000 001 ... 255 # full tiles (8192 B each for HASH_SIZE=32) @@ -228,7 +224,7 @@ Tile byte format: the `W` entries concatenated, each `HASH_SIZE` raw bytes ## 5. Architecture overview ``` - append(leaf hash) checkpoint()/flush + append(leaf hash) flush() caller ───────────────▶ TiledTreeT ───────────────▶ TileStoreT (disk I/O) │ owns Tree ▲ │ │ read tiles @@ -247,11 +243,11 @@ with default aliases mirroring the bottom of `merklecpp.h`: | Component | Responsibility | |----------------------|-----------------------------------------------------------------------| | `TileCoord` | pure index math + path encoding (no I/O) | -| `TileStoreT` | read/write tile & checkpoint files on a local filesystem | +| `TileStoreT` | read/write tile files on a local filesystem | | `TileWriterT` | compute & persist newly-complete full/partial tiles (write path) | | `HashSource` impls | resolve `subtree_root` from tiles, memory, or both | | `ProofEngineT` | `mth_range`, `inclusion_proof`, `consistency_proof`, verifiers | -| `TiledTreeT` | convenience wrapper: `append` / `checkpoint` / `prove*` / compaction | +| `TiledTreeT` | convenience wrapper: `append` / `flush` / `prove*` / compaction | The **only** (optional) change to the core header is a small, read-only, **non-hashing** accessor (`subtree_root`) used by `MemoryHashSource`; see @@ -343,10 +339,6 @@ public: // Single entry convenience (used heavily by TileHashSource) bool read_entry(uint8_t level, uint64_t global_entry, Hash& out, uint64_t size) const; - - // Checkpoint (unsigned: size + root) - void write_checkpoint(uint64_t size, const Hash& root); - bool read_checkpoint(uint64_t& size, Hash& root) const; }; ``` @@ -364,8 +356,8 @@ public: class TileWriterT { public: struct Options { - bool write_partial_tiles = true; // required for checkpoint sizes - bool write_higher_levels = true; // roll up L>=1 each checkpoint + bool write_partial_tiles = true; // required for arbitrary sizes + bool write_higher_levels = true; // roll up L>=1 each flush bool remove_superseded_partials = true; }; @@ -403,14 +395,13 @@ entry(L, g): which exist **before** flushing. Higher levels roll up from the level-0 tiles just written — independent of tree internals and robust to prior flushes. - `first_unwritten_full(L)` is tracked in the store (probe `has_full_tile`, or a - small persisted cursor) to keep each checkpoint O(new tiles). + small persisted cursor) to keep each flush O(new tiles). Compaction integration (the "on compaction" story): ```cpp -// In TiledTreeT::checkpoint(): write durable tiles, THEN free memory. +// In TiledTreeT::flush(): write durable tiles, THEN free memory. writer.write_up_to(size, [&](uint64_t i) -> const Hash& { return tree.leaf(i); }); -store.write_checkpoint(size, tree.root()); uint64_t covered = (size / TILE_WIDTH) * TILE_WIDTH; // full level-0 coverage tree.flush_to(covered - retention_margin); // never past `covered` ``` @@ -510,7 +501,7 @@ public: struct Config { std::filesystem::path prefix; uint64_t retention_margin = 0; // keep this many recent leaves resident - bool compact_on_checkpoint = false; // opt in to dropping tiled leaves + bool compact_on_flush = false; // opt in to dropping tiled leaves TileWriterT::Options writer = {}; }; explicit TiledTreeT(Config); @@ -519,9 +510,9 @@ public: uint64_t size() const; // tree.num_leaves Hash root(); // tree.root - // Write newly-complete tiles + partials + checkpoint. Compaction (dropping - // already-tiled leaves from memory) happens only if compact_on_checkpoint. - Stats checkpoint(); + // Write newly-complete tiles + partials. Compaction (dropping already-tiled + // leaves from memory) happens only if compact_on_flush. + Stats flush(); // Drop from memory the leaves already covered by a full tile (opt-in); // proofs for dropped leaves remain available from the tiles. @@ -549,7 +540,7 @@ entry bundles are an **application-owned** add-on, included for completeness: - The application is responsible for the leaf-hash derivation it uses (e.g. `leaf_hash = H(entry)`); merklecpp stores whatever leaf hash is inserted. - A `BundleWriter` mirrors `TileWriterT` (write full bundles on 256 boundaries, - partial bundle for the checkpoint size). Marked optional/secondary. + partial bundle for the current size). Marked optional/secondary. --- @@ -562,14 +553,13 @@ invariant: > coverage: `min_index() ≤ covered`, where > `covered = floor(size / 256) * 256` after `write_up_to(size, …)`. -Per checkpoint: +Per flush: 1. `append(...)` new leaf hashes; compute `root()`. 2. `write_up_to(size, leaf_at)` — persist newly-complete **full** tiles at all levels (incremental) and the **partial** tiles for `size`. -3. `write_checkpoint(size, root)`. -4. *(optional)* `compact()` → `flush_to(covered − retention_margin)` — reclaim - memory, only when `compact_on_checkpoint` is set (or `compact()` is called +3. *(optional)* `compact()` → `flush_to(covered − retention_margin)` — reclaim + memory, only when `compact_on_flush` is set (or `compact()` is called explicitly). By default nothing is dropped and the tree stays whole. Given the invariant, every leaf and every perfect subtree is resolvable: @@ -584,7 +574,7 @@ Hence inclusion and consistency proofs are always producible after compaction, from tiles alone, from memory alone (when nothing relevant was flushed), or from the combination — satisfying the request. -Cost per checkpoint is `O(new full tiles + partial tiles)`; higher-level tiles +Cost per flush is `O(new full tiles + partial tiles)`; higher-level tiles are cheap roll-ups of 256 child hashes. Proof generation is `O(log(size))` `mth_range` calls, each at most one tile read plus a `≤ 256`-leaf roll-up, served from cache in the common case. @@ -630,8 +620,8 @@ Each phase is independently testable; phases 1–4 require **no** core changes. group in `test/CMakeLists.txt` following `add_merklecpp_test`. **Phase 1 — Coordinates & store.** `TileCoord`/`encode_index`, `TileRef`, -`TileStoreT` (path building, atomic `write_tile`, `read_tile`, checkpoint -read/write). *Tests:* index-encoding vectors; tile byte round-trip. +`TileStoreT` (path building, atomic `write_tile`, `read_tile`). *Tests:* +index-encoding vectors; tile byte round-trip. **Phase 2 — Write path.** `TileWriterT::write_up_to` (level-0 from `leaf_at`, roll-ups, partials, incremental cursor; superseded-partial removal). *Tests:* @@ -646,8 +636,8 @@ equals `tree.past_path(i, m-1)`; consistency proof reconciles `tree.past_root(m-1)`/`tree.past_root(n-1)`. **Phase 4 — Combination & wrapper.** Optional core `subtree_root` accessor; -`MemoryHashSource`, `CombinedHashSource`, `TiledTreeT` (append/checkpoint/prove -with `flush_to`). *Tests:* build N leaves, checkpoint+flush, then prove +`MemoryHashSource`, `CombinedHashSource`, `TiledTreeT` (append/flush/prove +with `flush_to`). *Tests:* build N leaves, flush, then prove inclusion for a **flushed** index and a **resident** index against a non-flushed reference tree's root; consistency across a flush boundary. @@ -695,21 +685,18 @@ compatible": reader; atomic rename + "fetch full tile as fallback" (spec-sanctioned) mitigate this. - **`flush_to` alignment.** Must flush only to a 256-multiple (minus retention) - to uphold the coverage invariant; enforced inside `TiledTreeT::checkpoint`. + to uphold the coverage invariant; enforced inside `TiledTreeT::flush`. - **Rollback vs. immutable tiles.** Tiles are write-once, so rolling the tree back (`retract_to`) over already-tiled entries would leave stale, never- rewritten tiles and is forbidden: `TiledTreeT::retract_to` throws if the - resulting size is below `checkpoint_size()`, permitting only rollback of the - un-tiled (post-checkpoint) frontier. Retracting the underlying tree directly + resulting size is below `flushed_size()`, permitting only rollback of the + un-tiled (post-flush) frontier. Retracting the underlying tree directly via `tree_ref()` bypasses this guard and must be avoided. - **Very large indices.** Index math uses `uint64_t`; encoding handles multi-group indices. Level bound `≤ 63` per spec (8 suffices for `2^64`). - **Open question — `subtree_root` in core vs. `past_path`-derived memory source.** Recommend the tiny non-hashing accessor; falls back to zero-core- change if maintainers prefer. Either keeps hashing untouched. -- **Open question — checkpoint format.** We store unsigned `size + root`. - Aligning the text format with `tlog-checkpoint`/`signed-note` (origin line, - base64 root, signatures) can be layered on without affecting tiles or proofs. --- diff --git a/doc/tiles-guide.md b/doc/tiles-guide.md index 3c089c3..9af2f10 100644 --- a/doc/tiles-guide.md +++ b/doc/tiles-guide.md @@ -15,7 +15,7 @@ page is a practical how-to. - [Requirements and a note on hashing](#requirements-and-a-note-on-hashing) - [Quick start: `TiledTree`](#quick-start-tiledtree) -- [Checkpoints and compaction](#checkpoints-and-compaction) +- [Flushing and compaction](#flushing-and-compaction) - [Rollback](#rollback) - [Proofs](#proofs) - [Lower-level building blocks](#lower-level-building-blocks) @@ -45,14 +45,14 @@ page is a practical how-to. ## Quick start: `TiledTree` -`TiledTree` is the high-level wrapper: append leaf hashes, take checkpoints -(which write tiles), and ask for proofs. +`TiledTree` is the high-level wrapper: append leaf hashes, flush them to disk +(which writes tiles), and ask for proofs. ```cpp #include merkle::tiles::TiledTree::Config cfg; -cfg.prefix = "/var/log/mylog"; // directory for tile files + checkpoint +cfg.prefix = "/var/log/mylog"; // directory for tile files merkle::tiles::TiledTree log(cfg); @@ -60,8 +60,8 @@ merkle::tiles::TiledTree log(cfg); for (const merkle::Hash& leaf : batch) log.append(leaf); -// Persist newly-complete tiles (and a small checkpoint file) to disk. -log.checkpoint(); +// Persist newly-complete tiles to disk. +log.flush(); merkle::Hash root = log.root(); // current Merkle root uint64_t n = log.size(); // number of leaves @@ -74,26 +74,26 @@ assert(inclusion->verify(root)); auto consistency = log.consistency_proof(/*m=*/100, /*n=*/n); ``` -`checkpoint()` is incremental: each call writes only the tiles that became +`flush()` is incremental: each call writes only the tiles that became complete since the previous call, plus the partial tiles for the current size. Full tiles are immutable — written once and never rewritten. -## Checkpoints and compaction +## Flushing and compaction -By default `checkpoint()` only *writes* tiles; it keeps every leaf resident in +By default `flush()` only *writes* tiles; it keeps every leaf resident in memory. Dropping already-tiled leaves from memory ("compaction") is **opt-in**, because once you drop them you can only prove them from the tiles. ```cpp merkle::tiles::TiledTree::Config cfg; -cfg.prefix = "/var/log/mylog"; -cfg.compact_on_checkpoint = true; // drop tiled leaves after each checkpoint -cfg.retention_margin = 4096; // ...but keep the most recent 4096 resident +cfg.prefix = "/var/log/mylog"; +cfg.compact_on_flush = true; // drop tiled leaves after each flush +cfg.retention_margin = 4096; // ...but keep the most recent 4096 resident merkle::tiles::TiledTree log(cfg); ``` -- `compact_on_checkpoint` (default `false`): when set, `checkpoint()` calls +- `compact_on_flush` (default `false`): when set, `flush()` calls `compact()` for you. - `compact()` can also be called explicitly at any time. It drops from memory only the leaves already covered by a **durably written full tile**, keeping @@ -110,14 +110,14 @@ uint64_t resident_from = log.tree_ref().min_index(); ## Rollback Tiles are immutable, so you may only roll back entries that have **not yet been -committed to tiles** (i.e. appended since the last checkpoint). `retract_to` +committed to tiles** (i.e. appended since the last flush). `retract_to` enforces this: ```cpp log.retract_to(index); // keep leaves [0, index], drop the rest ``` -- Allowed when the resulting size is `>= checkpoint_size()` (only the un-tiled +- Allowed when the resulting size is `>= flushed_size()` (only the un-tiled frontier is removed). - Throws otherwise (rolling back committed entries would leave stale tiles). - `retract_to` mirrors `merkle::Tree::retract_to`: `index` is the new *last* @@ -148,10 +148,10 @@ bool ok = p->verify(root_at_size); - `size == log.size()` ⇒ equivalent to `merkle::Tree::path(index)`; verify against `log.root()`. - a past `size` ⇒ equivalent to `merkle::Tree::past_path(index, size - 1)`; - verify against the root at that size (e.g. the root from the checkpoint you are + verify against the root at that size (e.g. a past root you are auditing). -`size` may even exceed `checkpoint_size()`: the recent, not-yet-tiled frontier is +`size` may even exceed `flushed_size()`: the recent, not-yet-tiled frontier is taken from the resident tree while the older part comes from tiles. ### Consistency proofs @@ -243,7 +243,6 @@ Under the configured `prefix`: ``` / - checkpoint # latest tile/0/000, tile/0/001 … # level-0 tiles (leaf hashes), 256 hashes each tile/0/.p/ # partial tile of width W tile/1/… # higher levels (roll-ups of full tiles below) diff --git a/merklecpp_tiles.h b/merklecpp_tiles.h index 60c69ed..c7739fe 100644 --- a/merklecpp_tiles.h +++ b/merklecpp_tiles.h @@ -90,8 +90,7 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) } }; - /// @brief Reads and writes tlog-tiles tile and checkpoint files on a local - /// filesystem. + /// @brief Reads and writes tlog-tiles tile files on a local filesystem. /// @tparam HASH_SIZE Size of each hash in bytes /// @tparam HASH_FUNCTION The tree's node hash function (carried for use by /// later components; tile I/O itself does not hash). @@ -106,7 +105,7 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) using Hash = HashT; /// @brief Constructs a tile store rooted at @p prefix. - /// @param prefix The directory under which tiles and the checkpoint live. + /// @param prefix The directory under which the tiles live. explicit TileStoreT(std::filesystem::path prefix) : prefix(std::move(prefix)) {} @@ -147,12 +146,6 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) return prefix / rel; } - /// @brief The filesystem path of the checkpoint. - [[nodiscard]] std::filesystem::path checkpoint_path() const - { - return prefix / "checkpoint"; - } - /// @brief Whether a full tile exists on disk. [[nodiscard]] bool has_full_tile(uint8_t level, uint64_t index) const { @@ -207,39 +200,6 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) return hashes; } - /// @brief Writes the (unsigned) checkpoint: tree size and root hash. - void write_checkpoint(uint64_t size, const Hash& root_hash) - { - std::string text = - std::to_string(size) + "\n" + root_hash.to_string() + "\n"; - const std::vector bytes(text.begin(), text.end()); - write_file_atomically(checkpoint_path(), bytes); - } - - /// @brief Reads the checkpoint, if present. - /// @param size Set to the checkpoint tree size on success - /// @param root_hash Set to the checkpoint root hash on success - /// @return Whether a checkpoint was read - [[nodiscard]] bool read_checkpoint(uint64_t& size, Hash& root_hash) const - { - std::ifstream f(checkpoint_path(), std::ios::binary); - if (!f.good()) - { - return false; - } - std::string size_line; - std::string root_line; - if (!std::getline(f, size_line) || !std::getline(f, root_line)) - { - throw std::runtime_error("malformed checkpoint"); - } - trim(size_line); - trim(root_line); - size = std::stoull(size_line); - root_hash = Hash(root_line); - return true; - } - /// @brief Whether a full entry bundle exists on disk. [[nodiscard]] bool has_entry_bundle(uint64_t index) const { @@ -328,17 +288,6 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) /// @brief The root directory of the store. std::filesystem::path prefix; - /// @brief Removes trailing carriage-return/whitespace from a line. - static void trim(std::string& s) - { - while (!s.empty() && - (s.back() == '\r' || s.back() == '\n' || s.back() == ' ' || - s.back() == '\t')) - { - s.pop_back(); - } - } - /// @brief Reads an entire file into a byte vector. static std::vector read_file(const std::filesystem::path& path) { @@ -1150,10 +1099,10 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) }; /// @brief A merkle tree backed by tlog-tiles storage. - /// @note Appends grow an in-memory tree; checkpoint() durably writes the + /// @note Appends grow an in-memory tree; flush() durably writes the /// tiles. Compaction (dropping from memory the leaves already covered by a - /// durably-written full tile) is optional: enable it per checkpoint with - /// Config::compact_on_checkpoint, or call compact() explicitly. Proofs are + /// durably-written full tile) is optional: enable it per flush with + /// Config::compact_on_flush, or call compact() explicitly. Proofs are /// served from the combination of the resident tree and the tiles, so they /// remain available for compacted (dropped) leaves. template < @@ -1173,17 +1122,17 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) /// @brief Configuration for a tiled tree. struct Config { - /// @brief Root directory for tiles and the checkpoint. + /// @brief Root directory for the tiles. std::filesystem::path prefix; /// @brief Number of most-recent leaves to keep resident when /// compacting (i.e. never dropped from memory). uint64_t retention_margin = 0; - /// @brief If set, checkpoint() compacts after writing tiles, dropping + /// @brief If set, flush() compacts after writing tiles, dropping /// from memory the leaves already covered by a full tile. Off by /// default: tiles are written but the tree keeps every leaf resident. - bool compact_on_checkpoint = false; + bool compact_on_flush = false; /// @brief Tile writer options. typename Writer::Options writer = {}; @@ -1213,8 +1162,8 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) return tree.root(); } - /// @brief The tree size of the last checkpoint (the durable tile size). - [[nodiscard]] uint64_t checkpoint_size() const + /// @brief The number of leaves durably written to tiles. + [[nodiscard]] uint64_t flushed_size() const { return tiles_size; } @@ -1231,10 +1180,10 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) return store; } - /// @brief Writes tiles and a checkpoint; compacts only if - /// Config::compact_on_checkpoint is set. - /// @return Counts of the tiles written/removed by this checkpoint - Stats checkpoint() + /// @brief Writes newly-complete tiles to disk; compacts only if + /// Config::compact_on_flush is set. + /// @return Counts of the tiles written/removed by this flush + Stats flush() { Stats stats; const uint64_t n = tree.num_leaves(); @@ -1246,10 +1195,9 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) stats = writer.write_up_to(n, [this](uint64_t i) -> const Hash& { return tree.leaf((size_t)i); }); - store.write_checkpoint(n, tree.root()); tiles_size = n; - if (config.compact_on_checkpoint) + if (config.compact_on_flush) { compact(); } @@ -1282,7 +1230,7 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) /// removing all leaves after it (same semantics as TreeT::retract_to). /// @note Tiles are immutable, so entries already committed to tiles /// cannot be rolled back: this throws if the resulting size would be - /// smaller than checkpoint_size(). Only un-checkpointed (not-yet-tiled) + /// smaller than flushed_size(). Only un-flushed (not-yet-tiled) /// entries may be rolled back. (Retracting the underlying tree directly /// via tree_ref() bypasses this guard and can leave stale tiles -- do not /// do that.) @@ -1292,14 +1240,14 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) { throw std::runtime_error( "TiledTree::retract_to: cannot roll back entries already committed " - "to immutable tiles (resulting size < checkpoint size)"); + "to immutable tiles (resulting size < flushed size)"); } tree.retract_to(index); } /// @brief Inclusion proof for @p index in a tree of @p proof_size leaves. /// @note Served from tiles (flushed past) combined with the resident tree - /// (recent frontier); @p proof_size may exceed checkpoint_size(). + /// (recent frontier); @p proof_size may exceed flushed_size(). std::shared_ptr inclusion_proof(uint64_t index, uint64_t proof_size) { MemoryHashSourceT mem(tree); diff --git a/test/tiles_store.cpp b/test/tiles_store.cpp index 388ec88..a4c681e 100644 --- a/test/tiles_store.cpp +++ b/test/tiles_store.cpp @@ -85,8 +85,6 @@ int main() rel(store, store.entries_path(5, 3)), "tile/entries/005.p/3", "entries partial"); - expect_eq( - rel(store, store.checkpoint_path()), "checkpoint", "checkpoint path"); const size_t hsz = Hash().size(); @@ -133,28 +131,6 @@ int main() } expect(threw, "width mismatch rejected"); - // 4. Checkpoint round-trip against a real tree root. - merkle::Tree tree; - for (const auto& h : full) - { - tree.insert(h); - } - const Hash root = tree.root(); - store.write_checkpoint(tree.num_leaves(), root); - - uint64_t rd_size = 0; - Hash rd_root; - expect(store.read_checkpoint(rd_size, rd_root), "checkpoint present"); - expect(rd_size == tree.num_leaves(), "checkpoint size round-trip"); - expect(rd_root == root, "checkpoint root round-trip"); - - // Missing checkpoint reports absence rather than throwing. - const merkle::tiles::TileStore empty_store(dir / "does_not_exist"); - uint64_t s2 = 0; - Hash r2; - expect( - !empty_store.read_checkpoint(s2, r2), "missing checkpoint returns false"); - std::cout << "tiles_store: OK" << '\n'; std::error_code ec; diff --git a/test/tiles_tree.cpp b/test/tiles_tree.cpp index 023040b..297aaf3 100644 --- a/test/tiles_tree.cpp +++ b/test/tiles_tree.cpp @@ -136,8 +136,8 @@ int main() } std::cout << "memory source: OK" << '\n'; - // ---- Part 2: TiledTree checkpoint + flush, proofs over tiles + memory. - const uint64_t n1 = 1000; // first checkpoint size + // ---- Part 2: TiledTree flush, proofs over tiles + memory. + const uint64_t n1 = 1000; // first flush size const uint64_t N = 1500; // final size // Reference: a plain (never flushed) tree with the same leaves. @@ -148,7 +148,7 @@ int main() } const Hash ref_root = ref.root(); - // Default mode: checkpoint() writes tiles but drops nothing from memory; + // Default mode: flush() writes tiles but drops nothing from memory; // an explicit compact() drops only the leaves already covered by a tile. { TiledTree::Config dcfg; @@ -158,7 +158,7 @@ int main() { dtt.append(hashes[i]); } - dtt.checkpoint(); + dtt.flush(); expect(dtt.tree_ref().min_index() == 0, "default: nothing dropped"); dtt.compact(); expect(dtt.tree_ref().min_index() == 768, "compact() drops to 768"); @@ -167,15 +167,15 @@ int main() TiledTree::Config cfg; cfg.prefix = base / "tt"; cfg.retention_margin = 0; - cfg.compact_on_checkpoint = true; + cfg.compact_on_flush = true; TiledTree tt(cfg); for (uint64_t i = 0; i < n1; i++) { tt.append(hashes[i]); } - tt.checkpoint(); // tiles_size = 1000; compact to covered = 768 - expect(tt.checkpoint_size() == n1, "checkpoint size"); + tt.flush(); // tiles_size = 1000; compact to covered = 768 + expect(tt.flushed_size() == n1, "flushed size"); expect(tt.tree_ref().min_index() == 768, "compacted to 768"); for (uint64_t i = n1; i < N; i++) @@ -185,8 +185,8 @@ int main() expect(tt.size() == N, "size after appends"); expect(tt.root() == ref_root, "tiled root == reference root"); - // Indices that are: flushed (tiles only), in the checkpointed-but-resident - // overlap, and on the un-checkpointed resident frontier. + // Indices that are: flushed (tiles only), in the flushed-but-resident + // overlap, and on the un-flushed resident frontier. for (const uint64_t i : {(uint64_t)0, (uint64_t)767, @@ -215,7 +215,7 @@ int main() } expect(threw, "memory-only path throws for flushed index"); - // Consistency across the flush boundary: checkpoint size -> current size, + // Consistency across the flush boundary: flushed size -> current size, // and a flushed-era size -> current size. { const auto cp = tt.consistency_proof(n1, N); @@ -241,10 +241,10 @@ int main() "consistency index variant verifies"); } - // A second checkpoint flushes further; proofs for now-flushed indices still + // A second flush writes further tiles; proofs for now-flushed indices still // work (this also confirms the writer never reads a flushed leaf). - tt.checkpoint(); // tiles_size = 1500; flush to covered = 1280 - expect(tt.checkpoint_size() == N, "second checkpoint size"); + tt.flush(); // tiles_size = 1500; flush to covered = 1280 + expect(tt.flushed_size() == N, "second flushed size"); expect(tt.tree_ref().min_index() == 1280, "flushed to 1280"); for (const uint64_t i : @@ -265,9 +265,9 @@ int main() std::cout << "tiled tree (flush + combination): OK" << '\n'; // ---- Part 3: rollback. Tiles are immutable, so only un-tiled - // (post-checkpoint) entries may be rolled back. + // (post-flush) entries may be rolled back. { - // 3a. Before any checkpoint nothing is tiled, so rollback is + // 3a. Before any flush nothing is tiled, so rollback is // unrestricted. { TiledTree::Config cfg; @@ -278,10 +278,10 @@ int main() rb.append(hashes[i]); } rb.retract_to(29); // tiles_size == 0 -> allowed - expect(rb.size() == 30, "rb pre-checkpoint retract allowed"); + expect(rb.size() == 30, "rb pre-flush retract allowed"); } - // 3b. After a checkpoint, the un-tiled frontier can be rolled back and + // 3b. After a flush, the un-tiled frontier can be rolled back and // the // tiled region stays consistent and provable; committed entries // can't. @@ -293,7 +293,7 @@ int main() { rb.append(hashes[i]); } - rb.checkpoint(); // tiles_size = 300 + rb.flush(); // tiles_size = 300 for (uint64_t i = 300; i < 400; i++) { rb.append(hashes[i]); @@ -304,7 +304,7 @@ int main() { rb.append(hashes[1000 + i]); // re-append DIFFERENT leaves } - rb.checkpoint(); // tiles_size = 400 + rb.flush(); // tiles_size = 400 // Reference tree of the exact post-rollback state. merkle::Tree exp_tree; @@ -367,13 +367,13 @@ int main() { TiledTree::Config cfg; cfg.prefix = base / "rb_compact"; - cfg.compact_on_checkpoint = true; + cfg.compact_on_flush = true; TiledTree rb(cfg); for (uint64_t i = 0; i < 1000; i++) { rb.append(hashes[i]); } - rb.checkpoint(); // tiles_size = 1000; flush to 768 + rb.flush(); // tiles_size = 1000; flush to 768 expect(rb.tree_ref().min_index() == 768, "rb compact flushed to 768"); for (uint64_t i = 1000; i < 1200; i++) { From f2f81d77bdbd1d77e2f8d2eb75b9744120db8579 Mon Sep 17 00:00:00 2001 From: Amaury Chamayou Date: Sun, 14 Jun 2026 11:44:48 +0000 Subject: [PATCH 13/30] test(tiles): add time_tiles benchmark for the tiled-storage path Times the tiles feature end-to-end: append, flush() (writing tiles to disk), inclusion and consistency proof generation from both the resident tree and the on-disk tiles (after compaction), compaction itself, and a plain merkle::Tree::path() baseline for comparison. Registered via add_merklecpp_test; small sizes under a debug build, 1M leaves in release. Sizes keep a partial-tile frontier so compaction never attempts to drop the last leaf. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- test/CMakeLists.txt | 1 + test/time_tiles.cpp | 193 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 194 insertions(+) create mode 100644 test/time_tiles.cpp diff --git a/test/CMakeLists.txt b/test/CMakeLists.txt index 0e889f9..1580beb 100644 --- a/test/CMakeLists.txt +++ b/test/CMakeLists.txt @@ -34,6 +34,7 @@ add_merklecpp_test(tiles_writer tiles_writer.cpp) add_merklecpp_test(tiles_proofs tiles_proofs.cpp) add_merklecpp_test(tiles_tree tiles_tree.cpp) add_merklecpp_test(tiles_entries tiles_entries.cpp) +add_merklecpp_test(time_tiles time_tiles.cpp) if(TARGET evercrypt.host) add_merklecpp_test(compare_evercrypt compare_evercrypt.cpp) diff --git a/test/time_tiles.cpp b/test/time_tiles.cpp new file mode 100644 index 0000000..5ba141b --- /dev/null +++ b/test/time_tiles.cpp @@ -0,0 +1,193 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. + +#include "util.h" + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace fs = std::filesystem; +using merkle::Hash; +using merkle::tiles::TiledTree; + +static double secs_since( + const std::chrono::high_resolution_clock::time_point& start) +{ + const auto stop = std::chrono::high_resolution_clock::now(); + return (double)std::chrono::duration_cast( + stop - start) + .count() / + 1e9; +} + +// Uniform random in [0, bound). +static uint64_t rand_below(uint64_t bound) +{ + if (bound == 0) + { + return 0; + } + return (uint64_t)((std::rand() / (RAND_MAX + 1.0)) * (double)bound); +} + +int main() +{ + const auto seed = std::time(nullptr); + std::srand((unsigned)seed); + std::cout << "seed=" << seed << '\n'; + +#ifndef NDEBUG + const uint64_t num_leaves = 50000; + const uint64_t num_proofs = 100; +#else + const uint64_t num_leaves = 1000000; + const uint64_t num_proofs = 10000; +#endif + + const fs::path dir = fs::temp_directory_path() / + ("merklecpp_time_tiles_" + std::to_string((unsigned long long)seed)); + + // Accumulator that consumes each proof, so the work is not optimised away. + volatile uint64_t sink = 0; + + std::cout << std::fixed << std::setprecision(3); + + try + { + const auto hashes = make_hashes(num_leaves); + + TiledTree::Config cfg; + cfg.prefix = dir; + TiledTree log(cfg); + + // 1. Append: grow the in-memory tree. + auto t = std::chrono::high_resolution_clock::now(); + for (uint64_t i = 0; i < num_leaves; i++) + { + log.append(hashes[i]); + } + log.root(); + const double append_s = secs_since(t); + std::cout << "append : " << num_leaves << " leaves in " + << append_s << " sec (" + << (uint64_t)((double)num_leaves / append_s) << " leaves/sec)\n"; + + // 2. Flush: write newly-complete tiles to disk. + t = std::chrono::high_resolution_clock::now(); + const auto stats = log.flush(); + const double flush_s = secs_since(t); + const uint64_t tiles = stats.full_written + stats.partial_written; + std::cout << "flush (to disk) : " << stats.full_written << " full + " + << stats.partial_written << " partial tiles in " << flush_s + << " sec (" << (uint64_t)((double)tiles / flush_s) + << " tiles/sec)\n"; + + const uint64_t n = log.size(); + + // 3. Inclusion proofs while everything is still resident (memory path). + t = std::chrono::high_resolution_clock::now(); + for (uint64_t k = 0; k < num_proofs; k++) + { + sink += log.inclusion_proof(rand_below(n), n)->size(); + } + const double inc_mem_s = secs_since(t); + std::cout << "inclusion (memory) : " << num_proofs << " proofs in " + << inc_mem_s << " sec (" + << (uint64_t)((double)num_proofs / inc_mem_s) << " proofs/sec)\n"; + + // 4. Consistency proofs while resident (memory path). + t = std::chrono::high_resolution_clock::now(); + for (uint64_t k = 0; k < num_proofs; k++) + { + sink += log.consistency_proof(1 + rand_below(n - 1), n).size(); + } + const double con_mem_s = secs_since(t); + std::cout << "consistency(memory): " << num_proofs << " proofs in " + << con_mem_s << " sec (" + << (uint64_t)((double)num_proofs / con_mem_s) << " proofs/sec)\n"; + + // 5. Compact: drop tiled leaves from memory. + t = std::chrono::high_resolution_clock::now(); + const uint64_t min_idx = log.compact(); + const double compact_s = secs_since(t); + std::cout << "compact : dropped " << min_idx + << " leaves from memory in " << compact_s << " sec\n"; + + // 6. Inclusion proofs for evicted leaves: served from the on-disk tiles. + t = std::chrono::high_resolution_clock::now(); + for (uint64_t k = 0; k < num_proofs; k++) + { + sink += log.inclusion_proof(rand_below(min_idx), n)->size(); + } + const double inc_tile_s = secs_since(t); + std::cout << "inclusion (tiles) : " << num_proofs << " proofs in " + << inc_tile_s << " sec (" + << (uint64_t)((double)num_proofs / inc_tile_s) + << " proofs/sec)\n"; + + // 7. Consistency proofs spanning the tiled (evicted) past. + t = std::chrono::high_resolution_clock::now(); + for (uint64_t k = 0; k < num_proofs; k++) + { + sink += log.consistency_proof(1 + rand_below(min_idx - 1), n).size(); + } + const double con_tile_s = secs_since(t); + std::cout << "consistency(tiles) : " << num_proofs << " proofs in " + << con_tile_s << " sec (" + << (uint64_t)((double)num_proofs / con_tile_s) + << " proofs/sec)\n"; + + // 8. Baseline: identical inclusion proofs from a plain in-memory Tree. + merkle::Tree ref; + for (uint64_t i = 0; i < num_leaves; i++) + { + ref.insert(hashes[i]); + } + ref.root(); + t = std::chrono::high_resolution_clock::now(); + for (uint64_t k = 0; k < num_proofs; k++) + { + sink += ref.path(rand_below(n))->size(); + } + const double inc_ref_s = secs_since(t); + std::cout << "inclusion (Tree) : " << num_proofs << " proofs in " + << inc_ref_s << " sec (" + << (uint64_t)((double)num_proofs / inc_ref_s) << " proofs/sec)\n"; + + // Sanity: a tile-served proof still verifies against the reference root. + if (!log.inclusion_proof(0, n)->verify(ref.root())) + { + throw std::runtime_error("benchmark proof failed to verify"); + } + + std::cout << "time_tiles: OK (checksum " << sink << ")\n"; + + std::error_code ec; + fs::remove_all(dir, ec); + } + catch (std::exception& ex) + { + std::cout << "Error: " << ex.what() << '\n'; + std::error_code ec; + fs::remove_all(dir, ec); + return 1; + } + catch (...) + { + std::cout << "Error" << '\n'; + std::error_code ec; + fs::remove_all(dir, ec); + return 1; + } + + return 0; +} From 8b9cecc7cfd47a3e04bd6b97437c192ee6203c99 Mon Sep 17 00:00:00 2001 From: Amaury Chamayou Date: Sun, 14 Jun 2026 14:43:44 +0000 Subject: [PATCH 14/30] refactor(tiles): drop partial tiles; keep the un-tiled frontier in memory The tiling path now produces and reads only full, balanced (immutable) tiles. The incomplete frontier is never written to disk; it is retained in the in-memory tree and served from there, while full tiles serve the compacted past. This removes the partial-tile lifecycle (write/grow/supersede/remove) entirely from the hash-tile path. - TileWriterT: write full tiles only; drop write_partial_tiles / remove_superseded_partials options and partial_written / partial_removed stats, plus the write_partial/remove_partial_dir bookkeeping. - TileHashSourceT: resolve a subtree only within the full-tile-covered prefix (available_size is floored to a multiple of TILE_WIDTH); when a level has no full tile (it would have been the partial), descend to the highest available full tile and roll up. Never reads a partial tile. - TiledTreeT: flushed_size() is now the full-tile-covered prefix (256*floor(n/256)); compact() always retains the un-tiled frontier and never flushes the whole tree (merklecpp keeps >=1 resident leaf); retract_to() now only protects the immutable full-tile prefix, so the frontier is freely rollback-able. Entry bundles (EntryBundleWriterT) are a separate, application-owned resource and keep their partial-bundle behaviour. Tests updated: tiles_writer (full-tiles-only, asserts no partial dirs), tiles_proofs (combined full-tiles + in-memory frontier source), tiles_tree (flushed_size semantics, rollback boundary at the tiled prefix, exact-multiple compaction regression), time_tiles (drop partial stats). Design doc and tiles guide updated accordingly. All 17 tests pass; clang-format and clang-tidy clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/tlog-tiles.md | 159 +++++++++++++---------- doc/tiles-guide.md | 31 +++-- merklecpp_tiles.h | 271 ++++++++++++++++----------------------- test/tiles_proofs.cpp | 32 ++++- test/tiles_tree.cpp | 69 +++++++--- test/tiles_writer.cpp | 107 ++++++++-------- test/time_tiles.cpp | 9 +- 7 files changed, 359 insertions(+), 319 deletions(-) diff --git a/doc/design/tlog-tiles.md b/doc/design/tlog-tiles.md index 8f7180b..6857809 100644 --- a/doc/design/tlog-tiles.md +++ b/doc/design/tlog-tiles.md @@ -15,18 +15,20 @@ tiles, from the in-memory tree, or from a combination of the two. 1. Write the tree to disk as a set of immutable, cacheable **tile files** using the [tlog-tiles](https://c2sp.org/tlog-tiles) file/directory layout and tile - geometry (256-wide tiles, 8 tree levels per tile, partial-tile rules, path - encoding). + geometry (256-wide tiles, 8 tree levels per tile, path encoding). Only + **full, balanced** tiles are produced: the incomplete frontier is **never + tiled** (no partial tiles are written or read) and is instead kept in the + in-memory tree until it grows into a full tile. 2. Produce tiles **progressively**: each batch/flush writes only the newly completed tiles, and the write integrates with the existing `flush_to()` "compaction" so that flushed (evicted) subtrees are durably persisted before their nodes are freed from memory. 3. Provide an API to retrieve, **after compaction**, an **inclusion path** and a **consistency path**: - - purely from tiles, - - purely from the in-memory tree (existing behaviour), or - - from a combination of tiles (old, flushed part) and the in-memory tree - (recent frontier). + - from the in-memory tree alone (existing behaviour), + - from full tiles alone, for the tiled (full-tile-covered) prefix, or + - from a combination of full tiles (old, flushed part) and the in-memory + tree (recent frontier) — the normal case. 4. Keep everything **header-only** and templated, matching the existing `TreeT` style. @@ -75,6 +77,12 @@ A tiled log exposes the Merkle tree as a set of static resources: size `s` has `floor(s / 256^l) mod 256` entries. **Empty tiles MUST NOT be served.** Partial-tile entries are still complete `256^l`-leaf subtree roots; only the *count* is partial. + + > **merklecpp deviation.** This implementation never writes or reads partial + > *tiles*. The incomplete frontier (everything past the last full tile) is held + > in the in-memory tree, not on disk, so the on-disk tile set is the full-tile + > prefix only. Entry bundles (below) are a separate, application-owned resource + > and may still be partial. - **Entry bundles** at `/tile/entries/[.p/]`: big-endian `uint16` length-prefixed raw entries. (See [§6.8](#68-entry-bundles-optional).) - **Pruning**: a log keeps a *minimum index*; tiles/bundles whose end index is @@ -105,8 +113,9 @@ A tiled log exposes the Merkle tree as a set of static resources: ### 2.3 Compatibility statement -The tile geometry, path encoding, partial-tile rules, and the inclusion / -consistency proof **algorithms** are exactly those of tlog-tiles / RFC 6962. +The tile geometry, path encoding, and the inclusion / consistency proof +**algorithms** are exactly those of tlog-tiles / RFC 6962 (we emit only the +full tiles, never the partial tiles). The **hash values stored in tiles** are produced by the tree's existing `HASH_FUNCTION`. Because @@ -145,20 +154,23 @@ tree's `HASH_FUNCTION`; `MTH` denotes the Merkle Tree Hash computed with it. aligned entries is `perfect_root` of that run. Hence any RFC 6962 subtree root at tree level `k = 8L + r` is obtained from **one** level-`L` tile by hashing `2^r` consecutive entries (a single entry when `r = 0`). -- **Partial tiles.** Number of complete entries available at level `L` for size - `s` is `entries_L = floor(s / 256^L)`. Tile `N` holds entries - `[N·256, (N+1)·256)` clamped to `entries_L`. Width of tile `N`: - `clamp(entries_L − N·256, 0, 256)`. The partial (rightmost) width is - `entries_L mod 256`; width 0 ⇒ not served. +- **Full-tile prefix (no partials).** Only complete level-`L` tiles are written: + `full_tiles_L = floor( floor(s / 256^L) / 256 )`, together covering the leaf + prefix `covered = floor(s / 256) * 256`. The incomplete frontier `[covered, s)` + is not tiled; it is served from the in-memory tree. A subtree that falls in + what would have been a partial tile is resolved by **descending to the highest + available full tile and rolling up** (ultimately from full level-0 tiles). This yields a single read primitive that both proof types are built on: ``` -subtree_root(level k, index j) = MTH(D[j·2^k : (j+1)·2^k]) +subtree_root(level k, index j) = MTH(D[j·2^k : (j+1)·2^k]) # within `covered` L = k / 8 ; r = k % 8 - first = j << r // first level-L entry covered - N = first / 256 ; off = first % 256 // tile and offset within tile - return perfect_root( tile(L, N)[off : off + 2^r] ) // single entry if r==0 + first = j << r ; n = first / 256 ; off = first % 256 + if a full level-L tile n exists (first + 2^r within full_tiles_L · 256): + return perfect_root( tile(L, n)[off : off + 2^r] ) # single entry if r==0 + else: # would-be partial: split & descend + return HASH_FUNCTION( subtree_root(k-1, 2j), subtree_root(k-1, 2j+1) ) ``` and a range primitive for non-perfect (right-frontier) subtrees: @@ -184,11 +196,10 @@ Rooted at a configurable `prefix` directory on local disk: ``` / tile/ - 0/ # level 0 (leaf hashes) + 0/ # level 0 (leaf hashes), full tiles only 000 001 ... 255 # full tiles (8192 B each for HASH_SIZE=32) x001/ 000 ... 255 - .p/ # partial tile, e.g. 273.p/112 1/ # level 1 (roll-ups of level-0 full tiles) 000 ... .../ @@ -212,8 +223,10 @@ Examples: `5 → "005"`, `255 → "255"`, `1000 → "x001/000"`, Resource paths: -- Full tile: `tile//` -- Partial tile: `tile//.p/` +- Full tile: `tile//` (the only tiles this + implementation writes) +- Partial tile (tlog-tiles format; **not produced or read here**): + `tile//.p/` - Entry bundle: `tile/entries/[.p/]` Tile byte format: the `W` entries concatenated, each `HASH_SIZE` raw bytes @@ -244,7 +257,7 @@ with default aliases mirroring the bottom of `merklecpp.h`: |----------------------|-----------------------------------------------------------------------| | `TileCoord` | pure index math + path encoding (no I/O) | | `TileStoreT` | read/write tile files on a local filesystem | -| `TileWriterT` | compute & persist newly-complete full/partial tiles (write path) | +| `TileWriterT` | compute & persist newly-complete full tiles (write path) | | `HashSource` impls | resolve `subtree_root` from tiles, memory, or both | | `ProofEngineT` | `mth_range`, `inclusion_proof`, `consistency_proof`, verifiers | | `TiledTreeT` | convenience wrapper: `append` / `flush` / `prove*` / compaction | @@ -344,9 +357,8 @@ public: - Writes are **atomic** (write to a temp file, `rename`) so a crash never leaves a half-written immutable tile. -- Full tiles are written once and never rewritten (immutability). Partial tiles - are overwritten as the frontier advances and removed once the covering full - tile exists (spec-permitted; configurable). +- Full tiles are written once and never rewritten (immutability). No partial + tiles are produced, so every file under `tile//` is write-once. - An in-process LRU cache of recently read tiles avoids repeated I/O during proof generation. @@ -356,15 +368,13 @@ public: class TileWriterT { public: struct Options { - bool write_partial_tiles = true; // required for arbitrary sizes bool write_higher_levels = true; // roll up L>=1 each flush - bool remove_superseded_partials = true; }; TileWriterT(TileStoreT& store, Options = {}); - // Persist all newly-complete full tiles (all levels) and, for `size`, - // the partial tiles. Incremental: only writes tiles not already present. + // Persist all newly-complete full tiles (all levels). No partial tiles are + // written. Incremental: only writes tiles not already present. // `leaf_at` supplies level-0 leaf hashes for [0, size) (e.g. Tree::leaf). void write_up_to(uint64_t size, const std::function& leaf_at); @@ -379,12 +389,8 @@ for L = 0, 1, 2, ... while entries_L = size >> (8*L) is > 0: for N in [first_unwritten_full(L) .. full_L): # new full tiles only entries = [ entry(L, N*256 + i) for i in 0..255 ] store.write_tile({L, N, width=0}, entries) - if write_partial_tiles: - w = entries_L % 256 - if w > 0: - entries = [ entry(L, full_L*256 + i) for i in 0..w-1 ] - store.write_tile({L, full_L, width=w}, entries) - if remove_superseded_partials: remove stale partials < full_L + # the rightmost, incomplete (partial) entries are NOT written; they stay in + # the in-memory tree until they complete a full tile entry(L, g): if L == 0: return leaf_at(g) @@ -400,10 +406,11 @@ entry(L, g): Compaction integration (the "on compaction" story): ```cpp -// In TiledTreeT::flush(): write durable tiles, THEN free memory. +// In TiledTreeT::flush(): write durable full tiles, THEN free memory. writer.write_up_to(size, [&](uint64_t i) -> const Hash& { return tree.leaf(i); }); uint64_t covered = (size / TILE_WIDTH) * TILE_WIDTH; // full level-0 coverage -tree.flush_to(covered - retention_margin); // never past `covered` +flushed_size = covered; // only full tiles durable +// compact() drops only [min_index, covered); the frontier [covered, size) stays. ``` ### 6.5 `HashSource` — tiles, memory, or both @@ -417,9 +424,10 @@ struct HashSource { // concept (duck-typed or virtual) }; ``` -- `TileHashSource{store, size}` — resolves from tiles using the - `subtree_root` formula in [§3](#3-tile--merklecpp-mapping-the-math); returns - `false` when the requested subtree extends beyond what tiles cover for `size`. +- `TileHashSource{store, size}` — resolves from **full tiles** using the + `subtree_root` formula in [§3](#3-tile--merklecpp-mapping-the-math) (`size` is + rounded down to a whole number of full tiles); returns `false` when the + requested subtree reaches into the un-tiled frontier. - `MemoryHashSource{tree}` — `tree.subtree_root(level,index,out)` (or the `past_path`-derived fallback); resolves only resident, full subtrees (`≥ min_index`). @@ -510,12 +518,13 @@ public: uint64_t size() const; // tree.num_leaves Hash root(); // tree.root - // Write newly-complete tiles + partials. Compaction (dropping already-tiled + // Write newly-complete full tiles. Compaction (dropping already-tiled // leaves from memory) happens only if compact_on_flush. Stats flush(); - // Drop from memory the leaves already covered by a full tile (opt-in); - // proofs for dropped leaves remain available from the tiles. + // Drop from memory the leaves already covered by a full tile (opt-in); the + // un-tiled frontier is always retained, and proofs for dropped leaves remain + // available from the tiles. uint64_t compact(); // Proofs over tiles ∪ resident tree (works for flushed indices). @@ -557,27 +566,34 @@ Per flush: 1. `append(...)` new leaf hashes; compute `root()`. 2. `write_up_to(size, leaf_at)` — persist newly-complete **full** tiles at all - levels (incremental) and the **partial** tiles for `size`. + levels (incremental). The incomplete frontier is **not** tiled, so + `flushed_size = covered = floor(size / 256) * 256`. 3. *(optional)* `compact()` → `flush_to(covered − retention_margin)` — reclaim memory, only when `compact_on_flush` is set (or `compact()` is called - explicitly). By default nothing is dropped and the tree stays whole. + explicitly). It drops only the full-tile-covered prefix, so the un-tiled + frontier always stays resident. By default nothing is dropped and the tree + stays whole. Given the invariant, every leaf and every perfect subtree is resolvable: - leaf `i < covered` ⇒ in a full level-0 tile; leaf `i ≥ min_index` ⇒ resident. - Since `min_index ≤ covered`, **every** leaf is in tiles ∪ memory. + Since `min_index ≤ covered`, **every** leaf is in tiles ∪ memory. The frontier + `[covered, size)` is always resident, because `compact()` never flushes past + `covered`. - `mth_range` resolves a perfect subtree directly when it lies wholly in tiles (`end ≤ covered`) or wholly in memory (`start ≥ min_index`); otherwise it - splits and recurses, terminating at resolvable pieces (leaves at worst). + splits and recurses, terminating at resolvable pieces (leaves at worst). A + subtree within `covered` whose level has no full tile (it would have been the + partial) is resolved by descending to the highest available full tile. Hence inclusion and consistency proofs are always producible after compaction, -from tiles alone, from memory alone (when nothing relevant was flushed), or from -the combination — satisfying the request. +from full tiles alone (for the tiled prefix), from memory alone (when nothing +relevant was flushed), or from the combination — satisfying the request. -Cost per flush is `O(new full tiles + partial tiles)`; higher-level tiles -are cheap roll-ups of 256 child hashes. Proof generation is -`O(log(size))` `mth_range` calls, each at most one tile read plus a `≤ 256`-leaf -roll-up, served from cache in the common case. +Cost per flush is `O(new full tiles)`; higher-level tiles are cheap roll-ups of +256 child hashes. Proof generation is `O(log(size))` `mth_range` calls, each at +most a few tile reads plus a `≤ 256`-leaf roll-up, served from cache in the +common case. --- @@ -598,14 +614,14 @@ to the application, exactly as the spec leaves it to log ecosystems. ## 9. Worked examples (used as test vectors) -**Size 256:** one full level-0 tile (`tile/0/000`) and one partial level-1 tile -of width 1 (`tile/1/000.p/1`). No partial level-0 tile (width `256 mod 256 = 0`, -not served). +**Size 256:** exactly one full level-0 tile (`tile/0/000`). No level-1 tile (its +single entry is the un-tiled frontier root, kept in memory) and no partials. -**Size 70 000:** 273 full level-0 tiles + one partial level-0 of width 112 -(`70000 mod 256`); one full level-1 tile + one partial level-1 of width 17 -(`273 mod 256`); one partial level-2 of width 1 (`floor(70000/65536) = 1`). These -exact counts are asserted in tests. +**Size 70 000:** 273 full level-0 tiles (`covered = 69 888`) + one full level-1 +tile (`tile/1/000`, rolling up level-0 tiles 0..255) = **274 full tiles**. The +incomplete frontier at every level (the would-be partials) is not written; the +remaining 112 leaves and the higher-level roll-ups stay in the in-memory tree. +These exact counts are asserted in tests. **Index encoding:** `1234067 → x001/x234/067`; `1000 → x001/000`; `255 → 255`. @@ -624,9 +640,9 @@ group in `test/CMakeLists.txt` following `add_merklecpp_test`. index-encoding vectors; tile byte round-trip. **Phase 2 — Write path.** `TileWriterT::write_up_to` (level-0 from `leaf_at`, -roll-ups, partials, incremental cursor; superseded-partial removal). *Tests:* -size 256 & 70 000 produce exactly the expected tile set/widths; full-tile -immutability (re-running writes nothing new). +roll-ups, incremental cursor; full tiles only — the frontier is never tiled). +*Tests:* sizes 256 & 70 000 produce exactly the expected full-tile set with no +partial directories; full-tile immutability (re-running writes nothing new). **Phase 3 — Hash sources & proof engine.** `TileHashSource`, `mth_range`, `ProofEngineT::root/inclusion_proof/consistency_proof/verify_consistency`. @@ -659,8 +675,8 @@ compatible": `past_path`. This is stronger than `verify()` and pins byte-compatibility. - **Root agreement.** `ProofEngineT::root(size)` == `tree.root()` / `tree.past_root(size-1)` for many random sizes. -- **Spec vectors.** Exact tile counts/widths for sizes 256 and 70 000; the empty - partial-tile rule; index-encoding strings (incl. `x001/x234/067`). +- **Spec vectors.** Exact full-tile counts for sizes 256 and 70 000; the absence + of any partial-tile directory; index-encoding strings (incl. `x001/x234/067`). - **Flush/compaction.** Cross-check proofs for flushed indices against a twin tree that was never flushed (same inserts) — roots must match and proofs must verify. @@ -681,11 +697,14 @@ compatible": the consumer's choice and out of scope. - **Filesystem dependency.** Tile I/O needs ``/``; isolated in the companion header so the core stays dependency-free. -- **Partial-tile lifecycle.** Overwriting/removing partials must never race a - reader; atomic rename + "fetch full tile as fallback" (spec-sanctioned) - mitigate this. -- **`flush_to` alignment.** Must flush only to a 256-multiple (minus retention) - to uphold the coverage invariant; enforced inside `TiledTreeT::flush`. +- **No partial tiles.** The incomplete frontier is never tiled, so there are no + partial-tile rewrites to race a reader; every tile is write-once and immutable. + The cost is that a stand-alone tile reader cannot serve the frontier — that is + the in-memory tree's job (or the application must keep it elsewhere). +- **`flush_to` alignment.** Compaction flushes only to a 256-multiple (minus + retention) to uphold the coverage invariant; it also keeps at least one + resident leaf, since `flush_to` cannot drain the whole tree. Enforced inside + `TiledTreeT::compact`. - **Rollback vs. immutable tiles.** Tiles are write-once, so rolling the tree back (`retract_to`) over already-tiled entries would leave stale, never- rewritten tiles and is forbidden: `TiledTreeT::retract_to` throws if the diff --git a/doc/tiles-guide.md b/doc/tiles-guide.md index 9af2f10..29a1dfa 100644 --- a/doc/tiles-guide.md +++ b/doc/tiles-guide.md @@ -74,9 +74,10 @@ assert(inclusion->verify(root)); auto consistency = log.consistency_proof(/*m=*/100, /*n=*/n); ``` -`flush()` is incremental: each call writes only the tiles that became -complete since the previous call, plus the partial tiles for the current size. -Full tiles are immutable — written once and never rewritten. +`flush()` is incremental: each call writes only the full tiles that became +complete since the previous call. Full tiles are immutable — written once and +never rewritten — and the incomplete frontier is never tiled (it stays in +memory until it grows into a full tile). ## Flushing and compaction @@ -117,8 +118,10 @@ enforces this: log.retract_to(index); // keep leaves [0, index], drop the rest ``` -- Allowed when the resulting size is `>= flushed_size()` (only the un-tiled - frontier is removed). +- Allowed when the resulting size is `>= flushed_size()` — i.e. only the + un-tiled frontier is removed. `flushed_size()` is the full-tile-covered prefix + (a multiple of 256), so everything appended since the last full tile can be + rolled back. - Throws otherwise (rolling back committed entries would leave stale tiles). - `retract_to` mirrors `merkle::Tree::retract_to`: `index` is the new *last* leaf, so the resulting size is `index + 1`. @@ -191,18 +194,19 @@ for (auto& leaf : batch) tree.insert(leaf); merkle::tiles::TileStore store("/var/log/mylog"); merkle::tiles::TileWriter writer(store); -// Write all newly-complete full tiles and the partial tiles for this size. +// Write all newly-complete full tiles (the incomplete frontier is not tiled). auto stats = writer.write_up_to( tree.num_leaves(), [&](uint64_t i) -> const merkle::Hash& { return tree.leaf(i); }); -// stats.full_written / stats.partial_written / stats.partial_removed +// stats.full_written ``` ### Reading tiles and computing proofs A `HashSource` resolves the root of a complete subtree; pick where it reads from: -- `TileHashSource(store, available_size)` — from tile files. +- `TileHashSource(store, available_size)` — from full tile files; resolves the + full-tile-covered prefix only (the frontier needs a memory source). - `MemoryHashSource(tree)` — from a resident `merkle::Tree`. - `CombinedHashSource(primary, secondary)` — try `primary` first, then `secondary` (e.g. memory then tiles). @@ -216,6 +220,11 @@ auto inclusion = engine.inclusion_proof(index, size); auto consistency = engine.consistency_proof(m, n); ``` +A tile-only source can resolve proofs whose subtrees all lie within the +full-tile-covered prefix (`available_size` is rounded down to a whole number of +tiles). For the live frontier, combine it with a `MemoryHashSource` — which is +exactly what `TiledTree` does for you. + `TiledTree` simply wires a `CombinedHashSource(MemoryHashSource, TileHashSource)` into a `ProofEngine` for you. @@ -244,13 +253,13 @@ Under the configured `prefix`: ``` / tile/0/000, tile/0/001 … # level-0 tiles (leaf hashes), 256 hashes each - tile/0/.p/ # partial tile of width W tile/1/… # higher levels (roll-ups of full tiles below) tile/entries/… # optional raw entry bundles ``` Tile indices use the tlog-tiles path encoding: zero-padded 3-digit groups with -all but the last prefixed by `x` (e.g. index `1234067` → `x001/x234/067`). Full -tiles are immutable; partial tiles grow and may be removed once superseded. See +all but the last prefixed by `x` (e.g. index `1234067` → `x001/x234/067`). All +tiles are full (256-wide) and immutable; the incomplete frontier is never tiled. +See [`design/tlog-tiles.md`](design/tlog-tiles.md) for the full specification of the geometry and proof algorithms. diff --git a/merklecpp_tiles.h b/merklecpp_tiles.h index c7739fe..9009152 100644 --- a/merklecpp_tiles.h +++ b/merklecpp_tiles.h @@ -381,11 +381,11 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) /// @tparam HASH_SIZE Size of each hash in bytes /// @tparam HASH_FUNCTION The tree's node hash function /// @note Only balanced subtrees are tiled: a level-L entry is the root of a - /// complete 2**(8L)-leaf subtree. Full tiles (256 such entries) are - /// therefore immutable and written exactly once; the incomplete frontier is - /// never tiled, and partial tiles (complete entries, partial count) are the - /// only resources that may be re-written (grown) or removed once superseded - /// by a full tile. + /// complete 2**(8L)-leaf subtree. Only full tiles (256 such entries) are + /// written; they are therefore immutable and written exactly once. The + /// incomplete frontier is deliberately never tiled (no partial tiles are + /// produced): it is retained in memory by the owning tree until it grows + /// into a full tile. template < size_t HASH_SIZE, void HASH_FUNCTION( @@ -405,15 +405,8 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) /// @brief Write-path options. struct Options { - /// @brief Write the partial (rightmost) tiles for the current size. - bool write_partial_tiles = true; - /// @brief Roll up and write tiles at levels >= 1. bool write_higher_levels = true; - - /// @brief Remove partial tiles once superseded by a full tile (or a - /// wider partial at the same index). - bool remove_superseded_partials = true; }; /// @brief Counts of work performed by a write_up_to call. @@ -421,12 +414,6 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) { /// @brief Number of full tiles written. uint64_t full_written = 0; - - /// @brief Number of partial tiles written. - uint64_t partial_written = 0; - - /// @brief Number of superseded partial tiles removed. - uint64_t partial_removed = 0; }; /// @brief Constructs a writer over @p store. @@ -434,14 +421,14 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) store(store), options(options) {} - /// @brief Writes all newly-complete full tiles and the partial tiles for - /// a tree of @p size leaves. + /// @brief Writes all newly-complete full tiles for a tree of @p size + /// leaves. /// @param size The current tree size /// @param leaf_at Returns the level-0 leaf hash for a leaf index in /// [0, size); only ever queried for leaves of complete subtrees. - /// @return Counts of tiles written/removed + /// @return Counts of tiles written /// @note Incremental: full tiles already on disk are immutable and are - /// never rewritten. + /// never rewritten. The incomplete frontier is never tiled. Stats write_up_to(uint64_t size, const LeafFn& leaf_at) { Stats stats; @@ -481,16 +468,6 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) { next_full[level] = full_tiles; } - - if (options.write_partial_tiles) - { - write_partial( - level, - full_tiles, - (uint16_t)(entries % TILE_WIDTH), - leaf_at, - stats); - } } return stats; @@ -510,12 +487,6 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) /// @brief Per-level flag indicating next_full has been initialised. std::vector cursor_inited; - /// @brief Per-level last partial tile written by this writer. - std::vector last_partial; - - /// @brief Per-level flag indicating last_partial is valid. - std::vector has_last_partial; - /// @brief Number of complete level-@p level entries for a tree of @p /// size. static uint64_t entries_at_level(uint64_t size, uint8_t level) @@ -532,8 +503,6 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) { next_full.resize(needed, 0); cursor_inited.resize(needed, false); - last_partial.resize(needed, TileRef{}); - has_last_partial.resize(needed, false); } } @@ -592,67 +561,6 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) } return out; } - - /// @brief Writes (or removes) the partial tile at the rightmost index. - void write_partial( - uint8_t level, - uint64_t index, - uint16_t width, - const LeafFn& leaf_at, - Stats& stats) - { - if (width == 0) - { - // No partial at this size; a previous partial (if any) is now covered - // by a full tile and may be removed. - if (options.remove_superseded_partials && has_last_partial[level]) - { - remove_partial_dir(level, last_partial[level].index); - has_last_partial[level] = false; - stats.partial_removed++; - } - return; - } - - const TileRef ref{level, index, width}; - - if ( - has_last_partial[level] && last_partial[level].index == index && - last_partial[level].width == width) - { - return; // identical partial already written by this writer - } - - if (options.remove_superseded_partials && has_last_partial[level]) - { - if (last_partial[level].index != index) - { - // Previous partial's index is now covered by a full tile. - remove_partial_dir(level, last_partial[level].index); - stats.partial_removed++; - } - else - { - // Same index, narrower width: drop the stale width file(s). - remove_partial_dir(level, index); - } - } - - store.write_tile( - ref, collect(level, index * TILE_WIDTH, width, leaf_at)); - stats.partial_written++; - last_partial[level] = ref; - has_last_partial[level] = true; - } - - /// @brief Removes the partial-tile directory tile//.p. - void remove_partial_dir(uint8_t level, uint64_t index) - { - const std::filesystem::path dir = - store.tile_path(TileRef{level, index, 1}).parent_path(); - std::error_code ec; - std::filesystem::remove_all(dir, ec); - } }; /// @brief Abstract source of Merkle subtree roots for proof generation. @@ -685,10 +593,11 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) }; /// @brief Resolves subtree roots from tlog-tiles tile files. - /// @note @p available_size is the largest tree size whose tiles are durably - /// written; any complete subtree within it is resolvable. A missing tile - /// (e.g. a frontier partial that was not written) yields false so that a - /// proof builder can fall back to another source. + /// @note @p available_size is rounded down to a whole number of full tiles: + /// only complete, durably-written full tiles are read (no partial tiles). + /// A complete subtree within that full-tile prefix is resolvable; anything + /// reaching into the incomplete frontier yields false so that a proof + /// builder can fall back to another source (e.g. an in-memory tree). template < size_t HASH_SIZE, void HASH_FUNCTION( @@ -700,59 +609,83 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) using Store = TileStoreT; /// @brief Constructs a source over @p store for trees up to - /// @p available_size leaves. + /// @p available_size leaves. @p available_size is rounded down to a whole + /// number of full tiles, since only full tiles are durable. TileHashSourceT(const Store& store, uint64_t available_size) : - store(store), available_size(available_size) + store(store), available_size((available_size / TILE_WIDTH) * TILE_WIDTH) {} bool subtree_root(uint8_t level, uint64_t index, Hash& out) const override { - const uint8_t L = level / TILE_HEIGHT; - const uint8_t r = level % TILE_HEIGHT; - const uint64_t span = (uint64_t)1 << r; // entries spanned in level L - const uint64_t first = index << r; // first level-L entry index - - const uint64_t entries = entries_at_level(available_size, L); - if (first + span > entries) - { - return false; // extends into the incomplete frontier - } - - const uint64_t n = first / TILE_WIDTH; - const uint64_t off = first % TILE_WIDTH; - - // Choose the full or current partial tile holding these entries. - TileRef ref{L, n, 0}; - if ((n + 1) * TILE_WIDTH > entries) - { - ref.width = (uint16_t)(entries - n * TILE_WIDTH); - } - if (!store.has_tile(ref)) + // The subtree covers leaves [index << level, (index + 1) << level). It + // is resolvable only when it lies entirely within the full-tile-covered + // prefix; the incomplete frontier is served from another source. + if (level >= 64 || index >= (available_size >> level)) { return false; } - - const std::vector tile = store.read_tile(ref); - if (span == 1) - { - out = tile.at(off); - return true; - } - const std::vector sub( - tile.begin() + off, tile.begin() + off + span); - out = perfect_root(sub); + resolve(level, index, out); return true; } protected: // NOLINTNEXTLINE(cppcoreguidelines-avoid-const-or-ref-data-members) const Store& store; - uint64_t available_size; + uint64_t available_size; // full-tile prefix length (a multiple of WIDTH) - static uint64_t entries_at_level(uint64_t size, uint8_t level) + /// @brief Combines the @p span entries at @p off of @p tile into a root. + static Hash roll_up( + const std::vector& tile, uint64_t off, uint64_t span) { - const unsigned shift = 8U * (unsigned)level; - return shift >= 64 ? 0 : (size >> shift); + if (span == 1) + { + return tile.at(off); + } + return perfect_root(std::vector( + tile.begin() + (std::ptrdiff_t)off, + tile.begin() + (std::ptrdiff_t)(off + span))); + } + + /// @brief Resolves a complete subtree known to lie within the full-tile + /// prefix, reading the highest-level full tile that holds it (and rolling + /// up); descends to lower full tiles where a higher-level tile is not + /// full (it would otherwise have been the unwritten partial). Terminates + /// because full level-0 tiles always cover the prefix. + void resolve(uint8_t level, uint64_t index, Hash& out) const + { + if (level <= TILE_HEIGHT) + { + // Spans 2**level <= TILE_WIDTH leaves: held by one level-0 tile. + const uint64_t span = (uint64_t)1 << level; + const uint64_t start = index << level; + const std::vector tile = + store.read_tile(TileRef{0, start / TILE_WIDTH, 0}); + out = roll_up(tile, start % TILE_WIDTH, span); + return; + } + + const uint8_t L = level / TILE_HEIGHT; + const uint8_t r = level % TILE_HEIGHT; + const uint64_t first = index << r; // first level-L entry + const uint64_t n = first / TILE_WIDTH; // level-L tile index + const unsigned full_shift = 8U * ((unsigned)L + 1U); + const uint64_t full_tiles = + full_shift >= 64 ? 0 : (available_size >> full_shift); + + if (n < full_tiles) + { + // One full level-L tile holds all 2**r entries of this subtree. + const std::vector tile = store.read_tile(TileRef{L, n, 0}); + out = roll_up(tile, first % TILE_WIDTH, (uint64_t)1 << r); + return; + } + + // No full level-L tile here: split into two level-(level-1) subtrees. + Hash lo; + Hash hi; + resolve((uint8_t)(level - 1), index * 2, lo); + resolve((uint8_t)(level - 1), index * 2 + 1, hi); + HASH_FUNCTION(lo, hi, out); } }; @@ -1099,12 +1032,13 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) }; /// @brief A merkle tree backed by tlog-tiles storage. - /// @note Appends grow an in-memory tree; flush() durably writes the - /// tiles. Compaction (dropping from memory the leaves already covered by a - /// durably-written full tile) is optional: enable it per flush with - /// Config::compact_on_flush, or call compact() explicitly. Proofs are - /// served from the combination of the resident tree and the tiles, so they - /// remain available for compacted (dropped) leaves. + /// @note Appends grow an in-memory tree; flush() durably writes only full + /// (balanced) tiles, so the incomplete frontier is never tiled and stays + /// resident in memory. Compaction (dropping from memory the leaves already + /// covered by a full tile) is optional: enable it per flush with + /// Config::compact_on_flush, or call compact() explicitly; it never drops + /// the un-tiled frontier. Proofs are served from the combination of the + /// resident tree (frontier) and the full tiles (compacted past). template < size_t HASH_SIZE, void HASH_FUNCTION( @@ -1162,7 +1096,8 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) return tree.root(); } - /// @brief The number of leaves durably written to tiles. + /// @brief The number of leaves durably written to full tiles (always a + /// multiple of TILE_WIDTH; the incomplete frontier is not tiled). [[nodiscard]] uint64_t flushed_size() const { return tiles_size; @@ -1180,9 +1115,12 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) return store; } - /// @brief Writes newly-complete tiles to disk; compacts only if + /// @brief Writes newly-complete full tiles to disk; compacts only if /// Config::compact_on_flush is set. - /// @return Counts of the tiles written/removed by this flush + /// @return Counts of the full tiles written by this flush + /// @note Only full (balanced) tiles are written, so flushed_size() + /// advances to the largest multiple of TILE_WIDTH <= size(); the + /// incomplete frontier stays resident in memory. Stats flush() { Stats stats; @@ -1195,7 +1133,7 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) stats = writer.write_up_to(n, [this](uint64_t i) -> const Hash& { return tree.leaf((size_t)i); }); - tiles_size = n; + tiles_size = (n / TILE_WIDTH) * TILE_WIDTH; if (config.compact_on_flush) { @@ -1207,18 +1145,26 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) /// @brief Drops from the in-memory tree every leaf already covered by a /// durably-written full tile, keeping retention_margin recent leaves. /// @return The new minimum (smallest still-resident) leaf index - /// @note Only leaves whose full tile already exists are dropped, so - /// inclusion and consistency proofs remain available (served from the - /// tiles). Has no effect until tiling has produced full tiles. The flush - /// target is a multiple of TILE_WIDTH, so the coverage invariant - /// (min_index <= durable full-tile coverage) always holds. + /// @note Only leaves covered by a full tile are dropped, so the un-tiled + /// frontier is always retained in memory and inclusion/consistency proofs + /// remain available (the past from tiles, the frontier from memory). Has + /// no effect until tiling has produced full tiles. uint64_t compact() { + const uint64_t n = tree.num_leaves(); const uint64_t covered = (tiles_size / TILE_WIDTH) * TILE_WIDTH; uint64_t target = covered > config.retention_margin ? covered - config.retention_margin : 0; target = (target / TILE_WIDTH) * TILE_WIDTH; + // merklecpp's flush_to keeps at least one resident leaf, so never flush + // the whole tree (this can only arise when the size is an exact + // multiple of TILE_WIDTH and nothing is retained); the extra resident + // leaf is harmless (it is also covered by a full tile). + if (n > 0 && target >= n) + { + target = n - 1; + } if (target > tree.min_index()) { tree.flush_to((size_t)target); @@ -1228,12 +1174,11 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) /// @brief Rolls the tree back so that @p index becomes the last leaf, /// removing all leaves after it (same semantics as TreeT::retract_to). - /// @note Tiles are immutable, so entries already committed to tiles - /// cannot be rolled back: this throws if the resulting size would be - /// smaller than flushed_size(). Only un-flushed (not-yet-tiled) - /// entries may be rolled back. (Retracting the underlying tree directly - /// via tree_ref() bypasses this guard and can leave stale tiles -- do not - /// do that.) + /// @note Only full tiles are immutable: this throws if the resulting size + /// would be smaller than flushed_size() (the full-tile prefix). The + /// un-tiled frontier (>= flushed_size()) may be freely rolled back. + /// (Retracting the underlying tree directly via tree_ref() bypasses this + /// guard and can leave stale tiles -- do not do that.) void retract_to(size_t index) { if ((uint64_t)index + 1 < tiles_size) diff --git a/test/tiles_proofs.cpp b/test/tiles_proofs.cpp index 09666c9..6d0733a 100644 --- a/test/tiles_proofs.cpp +++ b/test/tiles_proofs.cpp @@ -17,6 +17,8 @@ namespace fs = std::filesystem; using merkle::Hash; +using merkle::tiles::CombinedHashSource; +using merkle::tiles::MemoryHashSource; using merkle::tiles::ProofEngine; using merkle::tiles::TileHashSource; using merkle::tiles::TileStore; @@ -31,7 +33,9 @@ static void expect(bool cond, const std::string& what) } // Exercises tile-derived proofs for a tree of `n` leaves against the existing -// library (which acts as the oracle: proofs must be byte-identical). +// library (which acts as the oracle: proofs must be byte-identical). Full tiles +// serve the covered prefix and an in-memory tree serves the un-tiled frontier, +// exactly as TiledTree combines them. static void check_size( const fs::path& dir, uint64_t n, const std::vector& hashes) { @@ -42,6 +46,7 @@ static void check_size( const auto leaf_at = [&](uint64_t i) -> const Hash& { return hashes[i]; }; writer.write_up_to(n, leaf_at); + // Oracle: a full, never-flushed tree with the same leaves. merkle::Tree tree; for (uint64_t i = 0; i < n; i++) { @@ -49,7 +54,28 @@ static void check_size( } const Hash root = tree.root(); - const TileHashSource source(store, n); + // Production-shaped source: full tiles serve the covered prefix, an in-memory + // tree serves the un-tiled frontier. Drop the tiled past from the frontier + // tree so proofs over it are genuinely served from the tiles. merklecpp keeps + // at least one resident leaf, so never flush the whole tree. + const uint64_t covered = (n / 256) * 256; // 256 == TILE_WIDTH + merkle::Tree frontier; + for (uint64_t i = 0; i < n; i++) + { + frontier.insert(hashes[i]); + } + uint64_t drop_to = covered; + if (n > 0 && drop_to >= n) + { + drop_to = n - 1; + } + if (drop_to > 0) + { + frontier.flush_to((size_t)drop_to); + } + const MemoryHashSource mem(frontier); + const TileHashSource tiles(store, covered); + const CombinedHashSource source(mem, tiles); const ProofEngine engine(source); // Root recomputed from tiles equals the library root. @@ -192,7 +218,7 @@ int main() } std::cout << "small/medium sizes: OK" << '\n'; - // Large tree: exercises full L1 tiles and an L2 partial. + // Large tree: exercises full L1 tiles and the in-memory frontier. check_size(base / "big", 70000, hashes); std::cout << "size 70000: OK" << '\n'; diff --git a/test/tiles_tree.cpp b/test/tiles_tree.cpp index 297aaf3..3baca19 100644 --- a/test/tiles_tree.cpp +++ b/test/tiles_tree.cpp @@ -174,8 +174,8 @@ int main() { tt.append(hashes[i]); } - tt.flush(); // tiles_size = 1000; compact to covered = 768 - expect(tt.flushed_size() == n1, "flushed size"); + tt.flush(); // flushes full tiles; flushed_size = covered = 768 + expect(tt.flushed_size() == 768, "flushed size"); expect(tt.tree_ref().min_index() == 768, "compacted to 768"); for (uint64_t i = n1; i < N; i++) @@ -243,8 +243,8 @@ int main() // A second flush writes further tiles; proofs for now-flushed indices still // work (this also confirms the writer never reads a flushed leaf). - tt.flush(); // tiles_size = 1500; flush to covered = 1280 - expect(tt.flushed_size() == N, "second flushed size"); + tt.flush(); // flushes full tiles; flushed_size = covered = 1280 + expect(tt.flushed_size() == 1280, "second flushed size"); expect(tt.tree_ref().min_index() == 1280, "flushed to 1280"); for (const uint64_t i : @@ -264,6 +264,38 @@ int main() std::cout << "tiled tree (flush + combination): OK" << '\n'; + // ---- Part 2b: compaction when the size is an exact multiple of TILE_WIDTH + // (here 512). flush_to cannot drain the whole tree, so compact() keeps + // one resident leaf rather than throwing; proofs still resolve. + { + TiledTree::Config mcfg; + mcfg.prefix = base / "tt_multiple"; + mcfg.compact_on_flush = true; + TiledTree mtt(mcfg); + for (uint64_t i = 0; i < 512; i++) + { + mtt.append(hashes[i]); + } + mtt.flush(); // covered == size == 512; compaction must not throw + expect(mtt.flushed_size() == 512, "multiple: flushed size 512"); + expect(mtt.tree_ref().min_index() == 511, "multiple: one leaf retained"); + + merkle::Tree mref; + for (uint64_t i = 0; i < 512; i++) + { + mref.insert(hashes[i]); + } + const Hash mroot = mref.root(); + expect(mtt.root() == mroot, "multiple: root matches reference"); + for (const uint64_t i : {(uint64_t)0, (uint64_t)256, (uint64_t)511}) + { + const auto p = mtt.inclusion_proof(i, 512); + expect(*p == *mref.path(i), "multiple: inclusion==ref"); + expect(p->verify(mroot), "multiple: inclusion verify"); + } + std::cout << "tiled tree (exact-multiple compaction): OK" << '\n'; + } + // ---- Part 3: rollback. Tiles are immutable, so only un-tiled // (post-flush) entries may be rolled back. { @@ -293,7 +325,7 @@ int main() { rb.append(hashes[i]); } - rb.flush(); // tiles_size = 300 + rb.flush(); // flushed_size = covered = 256 for (uint64_t i = 300; i < 400; i++) { rb.append(hashes[i]); @@ -304,7 +336,8 @@ int main() { rb.append(hashes[1000 + i]); // re-append DIFFERENT leaves } - rb.flush(); // tiles_size = 400 + rb.flush(); // flushed_size = covered = 256 (tile [0,256) already + // written) // Reference tree of the exact post-rollback state. merkle::Tree exp_tree; @@ -335,31 +368,33 @@ int main() 300, 400, *exp_tree.past_root(299), exp_root, cp), "rb consistency 300->400"); - // Rolling back committed entries is refused (below and at the - // boundary); rolling back to exactly the tiled size removes nothing and - // is allowed. + // Only the immutable full-tile prefix [0,256) is protected: rolling + // back into it is refused, while rolling back within the un-tiled + // frontier (>= flushed_size()) is allowed. + expect(rb.flushed_size() == 256, "rb flushed to full-tile prefix"); + bool threw = false; try { - rb.retract_to(100); + rb.retract_to(100); // size 101 < 256 } catch (const std::exception&) { threw = true; } - expect(threw, "rb retract below tiled size throws"); + expect(threw, "rb retract below tiled prefix throws"); threw = false; try { - rb.retract_to(398); + rb.retract_to(254); // size 255 < 256 } catch (const std::exception&) { threw = true; } - expect(threw, "rb retract to size 399 throws"); - rb.retract_to(399); - expect(rb.size() == 400, "rb retract to tiled size is a no-op"); + expect(threw, "rb retract just below tiled prefix throws"); + rb.retract_to(255); // size 256 == flushed_size(): drops only frontier + expect(rb.size() == 256, "rb retract to tiled prefix allowed"); } // 3c. Rollback interacts correctly with compaction (flushed + tiled @@ -373,13 +408,13 @@ int main() { rb.append(hashes[i]); } - rb.flush(); // tiles_size = 1000; flush to 768 + rb.flush(); // flushed_size = covered = 768; compact to 768 expect(rb.tree_ref().min_index() == 768, "rb compact flushed to 768"); for (uint64_t i = 1000; i < 1200; i++) { rb.append(hashes[i]); } - rb.retract_to(1099); // frontier rollback (>= tiles_size) is allowed + rb.retract_to(1099); // frontier rollback (>= flushed_size()) is allowed expect(rb.size() == 1100, "rb compact frontier retract ok"); merkle::Tree exp_tree; diff --git a/test/tiles_writer.cpp b/test/tiles_writer.cpp index 9a0f049..004aa42 100644 --- a/test/tiles_writer.cpp +++ b/test/tiles_writer.cpp @@ -3,6 +3,7 @@ #include "util.h" +#include #include #include #include @@ -28,9 +29,24 @@ static void expect(bool cond, const std::string& what) } } -static bool partial_dir_exists(const TileStore& store, uint8_t L, uint64_t idx) +// True if any partial-tile directory (".p") exists anywhere in the store: the +// writer must never produce one. +static bool any_partial_dirs(const TileStore& store) { - return fs::exists(store.tile_path(TileRef{L, idx, 1}).parent_path()); + const fs::path tiles = store.root() / "tile"; + if (!fs::exists(tiles)) + { + return false; + } + const fs::recursive_directory_iterator it(tiles); + return std::any_of(begin(it), end(it), [](const fs::directory_entry& e) { + if (!e.is_directory()) + { + return false; + } + const std::string name = e.path().filename().string(); + return name.size() >= 2 && name.substr(name.size() - 2) == ".p"; + }); } // Roll up a full level-0 tile and compare with a level-1 tile entry. @@ -51,8 +67,9 @@ int main() try { - // ---- A. size 256: one full L0 tile + one width-1 L1 partial, and the L1 - // entry equals the real merkle::Tree root (roll-up == tree hash). + // ---- A. size 256: exactly one full L0 tile, no partials and no L1 tile + // (the L1 root is not yet a full tile; it stays in memory). The + // level-1 entry it would hold equals the real merkle::Tree root. { const auto hashes = make_hashes(256); const auto leaf_at = [&](uint64_t i) -> const Hash& { return hashes[i]; }; @@ -62,17 +79,14 @@ int main() const auto s = writer.write_up_to(256, leaf_at); expect(s.full_written == 1, "A full_written"); - expect(s.partial_written == 1, "A partial_written"); - expect(s.partial_removed == 0, "A partial_removed"); expect(store.has_full_tile(0, 0), "A L0 full tile"); expect( fs::file_size(store.tile_path(TileRef{0, 0, 0})) == 256U * Hash().size(), "A L0 full tile size"); - expect(!partial_dir_exists(store, 0, 0), "A no L0 partial"); expect(!store.has_full_tile(1, 0), "A no L1 full tile"); - expect(store.has_tile(TileRef{1, 0, 1}), "A L1 partial width 1"); + expect(!any_partial_dirs(store), "A no partial tiles"); // Level-0 tile is the leaf hashes verbatim. const auto l0 = store.read_tile(TileRef{0, 0, 0}); @@ -81,22 +95,19 @@ int main() expect(l0[i] == hashes[i], "A L0 entry == leaf"); } - // Level-1 entry == root of the equivalent merkle::Tree. + // The (un-tiled) level-1 entry == root of the equivalent merkle::Tree. merkle::Tree tree; for (const auto& h : hashes) { tree.insert(h); } - const Hash root = tree.root(); - const auto l1 = store.read_tile(TileRef{1, 0, 1}); - expect(l1.size() == 1, "A L1 partial width"); - expect(l1[0] == root, "A L1 entry == tree root"); - expect(rollup(l0) == root, "A rollup(L0) == tree root"); + expect(rollup(l0) == tree.root(), "A rollup(L0) == tree root"); std::cout << "A (size 256): OK" << '\n'; } - // ---- B. size 70000: the spec's worked example, exact tile set/widths. + // ---- B. size 70000: full tiles only (273 full L0 + 1 full L1 = 274); the + // incomplete frontier at every level is left untiled. { const auto hashes = make_hashes(70000); const auto leaf_at = [&](uint64_t i) -> const Hash& { return hashes[i]; }; @@ -105,26 +116,18 @@ int main() TileWriter writer(store); const auto s = writer.write_up_to(70000, leaf_at); - // 273 full L0 + 1 full L1 = 274 full; partials at L0, L1, L2. expect(s.full_written == 274, "B full_written"); - expect(s.partial_written == 3, "B partial_written"); - expect(s.partial_removed == 0, "B partial_removed"); expect(store.has_full_tile(0, 0), "B L0 tile 0"); expect(store.has_full_tile(0, 272), "B L0 tile 272"); expect(!store.has_full_tile(0, 273), "B no L0 tile 273"); - expect(store.has_tile(TileRef{0, 273, 112}), "B L0 partial width 112"); - expect( - fs::file_size(store.tile_path(TileRef{0, 273, 112})) == - 112U * Hash().size(), - "B L0 partial size"); expect(store.has_full_tile(1, 0), "B L1 tile 0"); expect(!store.has_full_tile(1, 1), "B no L1 tile 1"); - expect(store.has_tile(TileRef{1, 1, 17}), "B L1 partial width 17"); - expect(store.has_tile(TileRef{2, 0, 1}), "B L2 partial width 1"); + expect(!fs::exists(store.root() / "tile" / "2"), "B no level 2"); expect(!fs::exists(store.root() / "tile" / "3"), "B no level 3"); + expect(!any_partial_dirs(store), "B no partial tiles"); // Higher-level entries are roll-ups of the complete child tiles. const auto l1 = store.read_tile(TileRef{1, 0, 0}); @@ -136,7 +139,8 @@ int main() std::cout << "B (size 70000): OK" << '\n'; } - // ---- C. incremental writes: immutability, idempotency, partial growth. + // ---- C. incremental writes: immutability and idempotency; never a + // partial. { const auto hashes = make_hashes(1024); const auto leaf_at = [&](uint64_t i) -> const Hash& { return hashes[i]; }; @@ -150,50 +154,53 @@ int main() // Re-running at the same size writes nothing (full tiles are immutable). const auto s2 = writer.write_up_to(256, leaf_at); expect(s2.full_written == 0, "C s2 full immutable"); - expect(s2.partial_written == 0, "C s2 partial idempotent"); - expect(s2.partial_removed == 0, "C s2 nothing removed"); - // Grow to 600: one new full L0 tile, new L0 partial, L1 partial widens. + // Grow to 600: one new full L0 tile (covering 512), nothing else. const auto s3 = writer.write_up_to(600, leaf_at); expect(s3.full_written == 1, "C s3 full"); - expect(s3.partial_written == 2, "C s3 partial"); expect(store.has_full_tile(0, 0), "C L0 tile 0"); expect(store.has_full_tile(0, 1), "C L0 tile 1"); - expect(store.has_tile(TileRef{0, 2, 88}), "C L0 partial width 88"); - expect(store.has_tile(TileRef{1, 0, 2}), "C L1 partial width 2"); - expect(!store.has_tile(TileRef{1, 0, 1}), "C old L1 partial removed"); + expect(!store.has_full_tile(0, 2), "C no L0 tile 2"); expect(!store.has_full_tile(1, 0), "C no L1 full tile"); + expect(!any_partial_dirs(store), "C no partial tiles"); std::cout << "C (incremental): OK" << '\n'; } - // ---- D. supersession: a partial whose index becomes a full tile is - // removed when it is covered. + // ---- D. crossing into a full level-1 tile: the roll-up appears as a full + // tile, prior full tiles are never rewritten, and no partial is + // ever produced. { - const auto hashes = make_hashes(1024); + const auto hashes = make_hashes(65536); const auto leaf_at = [&](uint64_t i) -> const Hash& { return hashes[i]; }; TileStore store(base / "d"); TileWriter writer(store); - writer.write_up_to(300, leaf_at); - expect(store.has_full_tile(0, 0), "D L0 tile 0"); - expect(store.has_tile(TileRef{0, 1, 44}), "D L0 partial width 44"); - expect(store.has_tile(TileRef{1, 0, 1}), "D L1 partial width 1"); + // One leaf short of a full L1 tile: 255 full L0 tiles, no L1 yet. + const auto s1 = writer.write_up_to(65535, leaf_at); + expect(s1.full_written == 255, "D s1 255 full L0"); + expect(!store.has_full_tile(1, 0), "D no L1 before completion"); + expect(!any_partial_dirs(store), "D no partial before completion"); + + // Completing the 256th L0 tile yields one new L0 tile and one full L1. + const auto s2 = writer.write_up_to(65536, leaf_at); + expect(s2.full_written == 2, "D s2 new L0 + new L1"); + expect(store.has_full_tile(0, 255), "D L0 tile 255"); + expect(store.has_full_tile(1, 0), "D L1 tile 0"); + expect(!store.has_full_tile(1, 1), "D no L1 tile 1"); + expect(!store.has_full_tile(2, 0), "D no L2 tile"); + expect(!any_partial_dirs(store), "D no partial tiles"); - const auto s = writer.write_up_to(512, leaf_at); - expect(s.full_written == 1, "D full"); - expect(s.partial_removed == 1, "D one partial superseded"); + const auto l1 = store.read_tile(TileRef{1, 0, 0}); + expect(l1[0] == rollup(store.read_tile(TileRef{0, 0, 0})), "D L1[0]"); - expect(store.has_full_tile(0, 1), "D L0 tile 1"); - expect( - !partial_dir_exists(store, 0, 1), "D superseded L0 partial removed"); - expect(!store.has_tile(TileRef{0, 1, 44}), "D old L0 partial gone"); - expect(store.has_tile(TileRef{1, 0, 2}), "D L1 partial width 2"); - expect(!store.has_tile(TileRef{1, 0, 1}), "D old L1 partial gone"); + // Re-running writes nothing (everything already full and immutable). + const auto s3 = writer.write_up_to(65536, leaf_at); + expect(s3.full_written == 0, "D s3 immutable rerun"); - std::cout << "D (supersession): OK" << '\n'; + std::cout << "D (full L1 tile): OK" << '\n'; } std::cout << "tiles_writer: OK" << '\n'; diff --git a/test/time_tiles.cpp b/test/time_tiles.cpp index 5ba141b..dce7d8d 100644 --- a/test/time_tiles.cpp +++ b/test/time_tiles.cpp @@ -81,14 +81,13 @@ int main() << append_s << " sec (" << (uint64_t)((double)num_leaves / append_s) << " leaves/sec)\n"; - // 2. Flush: write newly-complete tiles to disk. + // 2. Flush: write newly-complete full tiles to disk. t = std::chrono::high_resolution_clock::now(); const auto stats = log.flush(); const double flush_s = secs_since(t); - const uint64_t tiles = stats.full_written + stats.partial_written; - std::cout << "flush (to disk) : " << stats.full_written << " full + " - << stats.partial_written << " partial tiles in " << flush_s - << " sec (" << (uint64_t)((double)tiles / flush_s) + std::cout << "flush (to disk) : " << stats.full_written + << " full tiles in " << flush_s << " sec (" + << (uint64_t)((double)stats.full_written / flush_s) << " tiles/sec)\n"; const uint64_t n = log.size(); From 89b370e2e945f4299e7fad03ce546f36bfbdf045 Mon Sep 17 00:00:00 2001 From: Amaury Chamayou Date: Sun, 14 Jun 2026 20:02:29 +0000 Subject: [PATCH 15/30] refactor(tiles): drop dead partial machinery; full-only bundles; dedup proofs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The earlier refactor made the Merkle write/read paths full-tile-only but left the partial-tile apparatus stranded in the store, and entry bundles still wrote a mutable rightmost partial — the very pattern that was removed for tiles. Simplify three ways: 1. TileStoreT/TileRef: shrink TileRef to {level, index}; drop num_hashes(), is_partial(), the .p/ branch in tile_path(), and the unused has_tile(). write_tile/read_tile are full-only (TILE_WIDTH). Only the store's own unit test ever constructed a partial Merkle TileRef. 2. EntryBundleWriterT: make bundles full-only, mirroring the un-tiled Merkle frontier. Remove write_partial/remove_partial, the Options struct, the remove_superseded_partials behaviour, last_partial_* state, and the partial_written/partial_removed stats. entries_path/write_entry_bundle/ read_entry_bundle are full-only. The incomplete tail stays with the application until it completes a bundle. 3. TiledTreeT: factor the duplicated Memory+Tile+Combined+Engine setup in inclusion_proof/consistency_proof into a private with_engine() helper. Update tlog-tiles.md (§2.1, §4, §6.3, §6.4, §6.8) and tiles-guide.md to match, and drop a stale read_entry API sketch that never existed. Net -139 lines; 17/17 tests pass; clang-tidy clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/tlog-tiles.md | 38 ++++--- doc/tiles-guide.md | 5 +- merklecpp_tiles.h | 220 +++++++++++---------------------------- test/tiles_entries.cpp | 45 ++------ test/tiles_store.cpp | 37 ++----- test/tiles_writer.cpp | 14 +-- 6 files changed, 110 insertions(+), 249 deletions(-) diff --git a/doc/design/tlog-tiles.md b/doc/design/tlog-tiles.md index 6857809..8f03ce0 100644 --- a/doc/design/tlog-tiles.md +++ b/doc/design/tlog-tiles.md @@ -81,10 +81,11 @@ A tiled log exposes the Merkle tree as a set of static resources: > **merklecpp deviation.** This implementation never writes or reads partial > *tiles*. The incomplete frontier (everything past the last full tile) is held > in the in-memory tree, not on disk, so the on-disk tile set is the full-tile - > prefix only. Entry bundles (below) are a separate, application-owned resource - > and may still be partial. -- **Entry bundles** at `/tile/entries/[.p/]`: big-endian `uint16` - length-prefixed raw entries. (See [§6.8](#68-entry-bundles-optional).) + > prefix only. Entry bundles (below) follow the same rule: only full bundles + > are written, and the incomplete tail stays with the application. +- **Entry bundles** at `/tile/entries/`: big-endian `uint16` + length-prefixed raw entries, 256 per full bundle (full bundles only — see the + deviation above). (See [§6.8](#68-entry-bundles-optional).) - **Pruning**: a log keeps a *minimum index*; tiles/bundles whose end index is `≤ minimum index` may be denied. This maps cleanly onto merklecpp's `flush_to()` / `min_index()`. @@ -203,9 +204,8 @@ Rooted at a configurable `prefix` directory on local disk: 1/ # level 1 (roll-ups of level-0 full tiles) 000 ... .../ - entries/ # optional raw entry bundles + entries/ # optional raw entry bundles (full only) 000 001 ... - .p/ ``` ### Index encoding (`encode_tile_index`) @@ -227,9 +227,9 @@ Resource paths: implementation writes) - Partial tile (tlog-tiles format; **not produced or read here**): `tile//.p/` -- Entry bundle: `tile/entries/[.p/]` +- Entry bundle: `tile/entries/` (full bundles only) -Tile byte format: the `W` entries concatenated, each `HASH_SIZE` raw bytes +Tile byte format: the 256 entries concatenated, each `HASH_SIZE` raw bytes (`HashT::bytes`); a full tile is `256 * HASH_SIZE` bytes. --- @@ -334,7 +334,7 @@ recommended; both options keep hashing untouched. ### 6.3 `TileStoreT` — disk I/O ```cpp -struct TileRef { uint8_t level; uint64_t index; uint16_t width; }; // width 0 ⇒ full +struct TileRef { uint8_t level; uint64_t index; }; // full tiles only class TileStoreT { public: @@ -343,15 +343,12 @@ public: // Path helpers (pure) static std::string encode_index(uint64_t n); std::filesystem::path tile_path(const TileRef&) const; - std::filesystem::path entries_path(uint64_t n, uint16_t width = 0) const; + std::filesystem::path entries_path(uint64_t n) const; // Tiles bool has_full_tile(uint8_t level, uint64_t index) const; - std::vector read_tile(const TileRef&) const; // width entries + std::vector read_tile(const TileRef&) const; // 256 entries void write_tile(const TileRef&, const std::vector&); // atomic (tmp+rename) - - // Single entry convenience (used heavily by TileHashSource) - bool read_entry(uint8_t level, uint64_t global_entry, Hash& out, uint64_t size) const; }; ``` @@ -388,13 +385,13 @@ for L = 0, 1, 2, ... while entries_L = size >> (8*L) is > 0: full_L = entries_L / 256 for N in [first_unwritten_full(L) .. full_L): # new full tiles only entries = [ entry(L, N*256 + i) for i in 0..255 ] - store.write_tile({L, N, width=0}, entries) + store.write_tile({L, N}, entries) # the rightmost, incomplete (partial) entries are NOT written; they stay in # the in-memory tree until they complete a full tile entry(L, g): if L == 0: return leaf_at(g) - else: return perfect_root( store.read_tile({L-1, g, width=0}) ) # 256→1 + else: return perfect_root( store.read_tile({L-1, g}) ) # 256→1 ``` - Level 0 reads leaf hashes via the supplied `leaf_at` (e.g. `Tree::leaf(i)`), @@ -544,12 +541,13 @@ Stateless alternative for callers with their own storage: free functions merklecpp never sees raw entries (callers insert pre-computed leaf hashes), so entry bundles are an **application-owned** add-on, included for completeness: -- `tile/entries/[.p/]` stores big-endian `uint16` length-prefixed entries, - 256 per full bundle. +- `tile/entries/` stores big-endian `uint16` length-prefixed entries, + 256 per full bundle (full bundles only). - The application is responsible for the leaf-hash derivation it uses (e.g. `leaf_hash = H(entry)`); merklecpp stores whatever leaf hash is inserted. -- A `BundleWriter` mirrors `TileWriterT` (write full bundles on 256 boundaries, - partial bundle for the current size). Marked optional/secondary. +- An `EntryBundleWriterT` mirrors `TileWriterT`: it writes full bundles on + 256-entry boundaries only; the incomplete tail stays with the application. + Marked optional/secondary. --- diff --git a/doc/tiles-guide.md b/doc/tiles-guide.md index 29a1dfa..fa4e2c7 100644 --- a/doc/tiles-guide.md +++ b/doc/tiles-guide.md @@ -233,14 +233,15 @@ into a `ProofEngine` for you. If you also want to store the raw log entries (tlog-tiles "entry bundles"), use `EntryBundleWriter`. Bundles are level-0 only and application-owned — merklecpp stores leaf hashes; you supply the raw bytes and decide how an entry maps to its -leaf hash. +leaf hash. Only full bundles (256 entries) are written; the incomplete tail +stays with your application until it completes a bundle. ```cpp merkle::tiles::EntryBundleWriter bundles(store); bundles.write_up_to(num_entries, [&](uint64_t i) -> std::vector { return raw_entry_bytes(i); }); -// Read them back (full bundle = 256 entries; pass the width for a partial one). +// Read a full bundle back (256 entries). std::vector> e = store.read_entry_bundle(/*index=*/0); ``` diff --git a/merklecpp_tiles.h b/merklecpp_tiles.h index 9009152..4ae43e7 100644 --- a/merklecpp_tiles.h +++ b/merklecpp_tiles.h @@ -65,7 +65,9 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) return r; } - /// @brief Identifies a single tile within a tiled log. + /// @brief Identifies a single (full) tile within a tiled log. + /// @note Only full, TILE_WIDTH-wide tiles are produced and consumed; the + /// incomplete frontier is never tiled (see doc/design/tlog-tiles.md). struct TileRef { /// @brief The level of the tile (0 == leaf hashes). @@ -73,21 +75,6 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) /// @brief The index of the tile within its level. uint64_t index = 0; - - /// @brief The width of the tile; 0 denotes a full tile (TILE_WIDTH). - uint16_t width = 0; - - /// @brief The number of hashes the tile holds. - [[nodiscard]] uint16_t num_hashes() const - { - return width == 0 ? TILE_WIDTH : width; - } - - /// @brief Whether the tile is partial. - [[nodiscard]] bool is_partial() const - { - return width != 0; - } }; /// @brief Reads and writes tlog-tiles tile files on a local filesystem. @@ -125,45 +112,29 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) /// @brief The filesystem path of a tile. [[nodiscard]] std::filesystem::path tile_path(const TileRef& ref) const { - std::string rel = "tile/" + std::to_string((unsigned)ref.level) + "/" + - encode_tile_index(ref.index); - if (ref.is_partial()) - { - rel += ".p/" + std::to_string((unsigned)ref.width); - } - return prefix / rel; + return prefix / + ("tile/" + std::to_string((unsigned)ref.level) + "/" + + encode_tile_index(ref.index)); } /// @brief The filesystem path of an entry bundle. - [[nodiscard]] std::filesystem::path entries_path( - uint64_t index, uint16_t width = 0) const + [[nodiscard]] std::filesystem::path entries_path(uint64_t index) const { - std::string rel = "tile/entries/" + encode_tile_index(index); - if (width != 0) - { - rel += ".p/" + std::to_string((unsigned)width); - } - return prefix / rel; + return prefix / ("tile/entries/" + encode_tile_index(index)); } /// @brief Whether a full tile exists on disk. [[nodiscard]] bool has_full_tile(uint8_t level, uint64_t index) const { - return std::filesystem::exists(tile_path(TileRef{level, index, 0})); - } - - /// @brief Whether a specific tile exists on disk. - [[nodiscard]] bool has_tile(const TileRef& ref) const - { - return std::filesystem::exists(tile_path(ref)); + return std::filesystem::exists(tile_path(TileRef{level, index})); } /// @brief Writes a tile to disk atomically. /// @param ref The tile to write - /// @param hashes The tile's hashes (exactly ref.num_hashes() of them) + /// @param hashes The tile's hashes (exactly TILE_WIDTH of them) void write_tile(const TileRef& ref, const std::vector& hashes) { - if (hashes.size() != ref.num_hashes()) + if (hashes.size() != TILE_WIDTH) { throw std::runtime_error("tile width mismatch"); } @@ -180,20 +151,20 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) /// @brief Reads a tile from disk. /// @param ref The tile to read - /// @return The tile's hashes (ref.num_hashes() of them) + /// @return The tile's hashes (TILE_WIDTH of them) [[nodiscard]] std::vector read_tile(const TileRef& ref) const { std::vector bytes = read_file(tile_path(ref)); - const size_t expected = (size_t)ref.num_hashes() * HASH_SIZE; + const size_t expected = (size_t)TILE_WIDTH * HASH_SIZE; if (bytes.size() != expected) { throw std::runtime_error("unexpected tile size"); } std::vector hashes; - hashes.reserve(ref.num_hashes()); + hashes.reserve(TILE_WIDTH); size_t position = 0; - for (uint16_t i = 0; i < ref.num_hashes(); i++) + for (uint16_t i = 0; i < TILE_WIDTH; i++) { hashes.emplace_back(bytes, position); } @@ -203,38 +174,31 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) /// @brief Whether a full entry bundle exists on disk. [[nodiscard]] bool has_entry_bundle(uint64_t index) const { - return std::filesystem::exists(entries_path(index, 0)); + return std::filesystem::exists(entries_path(index)); } - /// @brief Writes an entry bundle to disk atomically. + /// @brief Writes a full entry bundle to disk atomically. /// @param index The bundle index - /// @param width The bundle width; 0 denotes a full bundle (TILE_WIDTH) - /// @param entries The raw log entries (exactly width, or TILE_WIDTH) + /// @param entries The raw log entries (exactly TILE_WIDTH of them) /// @note Entries are stored in the tlog-tiles entry-bundle format: a /// sequence of big-endian uint16 length-prefixed byte strings. void write_entry_bundle( - uint64_t index, - uint16_t width, - const std::vector>& entries) + uint64_t index, const std::vector>& entries) { - const uint16_t expected = width == 0 ? TILE_WIDTH : width; - if (entries.size() != expected) + if (entries.size() != TILE_WIDTH) { throw std::runtime_error("entry bundle width mismatch"); } - write_file_atomically( - entries_path(index, width), encode_entries(entries)); + write_file_atomically(entries_path(index), encode_entries(entries)); } - /// @brief Reads an entry bundle from disk. + /// @brief Reads a full entry bundle from disk. /// @param index The bundle index - /// @param width The bundle width; 0 denotes a full bundle (TILE_WIDTH) - /// @return The raw log entries + /// @return The raw log entries (TILE_WIDTH of them) [[nodiscard]] std::vector> read_entry_bundle( - uint64_t index, uint16_t width = 0) const + uint64_t index) const { - const uint16_t count = width == 0 ? TILE_WIDTH : width; - return decode_entries(read_file(entries_path(index, width)), count); + return decode_entries(read_file(entries_path(index)), TILE_WIDTH); } /// @brief Encodes log entries into the tlog-tiles entry-bundle format. @@ -460,7 +424,7 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) continue; // immutable: never rewrite an existing full tile } store.write_tile( - TileRef{level, n, 0}, + TileRef{level, n}, collect(level, n * TILE_WIDTH, TILE_WIDTH, leaf_at)); stats.full_written++; } @@ -556,7 +520,7 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) { // Roll up the complete child full tile (256 complete entries). out.push_back(perfect_root( - store.read_tile(TileRef{(uint8_t)(level - 1), g, 0}))); + store.read_tile(TileRef{(uint8_t)(level - 1), g}))); } } return out; @@ -659,7 +623,7 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) const uint64_t span = (uint64_t)1 << level; const uint64_t start = index << level; const std::vector tile = - store.read_tile(TileRef{0, start / TILE_WIDTH, 0}); + store.read_tile(TileRef{0, start / TILE_WIDTH}); out = roll_up(tile, start % TILE_WIDTH, span); return; } @@ -675,7 +639,7 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) if (n < full_tiles) { // One full level-L tile holds all 2**r entries of this subtree. - const std::vector tile = store.read_tile(TileRef{L, n, 0}); + const std::vector tile = store.read_tile(TileRef{L, n}); out = roll_up(tile, first % TILE_WIDTH, (uint64_t)1 << r); return; } @@ -1195,21 +1159,16 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) /// (recent frontier); @p proof_size may exceed flushed_size(). std::shared_ptr inclusion_proof(uint64_t index, uint64_t proof_size) { - MemoryHashSourceT mem(tree); - TileHashSourceT tile_src(store, tiles_size); - CombinedHashSourceT combined(mem, tile_src); - ProofEngineT engine(combined); - return engine.inclusion_proof(index, proof_size); + return with_engine([&](const auto& engine) { + return engine.inclusion_proof(index, proof_size); + }); } /// @brief Consistency proof between tree sizes @p m and @p n. std::vector consistency_proof(uint64_t m, uint64_t n) { - MemoryHashSourceT mem(tree); - TileHashSourceT tile_src(store, tiles_size); - CombinedHashSourceT combined(mem, tile_src); - ProofEngineT engine(combined); - return engine.consistency_proof(m, n); + return with_engine( + [&](const auto& engine) { return engine.consistency_proof(m, n); }); } /// @brief Consistency proof between the trees whose last leaves are at @@ -1229,6 +1188,21 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) Writer writer; Tree tree; uint64_t tiles_size = 0; + + /// @brief Builds a proof engine over the combined resident-tree (frontier) + /// and full-tile (flushed past) source, and invokes @p fn with it. + /// @note The sources and engine are stack-local; @p fn must consume the + /// engine before returning (proofs are returned by value, holding hash + /// copies, so the result outlives the engine). + template + auto with_engine(Fn fn) + { + MemoryHashSourceT mem(tree); + TileHashSourceT tile_src(store, tiles_size); + CombinedHashSourceT combined(mem, tile_src); + ProofEngineT engine(combined); + return fn(engine); + } }; /// @brief Writes tlog-tiles entry bundles (raw log entries) for a growing @@ -1236,8 +1210,11 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) /// @note Entry bundles are level-0 only and application-owned: merklecpp /// stores leaf hashes, while the raw entries (and the leaf-hash derivation /// linking each entry to its level-0 tile hash) are the application's - /// responsibility. Full bundles are immutable and written once; only the - /// rightmost partial bundle may grow or be removed once superseded. + /// responsibility. Only full bundles (TILE_WIDTH entries) are written; they + /// are immutable and written exactly once. The incomplete tail is never + /// bundled (no partial bundles are produced): it stays with the application + /// until it grows into a full bundle, mirroring the un-tiled Merkle + /// frontier. template < size_t HASH_SIZE, void HASH_FUNCTION( @@ -1250,35 +1227,23 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) /// @brief Supplies the raw bytes of the log entry at a given index. using EntryFn = std::function(uint64_t)>; - /// @brief Write-path options. - struct Options - { - /// @brief Write the partial (rightmost) bundle for the current size. - bool write_partial = true; - - /// @brief Remove a partial bundle once superseded by a full bundle (or - /// a wider partial at the same index). - bool remove_superseded_partials = true; - }; - /// @brief Counts of work performed by a write_up_to call. struct Stats { + /// @brief Number of full bundles written. uint64_t full_written = 0; - uint64_t partial_written = 0; - uint64_t partial_removed = 0; }; - explicit EntryBundleWriterT(Store& store, Options options = {}) : - store(store), options(options) - {} + explicit EntryBundleWriterT(Store& store) : store(store) {} - /// @brief Writes all newly-complete full bundles and the partial bundle - /// for a log of @p size entries. + /// @brief Writes all newly-complete full bundles for a log of @p size + /// entries. /// @param size The current number of entries /// @param entry_at Returns the raw bytes of the entry at an index in - /// [0, size) - /// @return Counts of bundles written/removed + /// [0, size); only ever queried for entries of complete bundles. + /// @return Counts of bundles written + /// @note Incremental: full bundles already on disk are immutable and are + /// never rewritten. The incomplete tail is never bundled. Stats write_up_to(uint64_t size, const EntryFn& entry_at) { Stats stats; @@ -1297,30 +1262,21 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) continue; // immutable: never rewrite an existing full bundle } store.write_entry_bundle( - n, 0, collect(n * TILE_WIDTH, TILE_WIDTH, entry_at)); + n, collect(n * TILE_WIDTH, TILE_WIDTH, entry_at)); stats.full_written++; } if (full > next_full) { next_full = full; } - - if (options.write_partial) - { - write_partial(full, (uint16_t)(size % TILE_WIDTH), entry_at, stats); - } return stats; } protected: // NOLINTNEXTLINE(cppcoreguidelines-avoid-const-or-ref-data-members) Store& store; - Options options; uint64_t next_full = 0; bool cursor_inited = false; - uint64_t last_partial_index = 0; - uint16_t last_partial_width = 0; - bool has_last_partial = false; std::vector> collect( uint64_t first, uint64_t count, const EntryFn& entry_at) @@ -1361,56 +1317,6 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) } return lo + 1; } - - void write_partial( - uint64_t index, uint16_t width, const EntryFn& entry_at, Stats& stats) - { - if (width == 0) - { - if (options.remove_superseded_partials && has_last_partial) - { - remove_partial(last_partial_index); - has_last_partial = false; - stats.partial_removed++; - } - return; - } - - if ( - has_last_partial && last_partial_index == index && - last_partial_width == width) - { - return; - } - - if (options.remove_superseded_partials && has_last_partial) - { - if (last_partial_index != index) - { - remove_partial(last_partial_index); - stats.partial_removed++; - } - else - { - remove_partial(index); - } - } - - store.write_entry_bundle( - index, width, collect(index * TILE_WIDTH, width, entry_at)); - stats.partial_written++; - last_partial_index = index; - last_partial_width = width; - has_last_partial = true; - } - - void remove_partial(uint64_t index) - { - const std::filesystem::path dir = - store.entries_path(index, 1).parent_path(); - std::error_code ec; - std::filesystem::remove_all(dir, ec); - } }; /// @brief Default tile store (SHA256, default hash function). diff --git a/test/tiles_entries.cpp b/test/tiles_entries.cpp index 8fa32bc..92b44ba 100644 --- a/test/tiles_entries.cpp +++ b/test/tiles_entries.cpp @@ -57,15 +57,14 @@ int main() return hashes[i]; // HashT -> vector }; - // 2. size 256: a single full bundle, no partial. + // 2. size 256: a single full bundle, no tail. { TileStore store(base / "e256"); EntryBundleWriter writer(store); const auto s = writer.write_up_to(256, entry_at); - expect(s.full_written == 1 && s.partial_written == 0, "256 counts"); + expect(s.full_written == 1, "256 counts"); expect(store.has_entry_bundle(0), "256 full bundle"); - expect( - !fs::exists(store.entries_path(1, 1).parent_path()), "256 no partial"); + expect(!store.has_entry_bundle(1), "256 no second bundle"); const auto b0 = store.read_entry_bundle(0); expect(b0.size() == 256, "256 bundle width"); @@ -75,30 +74,23 @@ int main() } } - // 3. size 70000: 273 full bundles + a width-112 partial, plus tile linkage. + // 3. size 70000: 273 full bundles (the 112-entry tail is not bundled), + // plus tile linkage. { TileStore store(base / "e70k"); EntryBundleWriter writer(store); const auto s = writer.write_up_to(70000, entry_at); - expect(s.full_written == 273 && s.partial_written == 1, "70000 counts"); + expect(s.full_written == 273, "70000 counts"); expect(store.has_entry_bundle(0), "70000 bundle 0"); expect(store.has_entry_bundle(272), "70000 bundle 272"); expect(!store.has_entry_bundle(273), "70000 no bundle 273"); - expect(fs::exists(store.entries_path(273, 112)), "70000 partial 112"); - - const auto bp = store.read_entry_bundle(273, 112); - expect(bp.size() == 112, "70000 partial width"); - for (size_t i = 0; i < 112; i++) - { - expect(Hash(bp[i]) == hashes[69888 + i], "70000 partial round-trip"); - } // Level-0 tile entries (leaf hashes) correspond to the bundle entries // under the identity leaf-hash used here. TileWriter tw(store); const auto leaf_at = [&](uint64_t i) -> const Hash& { return hashes[i]; }; tw.write_up_to(70000, leaf_at); - const auto tile0 = store.read_tile(TileRef{0, 0, 0}); + const auto tile0 = store.read_tile(TileRef{0, 0}); const auto bundle0 = store.read_entry_bundle(0); for (size_t i = 0; i < 256; i++) { @@ -106,33 +98,18 @@ int main() } } - // 4. Incremental writes: full bundles immutable, partial grows. + // 4. Incremental writes: full bundles are immutable, the tail is not + // bundled. { TileStore store(base / "einc"); EntryBundleWriter writer(store); expect(writer.write_up_to(256, entry_at).full_written == 1, "inc s1"); const auto s2 = writer.write_up_to(256, entry_at); - expect( - s2.full_written == 0 && s2.partial_written == 0, "inc immutable rerun"); + expect(s2.full_written == 0, "inc immutable rerun"); const auto s3 = writer.write_up_to(600, entry_at); expect(s3.full_written == 1, "inc s3 full"); expect(store.has_entry_bundle(1), "inc bundle 1"); - expect(fs::exists(store.entries_path(2, 88)), "inc partial 88"); - } - - // 5. Supersession: a partial whose index becomes a full bundle is removed. - { - TileStore store(base / "esup"); - EntryBundleWriter writer(store); - writer.write_up_to(300, entry_at); - expect(fs::exists(store.entries_path(1, 44)), "sup partial 44 before"); - const auto s = writer.write_up_to(512, entry_at); - expect(s.partial_removed == 1, "sup one removed"); - expect(store.has_entry_bundle(1), "sup full bundle 1"); - expect(!fs::exists(store.entries_path(1, 44)), "sup partial 44 gone"); - expect( - !fs::exists(store.entries_path(1, 1).parent_path()), - "sup partial dir gone"); + expect(!store.has_entry_bundle(2), "inc no tail bundle"); } std::cout << "tiles_entries: OK" << '\n'; diff --git a/test/tiles_store.cpp b/test/tiles_store.cpp index a4c681e..d0445f1 100644 --- a/test/tiles_store.cpp +++ b/test/tiles_store.cpp @@ -66,31 +66,23 @@ int main() merkle::tiles::TileStore store(dir); - // 2. Resource path layout. + // 2. Resource path layout (full tiles and bundles only). expect_eq( - rel(store, store.tile_path(TileRef{0, 0, 0})), + rel(store, store.tile_path(TileRef{0, 0})), "tile/0/000", - "tile_path L0 N0 full"); + "tile_path L0 N0"); expect_eq( - rel(store, store.tile_path(TileRef{0, 273, 112})), - "tile/0/273.p/112", - "tile_path L0 N273 partial"); - expect_eq( - rel(store, store.tile_path(TileRef{1, 1234067, 0})), + rel(store, store.tile_path(TileRef{1, 1234067})), "tile/1/x001/x234/067", "tile_path L1 big index"); expect_eq( rel(store, store.entries_path(5)), "tile/entries/005", "entries full"); - expect_eq( - rel(store, store.entries_path(5, 3)), - "tile/entries/005.p/3", - "entries partial"); const size_t hsz = Hash().size(); // 3a. Full tile byte round-trip. const auto full = make_hashes(merkle::tiles::TILE_WIDTH); - const TileRef full_ref{0, 0, 0}; + const TileRef full_ref{0, 0}; store.write_tile(full_ref, full); expect(store.has_full_tile(0, 0), "has_full_tile after write"); expect(!store.has_full_tile(0, 5), "missing full tile"); @@ -105,25 +97,12 @@ int main() expect(full_rt[i] == full[i], "full tile hash round-trip"); } - // 3b. Partial tile byte round-trip. - const std::vector partial(full.begin(), full.begin() + 3); - const TileRef partial_ref{0, 1, 3}; - store.write_tile(partial_ref, partial); - expect( - fs::file_size(store.tile_path(partial_ref)) == (uintmax_t)3 * hsz, - "partial tile file size"); - const auto partial_rt = store.read_tile(partial_ref); - expect(partial_rt.size() == 3, "partial tile width round-trip"); - for (size_t i = 0; i < partial.size(); i++) - { - expect(partial_rt[i] == partial[i], "partial tile hash round-trip"); - } - - // 3c. Width mismatches are rejected. + // 3b. Wrong-width writes are rejected (only 256-wide tiles are valid). + const std::vector three(full.begin(), full.begin() + 3); bool threw = false; try { - store.write_tile(TileRef{0, 2, 0}, partial); // 3 hashes for a full tile + store.write_tile(TileRef{0, 2}, three); // 3 hashes for a full tile } catch (const std::exception&) { diff --git a/test/tiles_writer.cpp b/test/tiles_writer.cpp index 004aa42..f8ebaa2 100644 --- a/test/tiles_writer.cpp +++ b/test/tiles_writer.cpp @@ -82,14 +82,14 @@ int main() expect(store.has_full_tile(0, 0), "A L0 full tile"); expect( - fs::file_size(store.tile_path(TileRef{0, 0, 0})) == + fs::file_size(store.tile_path(TileRef{0, 0})) == 256U * Hash().size(), "A L0 full tile size"); expect(!store.has_full_tile(1, 0), "A no L1 full tile"); expect(!any_partial_dirs(store), "A no partial tiles"); // Level-0 tile is the leaf hashes verbatim. - const auto l0 = store.read_tile(TileRef{0, 0, 0}); + const auto l0 = store.read_tile(TileRef{0, 0}); for (size_t i = 0; i < hashes.size(); i++) { expect(l0[i] == hashes[i], "A L0 entry == leaf"); @@ -130,11 +130,11 @@ int main() expect(!any_partial_dirs(store), "B no partial tiles"); // Higher-level entries are roll-ups of the complete child tiles. - const auto l1 = store.read_tile(TileRef{1, 0, 0}); + const auto l1 = store.read_tile(TileRef{1, 0}); expect(l1.size() == 256, "B L1 full width"); - expect(l1[0] == rollup(store.read_tile(TileRef{0, 0, 0})), "B L1[0]"); + expect(l1[0] == rollup(store.read_tile(TileRef{0, 0})), "B L1[0]"); expect( - l1[255] == rollup(store.read_tile(TileRef{0, 255, 0})), "B L1[255]"); + l1[255] == rollup(store.read_tile(TileRef{0, 255})), "B L1[255]"); std::cout << "B (size 70000): OK" << '\n'; } @@ -193,8 +193,8 @@ int main() expect(!store.has_full_tile(2, 0), "D no L2 tile"); expect(!any_partial_dirs(store), "D no partial tiles"); - const auto l1 = store.read_tile(TileRef{1, 0, 0}); - expect(l1[0] == rollup(store.read_tile(TileRef{0, 0, 0})), "D L1[0]"); + const auto l1 = store.read_tile(TileRef{1, 0}); + expect(l1[0] == rollup(store.read_tile(TileRef{0, 0})), "D L1[0]"); // Re-running writes nothing (everything already full and immutable). const auto s3 = writer.write_up_to(65536, leaf_at); From 18923ec28e080e72de0f7775a0d6b885034f583c Mon Sep 17 00:00:00 2001 From: Amaury Chamayou Date: Sun, 14 Jun 2026 21:09:01 +0000 Subject: [PATCH 16/30] feat(tiles): remove write_higher_levels flag; add audit-driven tests Per the review in audit.md: higher-level roll-up tiles are required for efficient proofs over compacted history, and the option to skip them was a broken footgun -- TileHashSourceT::resolve assumes they exist, so write_higher_levels=false threw "cannot open file" during proofs. Remove the option entirely (higher levels are always written), which resolves audit finding C1 and makes test item T3 moot. Implement the audit's test gaps: - T1: corrupt/truncated tile and entry-bundle reads are rejected (tiles_store). - T2: level-2 tiles end-to-end via a 256^3-leaf test that cross-checks the writer roll-up, resolve's level-2 read, and the leaf chain (new tiles_level2), plus level-2 resolve-descend coverage in the tiles_proofs oracle sweep (sizes 65536/65537/300000). - T4: a fresh writer over an existing store resumes via full_prefix_length, rewriting nothing (tiles_writer section E). - T5: compaction with a non-zero retention_margin keeps recent leaves resident while flushed indices are served from tiles (tiles_tree 2c). - T6: empty tree (flush/compact no-ops, no root) and the 65537 boundary. audit.md records the full review and the resolution status of each item. All 18 tests pass; clang-tidy clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- audit.md | 306 +++++++++++++++++++++++++++++++++++++++ doc/design/tlog-tiles.md | 12 +- merklecpp_tiles.h | 30 +--- test/CMakeLists.txt | 1 + test/tiles_level2.cpp | 138 ++++++++++++++++++ test/tiles_proofs.cpp | 16 +- test/tiles_store.cpp | 64 ++++++++ test/tiles_tree.cpp | 81 +++++++++++ test/tiles_writer.cpp | 48 ++++++ 9 files changed, 662 insertions(+), 34 deletions(-) create mode 100644 audit.md create mode 100644 test/tiles_level2.cpp diff --git a/audit.md b/audit.md new file mode 100644 index 0000000..8f86483 --- /dev/null +++ b/audit.md @@ -0,0 +1,306 @@ +# Audit: tiled storage & tile-backed proofs (`experiment/tiles`) + +**Scope.** All changes on `experiment/tiles` relative to `main` (15 commits, ++3,900 / −2 lines): the new header `merklecpp_tiles.h`, the `TreeT::subtree_root` +addition in `merklecpp.h`, the `test/tiles_*.cpp` / `test/time_tiles.cpp` suite, +and `doc/design/tlog-tiles.md` + `doc/tiles-guide.md`. + +**Method.** Four independent review passes: a manual read plus three specialist +reviewers. Two reviewers built and ran the tile suite under GCC **and** Clang +with **ASan + UBSan**, and one cross-checked **113,462** generated proofs against +the `merkle::TreeT` oracle across every tile boundary. Where a finding is marked +*reproduced*, it was triggered in a running build, not merely read. + +**Headline.** The core proof math is **sound**: inclusion proofs are byte-identical +to `TreeT::path()/past_path()`, consistency proofs round-trip through +`verify_consistency`, and both were checked exhaustively for small trees and by a +large oracle sweep. The shift in `subtree_root` that looks like 64-bit UB is in +fact **safe**. No correctness defect exists on the **default** code path. The +issues below are (a) a real **durability** gap whose docs overstate the guarantee, +(b) a **non-default config** (`write_higher_levels = false`) that crashes proofs, +and (c) several **robustness / hostile-input / test / doc** gaps. + +> **Update (follow-up commit).** Acting on this audit: the `write_higher_levels` +> option has been **removed** entirely — higher-level roll-up tiles are now +> always written — which makes **C1 unreachable** (no way to disable the tiles +> `resolve` relies on) and **T3 obsolete** (no flag to test). The test gaps +> **T1, T2, T4, T5, T6** are now **implemented**: corrupt/truncated-file reads +> (`tiles_store`), level-2 tiles end-to-end via a 256³-leaf test (`tiles_level2`) +> plus level-2 `resolve` descend coverage in the `tiles_proofs` oracle sweep +> (sizes 65536/65537/300000), writer reopen/resume (`tiles_writer` §E), +> non-zero `retention_margin` compaction (`tiles_tree` Part 2c), and the empty +> tree (`tiles_tree` Part 0). Still open for a future change: **C2/D2** (fsync +> durability), **C3/C4/C5** (robustness hardening), **P1/D1** (tile cache). + +--- + +## Findings summary + +| # | Severity | Area | Summary | +|---|----------|------|---------| +| C1 | Medium | Correctness (config) | `resolve` reads a higher-level tile by arithmetic, not existence → `write_higher_levels=false` throws `cannot open file` during proofs | +| C2 | Medium | Durability | `write_file_atomically` never `fsync`s → docs' crash-safety claim is false; a post-crash truncated tile permanently wedges the store | +| C3 | Low | Robustness (public API) | `TreeT::subtree_root` does not validate `level`/`index` → shift-UB at `level≥64`; silent wrong `true` on `lo+count` overflow | +| C4 | Low | Robustness (hostile input) | `largest_pow2_lt` infinite-loops for `size > 2^63`, hanging the public proof entry points | +| C5 | Low | Concurrency / cleanup | Fixed `*.tmp` name: concurrent-writer collision and temp-file leak on error | +| P1 | Low–Med | Performance | No tile read cache: `resolve`/roll-ups re-open & re-deserialize the same 8 KB tiles repeatedly (and the docs claim a cache that does not exist) | +| T1 | Medium | Tests | No corrupt/truncated-file read tests | +| T2 | Medium | Tests | No coverage of level-≥2 tiles (needs > 65,536 leaves) | +| T3 | Medium | Tests | No test with `write_higher_levels=false` (would have caught C1) | +| T4 | Low–Med | Tests | No reopen/resume test (the `full_prefix_length` cursor-resume path) | +| T5 | Low | Tests | No `compact()` with non-zero `retention_margin`; no retract-below-flush-after-margin | +| T6 | Low | Tests | Boundary sizes `0` and `65537` untested | +| D1 | Medium | Docs | Design doc describes an "in-process LRU cache" that does not exist | +| D2 | Low | Docs | Durability wording ("a crash never leaves a half-written tile") is false (see C2) | +| D3 | Low | Docs | Partial-tile / spec material retained as background; correct but worth a sharper "not implemented" caveat | + +--- + +## Correctness & robustness + +### C1 — `TileHashSourceT::resolve` assumes higher-level tiles exist (Medium) + +`merklecpp_tiles.h:636–642`. + +```cpp +const uint64_t full_tiles = full_shift >= 64 ? 0 : (available_size >> full_shift); +if (n < full_tiles) +{ + const std::vector tile = store.read_tile(TileRef{L, n}); // unchecked read + ... +} +``` + +`resolve` infers the presence of a level-`L` (`L ≥ 1`) tile purely from +`available_size`. That inference is valid only with the default +`write_higher_levels = true` (`merklecpp_tiles.h:373`). The option is public and +reachable through `TiledTreeT::Config::writer.write_higher_levels`, and the design +doc (`§6.4`) does not state it is incompatible with proof generation. With it set +to `false`, only level-0 tiles are written, but `resolve` still believes level-≥1 +tiles exist and calls `read_tile`, which **throws** rather than returning `false`, +so the `||` fallback in `CombinedHashSourceT::subtree_root` never engages. + +*Reproduced:* 70,000-leaf tree, `write_higher_levels=false`, +`TileHashSource(store, 69888)`, `engine.root(69888)` → +`cannot open file: .../tile/1/000`. Triggers for any subtree of height ≥ 9 once +the tiled prefix reaches 65,536 leaves. + +*Default path is safe:* `available_size >> (8·(L+1))` was shown (algebraically and +by the oracle sweep) to equal exactly the number of level-`L` tiles on disk when +higher levels are written, so the default never reads a missing tile. + +**Fix.** Gate the read on real presence and fall through to the existing split +path, which already reconstructs the same root from level-0 tiles: +```cpp +if (n < full_tiles && store.has_full_tile(L, n)) { ... } +``` +This makes proofs correct (just slower) regardless of the option. Alternatively, +remove/guard the option or document the incompatibility. + +### C2 — `write_file_atomically` is atomic but not durable; the store can wedge after a crash (Medium) + +`merklecpp_tiles.h:269–302` (`flush()` + `rename`, **no `fsync`** anywhere in the +file). + +`std::ofstream::flush()` only moves bytes into the OS page cache. With no `fsync` +of the temp file before `rename` and no `fsync` of the parent directory after, +a power-loss can persist the rename's directory entry while the file's data blocks +are lost, leaving a tile **present at its final path but zero-length / truncated**. +(ext4's flush-on-rename heuristic does not apply to write-once renames over a +non-existent target and is not portable to XFS/btrfs/NFS.) + +This contradicts the comment at `merklecpp_tiles.h:268` and +`doc/design/tlog-tiles.md:355–356`. + +*Why it bites (reproduced):* the store uses **file existence** as its idempotence +signal — `flush()` skips any tile where `has_full_tile()` is true +(`merklecpp_tiles.h:422`) and `full_prefix_length()` seeds its cursor from +existence (`:476–499`). A present-but-truncated tile is therefore treated as +durably written and **never rewritten**, yet `read_tile`'s exact-size check rejects +it on every read — so every later roll-up or proof over that tile throws +indefinitely until the file is deleted by hand. (Integrity is preserved — no wrong +proof is ever produced — but availability/self-healing is not.) + +**Fix.** `fsync` the temp fd before `rename`, then `fsync` the parent directory +(`FlushFileBuffers` on Windows). And/or treat a wrong-size tile as **absent** on +read so it is rewritten (self-heal). Soften the docs to an atomicity-only claim +(see D2). + +### C3 — `TreeT::subtree_root` does not validate its arguments (Low) + +`merklecpp.h:1333–1336`. + +```cpp +const size_t lo = index << level; // line 1333 +const size_t count = (size_t)1 << level; // line 1334 -- run BEFORE any guard +if (num_leaves() == 0 || lo < min_index() || lo + count > num_leaves()) ... +``` + +`subtree_root` is a public method (used by `MemoryHashSourceT`). It shifts by the +caller-supplied `uint8_t level` *before* any bound check: + +* **`level ≥ 64` → UB.** `index << level` / `1 << level` are shifts past the type + width. *Reproduced under UBSan:* `subtree_root(64, 0, out)` → + `shift exponent 64 is too large`. +* **`lo + count` overflow → silent wrong `true`.** With `level=1` and + `index ≈ 2^63`, `lo + count` wraps to `0`, the upper-bound guard passes, and the + call returns `true` with `out` set to an unrelated node hash (ASan-clean — + memory-safe, but a wrong positive). + +No in-repo caller hits either case (internal `level = log2_exact(w) ≤ 63` or `0`); +this is strictly out-of-contract input to a public API. + +> Related robustness note (no behavioural bug today): unlike the canonical +> `walk_to`, which guards its identical shift with `if (_root->height > 1)` +> (`merklecpp.h:1068`), `subtree_root`'s shift at `:1356` is left unguarded and is +> only safe because the `level==0` early-return plus the `_root->height ≥ +> target_height ≥ 2` guard keep the exponent in `[1,63]`. Verified safe under +> UBSan, but it is a fragile invariant; an explicit guard/assert would harden it. + +**Fix.** First lines of the method: +```cpp +if (level >= sizeof(size_t) * 8) return false; +if (num_leaves() == 0 || index > (num_leaves() >> level)) return false; // overflow-safe +``` + +### C4 — `largest_pow2_lt` hangs for `size > 2^63` (Low) + +`merklecpp_tiles.h:846–854`. + +```cpp +uint64_t k = 1; +while ((k << 1) < n) k <<= 1; // k reaches 2^63, k<<1 overflows to 0, loop never ends +``` + +Reachable from the public `root(size)`, `inclusion_proof(index, size)` and +`consistency_proof(m, n)` (via `mth_range`, *before* any leaf resolution), so an +oversized `size` argument hangs the call. *Reproduced:* `n = 2^63+1`, `2^63+5`, +`2^64−1` never terminate. Not reachable with a real tree (>9.2×10¹⁸ leaves); a +hostile/buggy size only. + +**Fix.** `while (k <= (n - 1) / 2) k <<= 1;` (or stop at `k == (1ull << 63)`). + +### C5 — fixed `*.tmp` name: concurrent collision + leak on error (Low) + +`merklecpp_tiles.h:282` (`tmp += ".tmp"`). + +A single fixed temp name per target means two writers persisting the *same* tile +race on the same `*.tmp`; the second `rename` can fail with `ENOENT` and throw, and +there is a window where one writer renames the other's partially-written temp into +place (corruption-safe only because tile content is deterministic). On a write or +`rename` error the `*.tmp` is also left behind. + +**Fix.** Unique temp suffix (pid + counter/random) and `remove` it on the error +path. + +--- + +## Performance + +### P1 — no tile read cache (Low–Medium) + +There is no caching anywhere in `merklecpp_tiles.h` (confirmed). `resolve` and the +writer's `collect` roll-ups call `store.read_tile`, which `open`s, reads and +deserializes a full 256-hash (8 KB at HASH_SIZE=32) tile **every** call. Near the +tiled/frontier boundary `resolve` descends and re-reads level-0 tiles, and across +many proofs the same hot tiles are re-read from scratch. The design doc assumes +this is mitigated by a cache that does not exist (see D1). + +**Fix.** A small LRU (or per-proof memo) keyed by `{level, index}` over decoded +tiles; or document that callers should wrap `TileStore` with their own cache. + +--- + +## Test coverage gaps + +Empirical validation is otherwise strong: `tiles_proofs.cpp` checks generated +proofs against `TreeT::path/past_path/past_root`, round-trips consistency proofs +through `verify_consistency`, exercises tamper/wrong-root rejection, and is +**exhaustive** over `(m,k)` consistency pairs for `n ≤ 16`. The gaps: + +* **T1 (Medium)** — No corrupt/truncated tile or entry-bundle read tests. The code + *is* defensive (`read_tile` exact-size check; `decode_entries` bounds checks, + fuzzed clean under ASan), but the error paths are unverified by the suite. Only + wrong-width *writes* are tested (`tiles_store.cpp`). +* **T2 (Medium)** — No level-≥2 tiles. The largest tree is 70,000 leaves + (`tiles_proofs.cpp`, `tiles_writer.cpp`), which reaches level-1 only; level-2 + needs > 256² = 65,536 (in practice 256³ ≈ 16.7 M for a full level-2 tile). +* **T3 (Medium)** — No `write_higher_levels=false` test; such a test would have + caught C1. +* **T4 (Low–Med)** — No reopen/resume test: every writer runs once against a fresh + temp dir, so the `full_prefix_length` binary-search resume path is unexercised. +* **T5 (Low)** — `compact()` is only tested with `retention_margin = 0`; no + retract-below-flush case after compaction with a non-zero margin. +* **T6 (Low)** — Boundary sizes `0` and `65537` untested (`1/256/257/65535/65536` + are covered). + +--- + +## Documentation + +* **D1 (Medium)** — `doc/design/tlog-tiles.md:359–360` ("An in-process LRU cache of + recently read tiles avoids repeated I/O") and `:593` ("served from cache") + describe a component that does not exist (see P1). Either implement it or remove + the claim. +* **D2 (Low)** — `doc/design/tlog-tiles.md:355–356` and the `:268` code comment + claim crashes never leave a half-written tile; false without `fsync` (see C2). + Reword to "atomic (no torn writes); durability requires `fsync`". +* **D3 (Low)** — Partial-tile spec material remains as background (`§2.1`, `§4`). + It is consistently caveated as the upstream format and "not produced or read + here", so this is acceptable; a one-line "the implementation has no notion of a + partial tile" near each occurrence would remove any doubt. + +--- + +## Examined and found correct (no action) + +Recorded so the breadth of the review is auditable: + +* **Shift safety** — every shift in `merklecpp_tiles.h` uses 64-bit operands or is + guarded; the `subtree_root` `:1356` shift is provably in `[1,63]` (UBSan-clean). + The only shift defect is C4. +* **Recursion / termination** — `resolve` decrements `level` to a `≤ 8` base + (depth ≤ ~55); `mth_range`/`subproof` strictly shrink their range; + `inclusion_proof` is iterative. No stack risk at 2⁴⁰ leaves. +* **Inclusion proofs** — sibling ranges, `PATH_LEFT/RIGHT` directions, leaf→root + ordering, and `max_index = size − 1` match `PathT::verify`; byte-identical to + `TreeT::path/past_path` across exhaustive index sweeps over the 256/512/768/ + 1024/2048 boundaries. +* **Consistency proofs** — `subproof` matches RFC 6962 SUBPROOF (pow2 and non-pow2 + `m`); `verify_consistency` matches the standard CT verifier (`m==n`, `m==0`, + `m>n`, `is_pow2(m)` prepend, the `fn/sn` reduction, final `sn==0`). Round-trips + for all `m 2³⁰ leaves). `subtree_root` calls `is_full()`, + but the same expression is already on the core insertion path, so this change + neither introduces nor newly reaches it. Flagged for awareness only. + +--- + +## Recommended priority + +1. **C2 + D2** — add `fsync` (or self-heal wrong-size tiles) and correct the + durability docs. Highest real-world risk. +2. **C1 (+ T3)** — make `resolve` gate on `has_full_tile`; add a + `write_higher_levels=false` proof test. +3. **C3, C4** — validate inputs to the public `subtree_root` and bound + `largest_pow2_lt`. Cheap hardening of public entry points. +4. **D1 / P1** — implement a tile cache or drop the doc claim. +5. **T1, T2, T4** — add corruption, level-≥2, and reopen/resume tests. +6. **C5, T5, T6, D3** — opportunistic cleanup. diff --git a/doc/design/tlog-tiles.md b/doc/design/tlog-tiles.md index 8f03ce0..4755e50 100644 --- a/doc/design/tlog-tiles.md +++ b/doc/design/tlog-tiles.md @@ -364,14 +364,11 @@ public: ```cpp class TileWriterT { public: - struct Options { - bool write_higher_levels = true; // roll up L>=1 each flush - }; - - TileWriterT(TileStoreT& store, Options = {}); + explicit TileWriterT(TileStoreT& store); - // Persist all newly-complete full tiles (all levels). No partial tiles are - // written. Incremental: only writes tiles not already present. + // Persist all newly-complete full tiles (all levels 0..63). No partial tiles + // are written. Incremental: only writes tiles not already present. Higher + // levels are always rolled up, since proof generation relies on them. // `leaf_at` supplies level-0 leaf hashes for [0, size) (e.g. Tree::leaf). void write_up_to(uint64_t size, const std::function& leaf_at); @@ -507,7 +504,6 @@ public: std::filesystem::path prefix; uint64_t retention_margin = 0; // keep this many recent leaves resident bool compact_on_flush = false; // opt in to dropping tiled leaves - TileWriterT::Options writer = {}; }; explicit TiledTreeT(Config); diff --git a/merklecpp_tiles.h b/merklecpp_tiles.h index 4ae43e7..1b9abef 100644 --- a/merklecpp_tiles.h +++ b/merklecpp_tiles.h @@ -366,13 +366,6 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) /// @brief Supplies the level-0 leaf hash for a given leaf index. using LeafFn = std::function; - /// @brief Write-path options. - struct Options - { - /// @brief Roll up and write tiles at levels >= 1. - bool write_higher_levels = true; - }; - /// @brief Counts of work performed by a write_up_to call. struct Stats { @@ -381,9 +374,7 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) }; /// @brief Constructs a writer over @p store. - explicit TileWriterT(Store& store, Options options = {}) : - store(store), options(options) - {} + explicit TileWriterT(Store& store) : store(store) {} /// @brief Writes all newly-complete full tiles for a tree of @p size /// leaves. @@ -392,13 +383,16 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) /// [0, size); only ever queried for leaves of complete subtrees. /// @return Counts of tiles written /// @note Incremental: full tiles already on disk are immutable and are - /// never rewritten. The incomplete frontier is never tiled. + /// never rewritten. The incomplete frontier is never tiled. Tiles are + /// always rolled up at every level (0..63), so the on-disk set always + /// contains the higher-level roll-ups that proof generation relies on. Stats write_up_to(uint64_t size, const LeafFn& leaf_at) { Stats stats; - const uint8_t max_level = options.write_higher_levels ? 63 : 0; - for (uint8_t level = 0; level <= max_level; level++) + // tlog-tiles defines levels 0..63; the loop stops early once a level + // has no complete entries (see the entries == 0 break below). + for (uint8_t level = 0; level <= 63; level++) { // Number of complete (balanced) level-L entries available; this // deliberately excludes the incomplete frontier subtree. @@ -442,9 +436,6 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) // NOLINTNEXTLINE(cppcoreguidelines-avoid-const-or-ref-data-members) Store& store; - /// @brief Write-path options. - Options options; - /// @brief Per-level index of the next full tile to write. std::vector next_full; @@ -1031,15 +1022,10 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) /// from memory the leaves already covered by a full tile. Off by /// default: tiles are written but the tree keeps every leaf resident. bool compact_on_flush = false; - - /// @brief Tile writer options. - typename Writer::Options writer = {}; }; explicit TiledTreeT(Config config) : - config(std::move(config)), - store(this->config.prefix), - writer(store, this->config.writer) + config(std::move(config)), store(this->config.prefix), writer(store) {} /// @brief Appends a leaf hash. diff --git a/test/CMakeLists.txt b/test/CMakeLists.txt index 1580beb..edbadf4 100644 --- a/test/CMakeLists.txt +++ b/test/CMakeLists.txt @@ -34,6 +34,7 @@ add_merklecpp_test(tiles_writer tiles_writer.cpp) add_merklecpp_test(tiles_proofs tiles_proofs.cpp) add_merklecpp_test(tiles_tree tiles_tree.cpp) add_merklecpp_test(tiles_entries tiles_entries.cpp) +add_merklecpp_test(tiles_level2 tiles_level2.cpp) add_merklecpp_test(time_tiles time_tiles.cpp) if(TARGET evercrypt.host) diff --git a/test/tiles_level2.cpp b/test/tiles_level2.cpp new file mode 100644 index 0000000..f0e38ab --- /dev/null +++ b/test/tiles_level2.cpp @@ -0,0 +1,138 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. + +// End-to-end coverage of the level-2 tile paths. A full level-2 tile requires +// 256^3 == 16,777,216 leaves, so this is a deliberately large test: it writes +// the tiles for that many leaves (~65k tile files) from a deterministic leaf +// source -- no in-memory tree is built -- and cross-checks the writer's +// level-by-level roll-up against TileHashSourceT::resolve (which reads the +// level-2 tile) and against the underlying leaves. + +#include "util.h" + +#include +#include +#include +#include +#include +#include +#include +#include + +namespace fs = std::filesystem; +using merkle::Hash; +using merkle::tiles::TileHashSource; +using merkle::tiles::TileRef; +using merkle::tiles::TileStore; +using merkle::tiles::TileWriter; +using merkle::tiles::TILE_WIDTH; + +static void expect(bool cond, const std::string& what) +{ + if (!cond) + { + throw std::runtime_error("check failed: " + what); + } +} + +// Roll up a perfect (power-of-two) set of hashes with the default combiner. +static Hash rollup(const std::vector& hashes) +{ + return merkle::tiles::perfect_root<32, merkle::sha256_compress>(hashes); +} + +int main() +{ + const fs::path dir = fs::temp_directory_path() / + ("merklecpp_tiles_level2_" + + std::to_string((unsigned long long)std::time(nullptr))); + + try + { + // 256^3 leaves == exactly one full level-2 tile. + const uint64_t n = + (uint64_t)TILE_WIDTH * (uint64_t)TILE_WIDTH * (uint64_t)TILE_WIDTH; + + TileStore store(dir); + TileWriter writer(store); + + // Deterministic leaf hash derived from the index (low 8 bytes); avoids + // materialising a 16.7M-entry vector. + Hash leaf; + const auto leaf_at = [&](uint64_t i) -> const Hash& { + leaf.zero(); + for (int b = 0; b < 8; b++) + { + leaf.bytes[b] = (uint8_t)(i >> (8 * b)); + } + return leaf; + }; + + const auto stats = writer.write_up_to(n, leaf_at); + + // 65536 full L0 tiles + 256 full L1 tiles + 1 full L2 tile. + expect( + stats.full_written == + (uint64_t)TILE_WIDTH * TILE_WIDTH + TILE_WIDTH + 1, + "level2: tile counts"); + expect(store.has_full_tile(2, 0), "level2: L2 tile present"); + expect(!store.has_full_tile(2, 1), "level2: no second L2 tile"); + expect(!store.has_full_tile(3, 0), "level2: no L3 tile"); + + const auto l2 = store.read_tile(TileRef{2, 0}); + expect(l2.size() == TILE_WIDTH, "level2: L2 tile width"); + + const TileHashSource src(store, n); + + // Each level-2 entry j is the root of level-1 tile j, which rolls up level-0 + // tiles, which are the leaves verbatim. Cross-check the writer's roll-up, + // resolve's level-2 read, and the leaf chain on a sample of indices. + for (const uint64_t j : + {(uint64_t)0, (uint64_t)1, (uint64_t)200, (uint64_t)255}) + { + const auto l1j = store.read_tile(TileRef{1, j}); + expect(l2[j] == rollup(l1j), "level2: L2[j] == rollup(L1 tile j)"); + + // resolve reads the level-2 tile for the 2^16-leaf subtree at index j. + Hash r16; + expect( + src.subtree_root(16, j, r16), "level2: subtree_root(16,j) resolves"); + expect(r16 == l2[j], "level2: resolve(16,j) == L2[j]"); + + // Anchor to leaves: L1 tile j entry 0 == root of L0 tile (j*256), whose + // first entry is leaf (j * 65536). + const auto l0 = store.read_tile(TileRef{0, j * TILE_WIDTH}); + expect(l1j[0] == rollup(l0), "level2: L1[j][0] == rollup(L0 tile)"); + expect( + l0[0] == leaf_at(j * (uint64_t)TILE_WIDTH * TILE_WIDTH), + "level2: L0 entry == leaf"); + } + + // Intra-tile roll-up: the 2^17-leaf subtree at 0 hashes L2[0] and L2[1]. + Hash r17; + expect(src.subtree_root(17, 0, r17), "level2: subtree_root(17,0) resolves"); + expect( + r17 == rollup({l2[0], l2[1]}), "level2: resolve(17,0) == H(L2[0],L2[1])"); + + std::cout << "tiles_level2: OK" << '\n'; + + std::error_code ec; + fs::remove_all(dir, ec); + } + catch (std::exception& ex) + { + std::cout << "Error: " << ex.what() << '\n'; + std::error_code ec; + fs::remove_all(dir, ec); + return 1; + } + catch (...) + { + std::cout << "Error" << '\n'; + std::error_code ec; + fs::remove_all(dir, ec); + return 1; + } + + return 0; +} diff --git a/test/tiles_proofs.cpp b/test/tiles_proofs.cpp index 6d0733a..2bcbe0a 100644 --- a/test/tiles_proofs.cpp +++ b/test/tiles_proofs.cpp @@ -197,7 +197,7 @@ int main() try { - const auto hashes = make_hashes(70000); + const auto hashes = make_hashes(300000); for (const uint64_t n : {(uint64_t)1, @@ -218,9 +218,17 @@ int main() } std::cout << "small/medium sizes: OK" << '\n'; - // Large tree: exercises full L1 tiles and the in-memory frontier. - check_size(base / "big", 70000, hashes); - std::cout << "size 70000: OK" << '\n'; + // Large trees. 65536 == 256 full L0 tiles == one full L1 tile (exact L1 + // boundary); 65537 is one past it; 70000 exercises a full L1 tile plus an + // in-memory frontier; 300000 forces proofs over height->=16 subtrees, so + // TileHashSource::resolve descends through level-2 logic (full_shift = 24) + // before reaching the level-1 tiles -- the only coverage of the L>=2 path. + for (const uint64_t n : + {(uint64_t)65536, (uint64_t)65537, (uint64_t)70000, (uint64_t)300000}) + { + check_size(base / ("big" + std::to_string(n)), n, hashes); + std::cout << "size " << n << ": OK" << '\n'; + } std::cout << "tiles_proofs: OK" << '\n'; diff --git a/test/tiles_store.cpp b/test/tiles_store.cpp index d0445f1..a23a3a7 100644 --- a/test/tiles_store.cpp +++ b/test/tiles_store.cpp @@ -7,6 +7,8 @@ #include #include #include +#include +#include #include #include #include @@ -40,6 +42,15 @@ static std::string rel(const merkle::tiles::TileStore& store, const fs::path& p) return p.lexically_relative(store.root()).generic_string(); } +// Overwrites a file with exactly the given bytes (to simulate corruption). +static void overwrite_file(const fs::path& p, const std::vector& bytes) +{ + std::ofstream f(p, std::ios::binary | std::ios::trunc); + f.write( + reinterpret_cast(bytes.data()), + (std::streamsize)bytes.size()); +} + int main() { const auto seed = std::time(nullptr); @@ -110,6 +121,59 @@ int main() } expect(threw, "width mismatch rejected"); + // 3c. Corrupt / truncated files are rejected on read (integrity check), so + // a torn write can never be served as a valid tile or bundle. + { + const auto expect_throws = + [](const std::function& fn, const std::string& what) { + bool t = false; + try + { + fn(); + } + catch (const std::exception&) + { + t = true; + } + expect(t, what); + }; + + // Truncated tile: fewer bytes than a full tile. + overwrite_file(store.tile_path(full_ref), std::vector(hsz, 0)); + expect_throws( + [&] { (void)store.read_tile(full_ref); }, "truncated tile rejected"); + + // Oversized tile: more bytes than a full tile. + overwrite_file( + store.tile_path(full_ref), + std::vector((merkle::tiles::TILE_WIDTH + 1) * hsz, 0)); + expect_throws( + [&] { (void)store.read_tile(full_ref); }, "oversized tile rejected"); + + // A valid full entry bundle whose file is then cut short. + std::vector> entries(merkle::tiles::TILE_WIDTH); + for (size_t i = 0; i < entries.size(); i++) + { + entries[i] = {(uint8_t)i, 0x7F}; + } + store.write_entry_bundle(0, entries); + expect( + store.read_entry_bundle(0).size() == merkle::tiles::TILE_WIDTH, + "bundle valid before truncation"); + // Claims a 5-byte entry but supplies only one trailing byte. + overwrite_file(store.entries_path(0), {0x00, 0x05, 0x01}); + expect_throws( + [&] { (void)store.read_entry_bundle(0); }, + "truncated entry bundle rejected"); + + // decode_entries directly: a length prefix that overruns the buffer. + expect_throws( + [] { + (void)merkle::tiles::TileStore::decode_entries({0xFF, 0xFF, 0x00}, 1); + }, + "decode_entries oversized length rejected"); + } + std::cout << "tiles_store: OK" << '\n'; std::error_code ec; diff --git a/test/tiles_tree.cpp b/test/tiles_tree.cpp index 3baca19..7706e5a 100644 --- a/test/tiles_tree.cpp +++ b/test/tiles_tree.cpp @@ -119,6 +119,29 @@ int main() { const auto hashes = make_hashes(1500); + // ---- Part 0: the empty tree. flush()/compact() are no-ops and there is no + // root. + { + TiledTree::Config ecfg; + ecfg.prefix = base / "tt_empty"; + TiledTree ett(ecfg); + expect(ett.size() == 0, "empty: size 0"); + expect(ett.flush().full_written == 0, "empty: flush writes nothing"); + expect(ett.flushed_size() == 0, "empty: flushed size 0"); + expect(ett.compact() == 0, "empty: compact no-op"); + bool ethrew = false; + try + { + (void)ett.root(); + } + catch (const std::exception&) + { + ethrew = true; + } + expect(ethrew, "empty: root throws"); + std::cout << "empty tree: OK" << '\n'; + } + // ---- Part 1: memory-source proofs (exercises TreeT::subtree_root). for (const uint64_t n : {(uint64_t)1, @@ -296,6 +319,64 @@ int main() std::cout << "tiled tree (exact-multiple compaction): OK" << '\n'; } + // ---- Part 2c: compaction with a non-zero retention_margin keeps the most + // recent leaves resident while flushed indices are still served from + // tiles. The immutable prefix remains the full-tile prefix, regardless + // of the margin. + { + TiledTree::Config rcfg; + rcfg.prefix = base / "tt_margin"; + rcfg.retention_margin = 300; + rcfg.compact_on_flush = true; + TiledTree rtt(rcfg); + for (uint64_t i = 0; i < n1; i++) // n1 == 1000 + { + rtt.append(hashes[i]); + } + rtt.flush(); // covered = 768; target = floor((768 - 300) / 256) * 256 = 256 + expect(rtt.flushed_size() == 768, "margin: flushed size 768"); + expect( + rtt.tree_ref().min_index() == 256, + "margin: retained >= 300 recent leaves"); + + for (uint64_t i = n1; i < N; i++) + { + rtt.append(hashes[i]); + } + expect(rtt.root() == ref_root, "margin: root matches reference"); + + // Flushed-only (0, 255), flushed-but-resident overlap (256, 767), and the + // un-flushed frontier (1000, 1499). + for (const uint64_t i : + {(uint64_t)0, + (uint64_t)255, + (uint64_t)256, + (uint64_t)767, + (uint64_t)1000, + (uint64_t)1499}) + { + const auto p = rtt.inclusion_proof(i, N); + expect( + *p == *ref.path(i), "margin inclusion==ref i=" + std::to_string(i)); + expect( + p->verify(ref_root), + "margin inclusion verify i=" + std::to_string(i)); + } + + bool threw = false; + try + { + rtt.retract_to(700); // size 701 < flushed_size 768 + } + catch (const std::exception&) + { + threw = true; + } + expect(threw, "margin: retract below flushed prefix throws"); + + std::cout << "tiled tree (retention margin): OK" << '\n'; + } + // ---- Part 3: rollback. Tiles are immutable, so only un-tiled // (post-flush) entries may be rolled back. { diff --git a/test/tiles_writer.cpp b/test/tiles_writer.cpp index f8ebaa2..f7f9e5f 100644 --- a/test/tiles_writer.cpp +++ b/test/tiles_writer.cpp @@ -203,6 +203,54 @@ int main() std::cout << "D (full L1 tile): OK" << '\n'; } + // ---- E. resume: a fresh writer over an existing store rebuilds its cursor + // from disk via full_prefix_length, never rewriting full tiles and + // writing only the newly-complete ones. + { + const auto hashes = make_hashes(70000); + const auto leaf_at = [&](uint64_t i) -> const Hash& { return hashes[i]; }; + + const fs::path dir = base / "e"; + { + TileStore store(dir); + TileWriter writer(store); + expect( + writer.write_up_to(600, leaf_at).full_written == 2, + "E first writer 2 L0 tiles"); // 600 / 256 == 2 full L0 tiles + } + + // A brand-new store + writer over the same directory: its cursor must + // resume from what is already on disk. + TileStore store(dir); + TileWriter writer(store); + expect( + writer.write_up_to(600, leaf_at).full_written == 0, + "E resume rewrites nothing"); + + // Extending to 70000 writes only the missing tiles: 273 - 2 == 271 new L0 + // plus 1 new L1 (which rolls up the L0 tiles the first writer wrote). + expect( + writer.write_up_to(70000, leaf_at).full_written == 272, + "E resume writes only new tiles"); + expect(store.has_full_tile(0, 0), "E L0 tile 0 still present"); + expect(store.has_full_tile(0, 272), "E L0 tile 272"); + expect(store.has_full_tile(1, 0), "E L1 tile 0"); + expect( + store.read_tile(TileRef{1, 0})[0] == + rollup(store.read_tile(TileRef{0, 0})), + "E L1[0] rolled up from resumed L0"); + expect(!any_partial_dirs(store), "E no partial tiles"); + + // A third fresh writer confirms full idempotence after a resume. + TileStore store3(dir); + TileWriter writer3(store3); + expect( + writer3.write_up_to(70000, leaf_at).full_written == 0, + "E second resume idempotent"); + + std::cout << "E (writer resume): OK" << '\n'; + } + std::cout << "tiles_writer: OK" << '\n'; std::error_code ec; From 577b4e77bf7cb2b5bcd4e9d014bb56345dd0a2fd Mon Sep 17 00:00:00 2001 From: achamayou Date: Thu, 2 Jul 2026 10:17:49 +0000 Subject: [PATCH 17/30] Harden tile storage --- audit.md | 149 +++++++-------- doc/design/tlog-tiles.md | 15 +- doc/tiles-guide.md | 9 +- merklecpp.h | 36 ++-- merklecpp_tiles.h | 313 +++++++++++++++++++++++++++++--- test/compare_hash_functions.cpp | 10 +- test/coverage.cpp | 15 +- test/demo_tree.cpp | 13 +- test/flush.cpp | 4 +- test/partial_serialisation.cpp | 10 +- test/past_paths.cpp | 20 +- test/past_root.cpp | 13 +- test/paths.cpp | 18 +- test/retract.cpp | 11 +- test/serialisation.cpp | 16 +- test/tiles_level2.cpp | 12 +- test/tiles_store.cpp | 53 +++++- test/tiles_tree.cpp | 38 +++- test/tiles_writer.cpp | 6 +- test/time_large_trees.cpp | 22 ++- test/unit_tests.cpp | 4 +- test/util.h | 6 +- 22 files changed, 584 insertions(+), 209 deletions(-) diff --git a/audit.md b/audit.md index 8f86483..c64cd96 100644 --- a/audit.md +++ b/audit.md @@ -29,8 +29,13 @@ and (c) several **robustness / hostile-input / test / doc** gaps. > plus level-2 `resolve` descend coverage in the `tiles_proofs` oracle sweep > (sizes 65536/65537/300000), writer reopen/resume (`tiles_writer` §E), > non-zero `retention_margin` compaction (`tiles_tree` Part 2c), and the empty -> tree (`tiles_tree` Part 0). Still open for a future change: **C2/D2** (fsync -> durability), **C3/C4/C5** (robustness hardening), **P1/D1** (tile cache). +> tree (`tiles_tree` Part 0). +> +> **Update (hardening commit).** The remaining implementation findings are now +> addressed: **C2/D2** by synced unique-temp writes, parent-directory sync on +> POSIX, and size-validated tile presence; **C3/C4** by overflow-safe public +> arithmetic guards; **C5** by unique temp names and cleanup; and **P1/D1** by a +> per-source tile read cache plus corrected docs. --- @@ -38,20 +43,20 @@ and (c) several **robustness / hostile-input / test / doc** gaps. | # | Severity | Area | Summary | |---|----------|------|---------| -| C1 | Medium | Correctness (config) | `resolve` reads a higher-level tile by arithmetic, not existence → `write_higher_levels=false` throws `cannot open file` during proofs | -| C2 | Medium | Durability | `write_file_atomically` never `fsync`s → docs' crash-safety claim is false; a post-crash truncated tile permanently wedges the store | -| C3 | Low | Robustness (public API) | `TreeT::subtree_root` does not validate `level`/`index` → shift-UB at `level≥64`; silent wrong `true` on `lo+count` overflow | -| C4 | Low | Robustness (hostile input) | `largest_pow2_lt` infinite-loops for `size > 2^63`, hanging the public proof entry points | -| C5 | Low | Concurrency / cleanup | Fixed `*.tmp` name: concurrent-writer collision and temp-file leak on error | -| P1 | Low–Med | Performance | No tile read cache: `resolve`/roll-ups re-open & re-deserialize the same 8 KB tiles repeatedly (and the docs claim a cache that does not exist) | -| T1 | Medium | Tests | No corrupt/truncated-file read tests | -| T2 | Medium | Tests | No coverage of level-≥2 tiles (needs > 65,536 leaves) | -| T3 | Medium | Tests | No test with `write_higher_levels=false` (would have caught C1) | -| T4 | Low–Med | Tests | No reopen/resume test (the `full_prefix_length` cursor-resume path) | -| T5 | Low | Tests | No `compact()` with non-zero `retention_margin`; no retract-below-flush-after-margin | -| T6 | Low | Tests | Boundary sizes `0` and `65537` untested | -| D1 | Medium | Docs | Design doc describes an "in-process LRU cache" that does not exist | -| D2 | Low | Docs | Durability wording ("a crash never leaves a half-written tile") is false (see C2) | +| C1 | Medium | Correctness (config) | **Fixed:** the `write_higher_levels` option was removed, so required roll-up tiles are always written | +| C2 | Medium | Durability | **Fixed:** `write_file_atomically` syncs unique temp files, atomically replaces, syncs POSIX parent dirs, and bad-size tiles are rewritten | +| C3 | Low | Robustness (public API) | **Fixed:** `TreeT::subtree_root` validates `level`/`index` before shifting and uses overflow-safe range checks | +| C4 | Low | Robustness (hostile input) | **Fixed:** `largest_pow2_lt` uses an overflow-safe loop condition for `size > 2^63` | +| C5 | Low | Concurrency / cleanup | **Fixed:** temp file names are unique per process/time/counter and cleaned up on error | +| P1 | Low–Med | Performance | **Fixed:** `TileHashSource` keeps a small decoded tile cache for proof generation | +| T1 | Medium | Tests | **Fixed:** corrupt/truncated tile and entry-bundle read tests added | +| T2 | Medium | Tests | **Fixed:** level-2 coverage added via `tiles_level2` and large proof sweeps | +| T3 | Medium | Tests | **Obsolete:** the `write_higher_levels` flag was removed with C1 | +| T4 | Low–Med | Tests | **Fixed:** reopen/resume coverage added for `full_prefix_length` cursor recovery | +| T5 | Low | Tests | **Fixed:** non-zero `retention_margin` compaction and guarded rollback are covered | +| T6 | Low | Tests | **Fixed:** empty tree and `65537` boundary coverage added | +| D1 | Medium | Docs | **Fixed:** design doc now matches the per-source tile cache | +| D2 | Low | Docs | **Fixed:** durability wording now describes synced temp-file atomic replace and bad-size tile rewrite | | D3 | Low | Docs | Partial-tile / spec material retained as background; correct but worth a sharper "not implemented" caveat | --- @@ -62,6 +67,10 @@ and (c) several **robustness / hostile-input / test / doc** gaps. `merklecpp_tiles.h:636–642`. +> **Status:** Fixed by removing the public `write_higher_levels` option. Higher +> level roll-up tiles are now always written, so the arithmetic existence +> invariant used by `resolve` is maintained. + ```cpp const uint64_t full_tiles = full_shift >= 64 ? 0 : (available_size >> full_shift); if (n < full_tiles) @@ -99,35 +108,25 @@ remove/guard the option or document the incompatibility. ### C2 — `write_file_atomically` is atomic but not durable; the store can wedge after a crash (Medium) -`merklecpp_tiles.h:269–302` (`flush()` + `rename`, **no `fsync`** anywhere in the -file). - -`std::ofstream::flush()` only moves bytes into the OS page cache. With no `fsync` -of the temp file before `rename` and no `fsync` of the parent directory after, -a power-loss can persist the rename's directory entry while the file's data blocks -are lost, leaving a tile **present at its final path but zero-length / truncated**. -(ext4's flush-on-rename heuristic does not apply to write-once renames over a -non-existent target and is not portable to XFS/btrfs/NFS.) - -This contradicts the comment at `merklecpp_tiles.h:268` and -`doc/design/tlog-tiles.md:355–356`. - -*Why it bites (reproduced):* the store uses **file existence** as its idempotence -signal — `flush()` skips any tile where `has_full_tile()` is true -(`merklecpp_tiles.h:422`) and `full_prefix_length()` seeds its cursor from -existence (`:476–499`). A present-but-truncated tile is therefore treated as -durably written and **never rewritten**, yet `read_tile`'s exact-size check rejects -it on every read — so every later roll-up or proof over that tile throws -indefinitely until the file is deleted by hand. (Integrity is preserved — no wrong -proof is ever produced — but availability/self-healing is not.) - -**Fix.** `fsync` the temp fd before `rename`, then `fsync` the parent directory -(`FlushFileBuffers` on Windows). And/or treat a wrong-size tile as **absent** on -read so it is rewritten (self-heal). Soften the docs to an atomicity-only claim -(see D2). +> **Status:** Fixed by the hardening commit. `write_file_atomically` now writes a +> unique temp file, syncs it, publishes it with atomic replace, syncs the POSIX +> parent directory, and removes the temp file on error. `has_full_tile` also +> requires the exact full-tile byte size, so a truncated/oversized tile is +> rewritten instead of treated as durable. + +The original implementation used stream flush plus rename and treated file +existence as the idempotence signal. A crash could therefore leave a present but +wrong-size tile that future writes skipped and future reads rejected. The current +implementation syncs the temp file before publishing, syncs POSIX parent +directories after publishing, and makes `has_full_tile` require the exact full +tile size so wrong-size tiles are rewritten. ### C3 — `TreeT::subtree_root` does not validate its arguments (Low) +> **Status:** Fixed by the hardening commit. The method now rejects unsupported +> levels and overflowing indices before shifting, and uses an overflow-safe +> `lo/count/leaves` containment check. + `merklecpp.h:1333–1336`. ```cpp @@ -165,6 +164,10 @@ if (num_leaves() == 0 || index > (num_leaves() >> level)) return false; // overf ### C4 — `largest_pow2_lt` hangs for `size > 2^63` (Low) +> **Status:** Fixed by the hardening commit. The loop condition is now based on +> `(n - 1) / 2`, so it cannot shift past `2^63` while searching for the largest +> power of two below `n`. + `merklecpp_tiles.h:846–854`. ```cpp @@ -182,6 +185,10 @@ hostile/buggy size only. ### C5 — fixed `*.tmp` name: concurrent collision + leak on error (Low) +> **Status:** Fixed by the hardening commit. Temp names now include process, +> timestamp, and an atomic counter, and a guard removes the temp path unless the +> atomic replace succeeds. + `merklecpp_tiles.h:282` (`tmp += ".tmp"`). A single fixed temp name per target means two writers persisting the *same* tile @@ -199,15 +206,13 @@ path. ### P1 — no tile read cache (Low–Medium) -There is no caching anywhere in `merklecpp_tiles.h` (confirmed). `resolve` and the -writer's `collect` roll-ups call `store.read_tile`, which `open`s, reads and -deserializes a full 256-hash (8 KB at HASH_SIZE=32) tile **every** call. Near the -tiled/frontier boundary `resolve` descends and re-reads level-0 tiles, and across -many proofs the same hot tiles are re-read from scratch. The design doc assumes -this is mitigated by a cache that does not exist (see D1). +> **Status:** Fixed for proof generation by the hardening commit. `TileHashSource` +> now keeps a small decoded tile cache and serves repeated reads from memory. -**Fix.** A small LRU (or per-proof memo) keyed by `{level, index}` over decoded -tiles; or document that callers should wrap `TileStore` with their own cache. +The original implementation opened, read, and deserialized a full 256-hash tile +for every `resolve` call, and the design doc described a cache that had not yet +been implemented. `TileHashSource` now keeps a small decoded tile cache keyed by +`{level, index}` and reuses hot tiles during proof generation. --- @@ -218,33 +223,20 @@ proofs against `TreeT::path/past_path/past_root`, round-trips consistency proofs through `verify_consistency`, exercises tamper/wrong-root rejection, and is **exhaustive** over `(m,k)` consistency pairs for `n ≤ 16`. The gaps: -* **T1 (Medium)** — No corrupt/truncated tile or entry-bundle read tests. The code - *is* defensive (`read_tile` exact-size check; `decode_entries` bounds checks, - fuzzed clean under ASan), but the error paths are unverified by the suite. Only - wrong-width *writes* are tested (`tiles_store.cpp`). -* **T2 (Medium)** — No level-≥2 tiles. The largest tree is 70,000 leaves - (`tiles_proofs.cpp`, `tiles_writer.cpp`), which reaches level-1 only; level-2 - needs > 256² = 65,536 (in practice 256³ ≈ 16.7 M for a full level-2 tile). -* **T3 (Medium)** — No `write_higher_levels=false` test; such a test would have - caught C1. -* **T4 (Low–Med)** — No reopen/resume test: every writer runs once against a fresh - temp dir, so the `full_prefix_length` binary-search resume path is unexercised. -* **T5 (Low)** — `compact()` is only tested with `retention_margin = 0`; no - retract-below-flush case after compaction with a non-zero margin. -* **T6 (Low)** — Boundary sizes `0` and `65537` untested (`1/256/257/65535/65536` - are covered). +The historical gaps listed in the first audit pass are now covered: corrupt and +wrong-size tile/bundle files, level-2 tiles, reopen/resume, non-zero retention +margin, guarded rollback below the flushed prefix, empty trees, and `65537` / +large-boundary proof cases. The `write_higher_levels=false` test became obsolete +when that option was removed. --- ## Documentation -* **D1 (Medium)** — `doc/design/tlog-tiles.md:359–360` ("An in-process LRU cache of - recently read tiles avoids repeated I/O") and `:593` ("served from cache") - describe a component that does not exist (see P1). Either implement it or remove - the claim. -* **D2 (Low)** — `doc/design/tlog-tiles.md:355–356` and the `:268` code comment - claim crashes never leave a half-written tile; false without `fsync` (see C2). - Reword to "atomic (no torn writes); durability requires `fsync`". +* **D1 (Medium)** — Fixed: `doc/design/tlog-tiles.md` now describes the + implemented per-source tile cache. +* **D2 (Low)** — Fixed: `doc/design/tlog-tiles.md` and the code comment now + describe synced unique-temp writes, atomic replace, and bad-size tile rewrite. * **D3 (Low)** — Partial-tile spec material remains as background (`§2.1`, `§4`). It is consistently caveated as the upstream format and "not produced or read here", so this is acceptable; a one-line "the implementation has no notion of a @@ -295,12 +287,7 @@ Recorded so the breadth of the review is auditable: ## Recommended priority -1. **C2 + D2** — add `fsync` (or self-heal wrong-size tiles) and correct the - durability docs. Highest real-world risk. -2. **C1 (+ T3)** — make `resolve` gate on `has_full_tile`; add a - `write_higher_levels=false` proof test. -3. **C3, C4** — validate inputs to the public `subtree_root` and bound - `largest_pow2_lt`. Cheap hardening of public entry points. -4. **D1 / P1** — implement a tile cache or drop the doc claim. -5. **T1, T2, T4** — add corruption, level-≥2, and reopen/resume tests. -6. **C5, T5, T6, D3** — opportunistic cleanup. +All implementation, test, and documentation findings from this audit have now +been addressed except **D3**, which remains a low-priority wording polish item: +the design doc still includes partial-tile spec background, but consistently +caveats that this implementation does not write or read partial tiles. diff --git a/doc/design/tlog-tiles.md b/doc/design/tlog-tiles.md index 4755e50..448b95a 100644 --- a/doc/design/tlog-tiles.md +++ b/doc/design/tlog-tiles.md @@ -348,15 +348,18 @@ public: // Tiles bool has_full_tile(uint8_t level, uint64_t index) const; std::vector read_tile(const TileRef&) const; // 256 entries - void write_tile(const TileRef&, const std::vector&); // atomic (tmp+rename) + void write_tile(const TileRef&, const std::vector&); // synced atomic replace }; ``` -- Writes are **atomic** (write to a temp file, `rename`) so a crash never leaves - a half-written immutable tile. +- Writes use a unique temporary file, sync the file contents, publish with an + atomic replace, and on POSIX sync the parent directory after the rename. A + crash should leave either the old tile or the new complete tile visible; a + wrong-size tile is not treated as a durable full tile and is rewritten by the + writer. - Full tiles are written once and never rewritten (immutability). No partial tiles are produced, so every file under `tile//` is write-once. -- An in-process LRU cache of recently read tiles avoids repeated I/O during +- A small in-process cache of recently read tiles avoids repeated I/O during proof generation. ### 6.4 Write path — `TileWriterT` (progressive, on compaction) @@ -586,8 +589,8 @@ relevant was flushed), or from the combination — satisfying the request. Cost per flush is `O(new full tiles)`; higher-level tiles are cheap roll-ups of 256 child hashes. Proof generation is `O(log(size))` `mth_range` calls, each at -most a few tile reads plus a `≤ 256`-leaf roll-up, served from cache in the -common case. +most a few tile reads plus a `≤ 256`-leaf roll-up, with repeated tile reads +served from the per-source cache in the common case. --- diff --git a/doc/tiles-guide.md b/doc/tiles-guide.md index fa4e2c7..294caef 100644 --- a/doc/tiles-guide.md +++ b/doc/tiles-guide.md @@ -24,7 +24,8 @@ page is a practical how-to. ## Requirements and a note on hashing -- C++17 (the header uses `` and ``). +- C++17 (the header uses `` and small platform-specific file-sync + calls for durable tile writes). - Include the companion header; it pulls in `merklecpp.h` for you: ```cpp @@ -79,6 +80,12 @@ complete since the previous call. Full tiles are immutable — written once and never rewritten — and the incomplete frontier is never tiled (it stays in memory until it grows into a full tile). +Tile files are written through unique temporary files, synced, then published +with an atomic replace. On POSIX systems the parent directory is also synced +after the rename. A wrong-size tile file is not considered durable by the +writer, so a later flush rewrites it instead of treating mere file existence as +success. + ## Flushing and compaction By default `flush()` only *writes* tiles; it keeps every leaf resident in diff --git a/merklecpp.h b/merklecpp.h index a4cee98..274d92a 100644 --- a/merklecpp.h +++ b/merklecpp.h @@ -10,6 +10,7 @@ #include #include #include +#include #include #include #include @@ -142,8 +143,9 @@ namespace merkle { uint8_t high = 0; uint8_t low = 0; - if (!decode_hex_digit(s[2 * i], high) || - !decode_hex_digit(s[2 * i + 1], low)) + if ( + !decode_hex_digit(s[2 * i], high) || + !decode_hex_digit(s[2 * i + 1], low)) { throw std::runtime_error("invalid hash string"); } @@ -454,14 +456,13 @@ namespace merkle /// @brief The size of the serialised path in number of bytes [[nodiscard]] size_t serialised_size() const { - return sizeof(_leaf) + - sizeof(uint64_t) + // leaf index - sizeof(uint64_t) + // max index - sizeof(uint64_t) + // number of elements - elements.size() * ( - sizeof(Element::hash) + // hash - sizeof(uint8_t) // direction - ); + return sizeof(_leaf) + sizeof(uint64_t) + // leaf index + sizeof(uint64_t) + // max index + sizeof(uint64_t) + // number of elements + elements.size() * + (sizeof(Element::hash) + // hash + sizeof(uint8_t) // direction + ); } /// @brief Index of the leaf of the path @@ -1330,10 +1331,20 @@ namespace merkle /// [index << level, (index + 1) << level). bool subtree_root(uint8_t level, size_t index, Hash& out) { + const size_t leaves = num_leaves(); + if (leaves == 0 || level >= std::numeric_limits::digits) + { + return false; + } + if (index > (std::numeric_limits::max() >> level)) + { + return false; + } + const size_t lo = index << level; const size_t count = (size_t)1 << level; - if (num_leaves() == 0 || lo < min_index() || lo + count > num_leaves()) + if (lo < min_index() || count > leaves || lo > leaves - count) { return false; } @@ -1775,8 +1786,7 @@ namespace merkle protected: void validate_partial_range(size_t from, size_t to) const { - if ( - empty() || !(min_index() <= from && from <= to && to <= max_index())) + if (empty() || !(min_index() <= from && from <= to && to <= max_index())) { throw std::runtime_error("invalid leaf indices"); } diff --git a/merklecpp_tiles.h b/merklecpp_tiles.h index 1b9abef..cd7c505 100644 --- a/merklecpp_tiles.h +++ b/merklecpp_tiles.h @@ -5,16 +5,33 @@ #include "merklecpp.h" +#include +#include +#include +#include #include #include +#include #include #include #include +#include #include #include +#include #include #include +#ifdef _WIN32 +# ifndef NOMINMAX +# define NOMINMAX +# endif +# include +#else +# include +# include +#endif + // Tiled storage for merklecpp trees, following the C2SP tlog-tiles layout // (https://c2sp.org/tlog-tiles). The tile geometry, path encoding and partial // tile rules match the specification; the hash values stored in tiles are @@ -126,7 +143,15 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) /// @brief Whether a full tile exists on disk. [[nodiscard]] bool has_full_tile(uint8_t level, uint64_t index) const { - return std::filesystem::exists(tile_path(TileRef{level, index})); + std::error_code ec; + const auto path = tile_path(TileRef{level, index}); + if (!std::filesystem::is_regular_file(path, ec) || ec) + { + return false; + } + return std::filesystem::file_size(path, ec) == + (uintmax_t)TILE_WIDTH * HASH_SIZE && + !ec; } /// @brief Writes a tile to disk atomically. @@ -264,8 +289,10 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) std::istreambuf_iterator(f), std::istreambuf_iterator()}; } - /// @brief Writes a file atomically via a temporary file and rename. - /// @note Immutable tiles are never left half-written after a crash. + /// @brief Writes a file atomically via a synced temporary file. + /// @note Uses unique temp names, cleans them up on errors, and syncs the + /// file before publishing it with an atomic replace. POSIX builds also + /// sync the parent directory after rename. static void write_file_atomically( const std::filesystem::path& path, const std::vector& bytes) { @@ -278,27 +305,237 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) ec.message()); } + const std::filesystem::path tmp = temp_path(path); + TempFileGuard guard(tmp); + write_and_sync_file(tmp, bytes); + replace_file(tmp, path); + sync_directory(path.parent_path()); + guard.dismiss(); + } + + class TempFileGuard + { + public: + explicit TempFileGuard(std::filesystem::path path) : + path(std::move(path)) + {} + + ~TempFileGuard() + { + if (active) + { + std::error_code ec; + std::filesystem::remove(path, ec); + } + } + + void dismiss() + { + active = false; + } + + private: + std::filesystem::path path; + bool active = true; + }; + + static std::filesystem::path temp_path(const std::filesystem::path& path) + { + static std::atomic counter{0}; + const auto stamp = + std::chrono::steady_clock::now().time_since_epoch().count(); std::filesystem::path tmp = path; - tmp += ".tmp"; + tmp += ".tmp." + std::to_string(process_id()) + "." + + std::to_string((uint64_t)stamp) + "." + + std::to_string(counter.fetch_add(1, std::memory_order_relaxed)); + return tmp; + } + + static uint64_t process_id() + { +#ifdef _WIN32 + return (uint64_t)GetCurrentProcessId(); +#else + return (uint64_t)::getpid(); +#endif + } + + static std::string system_error_message(const std::string& what) + { +#ifdef _WIN32 + return what + ": error " + std::to_string(GetLastError()); +#else + return what + ": " + std::strerror(errno); +#endif + } + + static void write_and_sync_file( + const std::filesystem::path& path, const std::vector& bytes) + { +#ifdef _WIN32 + HANDLE handle = CreateFileW( + path.wstring().c_str(), + GENERIC_WRITE, + 0, + nullptr, + CREATE_ALWAYS, + FILE_ATTRIBUTE_NORMAL, + nullptr); + if (handle == INVALID_HANDLE_VALUE) + { + throw std::runtime_error("cannot open file: " + path.string()); + } + bool close_handle = true; + try { - std::ofstream f(tmp, std::ios::binary | std::ios::trunc); - if (!f.good()) + size_t written = 0; + while (written < bytes.size()) { - throw std::runtime_error("cannot open file: " + tmp.string()); + const auto remaining = bytes.size() - written; + const auto chunk = (DWORD)std::min( + remaining, (size_t)std::numeric_limits::max()); + DWORD done = 0; + if (!WriteFile( + handle, bytes.data() + written, chunk, &done, nullptr)) + { + throw std::runtime_error( + system_error_message("error writing file " + path.string())); + } + written += done; + } + if (!FlushFileBuffers(handle)) + { + throw std::runtime_error( + system_error_message("error syncing file " + path.string())); } - if (!bytes.empty()) + close_handle = false; + if (!CloseHandle(handle)) { - f.write( - reinterpret_cast(bytes.data()), - (std::streamsize)bytes.size()); + throw std::runtime_error( + system_error_message("error closing file " + path.string())); } - f.flush(); - if (!f.good()) + } + catch (...) + { + if (close_handle) { - throw std::runtime_error("error writing file: " + tmp.string()); + CloseHandle(handle); } + throw; } - std::filesystem::rename(tmp, path); +#else + int flags = O_WRONLY | O_CREAT | O_TRUNC; +# ifdef O_CLOEXEC + flags |= O_CLOEXEC; +# endif + int fd = ::open(path.c_str(), flags, 0666); + if (fd < 0) + { + throw std::runtime_error( + system_error_message("cannot open file " + path.string())); + } + try + { + size_t written = 0; + while (written < bytes.size()) + { + const ssize_t done = + ::write(fd, bytes.data() + written, bytes.size() - written); + if (done < 0) + { + if (errno == EINTR) + { + continue; + } + throw std::runtime_error( + system_error_message("error writing file " + path.string())); + } + if (done == 0) + { + throw std::runtime_error("short write: " + path.string()); + } + written += (size_t)done; + } + if (::fsync(fd) != 0) + { + throw std::runtime_error( + system_error_message("error syncing file " + path.string())); + } + if (::close(fd) != 0) + { + fd = -1; + throw std::runtime_error( + system_error_message("error closing file " + path.string())); + } + fd = -1; + } + catch (...) + { + if (fd >= 0) + { + ::close(fd); + } + throw; + } +#endif + } + + static void replace_file( + const std::filesystem::path& tmp, const std::filesystem::path& path) + { +#ifdef _WIN32 + if (!MoveFileExW( + tmp.wstring().c_str(), + path.wstring().c_str(), + MOVEFILE_REPLACE_EXISTING | MOVEFILE_WRITE_THROUGH)) + { + throw std::runtime_error(system_error_message( + "cannot rename temp file " + tmp.string() + " to " + + path.string())); + } +#else + std::error_code ec; + std::filesystem::rename(tmp, path, ec); + if (ec) + { + throw std::runtime_error( + "cannot rename temp file " + tmp.string() + " to " + path.string() + + ": " + ec.message()); + } +#endif + } + + static void sync_directory(const std::filesystem::path& path) + { +#ifndef _WIN32 + int flags = O_RDONLY; +# ifdef O_DIRECTORY + flags |= O_DIRECTORY; +# endif +# ifdef O_CLOEXEC + flags |= O_CLOEXEC; +# endif + int fd = ::open(path.c_str(), flags); + if (fd < 0) + { + throw std::runtime_error( + system_error_message("cannot open directory " + path.string())); + } + if (::fsync(fd) != 0) + { + const std::string message = + system_error_message("error syncing directory " + path.string()); + ::close(fd); + throw std::runtime_error(message); + } + if (::close(fd) != 0) + { + throw std::runtime_error( + system_error_message("error closing directory " + path.string())); + } +#else + (void)path; +#endif } }; @@ -568,7 +805,9 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) /// number of full tiles, since only full tiles are durable. TileHashSourceT(const Store& store, uint64_t available_size) : store(store), available_size((available_size / TILE_WIDTH) * TILE_WIDTH) - {} + { + tile_cache.reserve(TILE_CACHE_SIZE); + } bool subtree_root(uint8_t level, uint64_t index, Hash& out) const override { @@ -614,7 +853,7 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) const uint64_t span = (uint64_t)1 << level; const uint64_t start = index << level; const std::vector tile = - store.read_tile(TileRef{0, start / TILE_WIDTH}); + read_tile(TileRef{0, start / TILE_WIDTH}); out = roll_up(tile, start % TILE_WIDTH, span); return; } @@ -630,7 +869,7 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) if (n < full_tiles) { // One full level-L tile holds all 2**r entries of this subtree. - const std::vector tile = store.read_tile(TileRef{L, n}); + const std::vector tile = read_tile(TileRef{L, n}); out = roll_up(tile, first % TILE_WIDTH, (uint64_t)1 << r); return; } @@ -642,6 +881,37 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) resolve((uint8_t)(level - 1), index * 2 + 1, hi); HASH_FUNCTION(lo, hi, out); } + + struct TileCacheEntry + { + TileRef ref; + std::vector hashes; + }; + + static constexpr size_t TILE_CACHE_SIZE = 64; + mutable std::vector tile_cache; + + std::vector read_tile(const TileRef& ref) const + { + for (auto it = tile_cache.begin(); it != tile_cache.end(); it++) + { + if (it->ref.level == ref.level && it->ref.index == ref.index) + { + TileCacheEntry entry = std::move(*it); + tile_cache.erase(it); + std::vector hashes = entry.hashes; + tile_cache.push_back(std::move(entry)); + return hashes; + } + } + + if (tile_cache.size() >= TILE_CACHE_SIZE) + { + tile_cache.erase(tile_cache.begin()); + } + tile_cache.push_back(TileCacheEntry{ref, store.read_tile(ref)}); + return tile_cache.back().hashes; + } }; /// @brief Builds and verifies inclusion and consistency proofs. @@ -837,7 +1107,7 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) static uint64_t largest_pow2_lt(uint64_t n) { uint64_t k = 1; - while ((k << 1) < n) + while (k <= (n - 1) / 2) { k <<= 1; } @@ -1175,8 +1445,9 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) Tree tree; uint64_t tiles_size = 0; - /// @brief Builds a proof engine over the combined resident-tree (frontier) - /// and full-tile (flushed past) source, and invokes @p fn with it. + /// @brief Builds a proof engine over the combined resident-tree + /// (frontier) and full-tile (flushed past) source, and invokes @p fn with + /// it. /// @note The sources and engine are stack-local; @p fn must consume the /// engine before returning (proofs are returned by value, holding hash /// copies, so the result outlives the engine). diff --git a/test/compare_hash_functions.cpp b/test/compare_hash_functions.cpp index 55b9245..9c4b0c4 100644 --- a/test/compare_hash_functions.cpp +++ b/test/compare_hash_functions.cpp @@ -125,7 +125,6 @@ void compare_compression_hashes() #ifdef HAVE_EVERCRYPT compare_roots(mt, mte, "EverCrypt"); #endif - } std::cout << num_trees << " trees, " << total_inserts << " inserts, " @@ -182,7 +181,6 @@ void compare_full_hashes() # ifdef HAVE_EVERCRYPT compare_roots(mto, mte, "OpenSSL"); # endif - } std::cout << num_trees << " trees, " << total_inserts << " inserts, " @@ -210,7 +208,9 @@ void bench( mt.root(); auto stop = std::chrono::high_resolution_clock::now(); const double seconds = - static_cast(std::chrono::duration_cast(stop - start).count()) / + static_cast( + std::chrono::duration_cast(stop - start) + .count()) / 1e9; std::cout << std::left << std::setw(10) << name << ": " << mt.statistics.num_insert << " insertions, " @@ -238,7 +238,9 @@ void benchT( mt.root(); auto stop = std::chrono::high_resolution_clock::now(); const double seconds = - static_cast(std::chrono::duration_cast(stop - start).count()) / + static_cast( + std::chrono::duration_cast(stop - start) + .count()) / 1e9; std::cout << std::left << std::setw(10) << name << ": " << mt.statistics.num_insert << " insertions, " diff --git a/test/coverage.cpp b/test/coverage.cpp index a930d17..21967b5 100644 --- a/test/coverage.cpp +++ b/test/coverage.cpp @@ -121,7 +121,9 @@ namespace mixed_case_hex[0] = 'A'; mixed_case_hex[1] = 'f'; const merkle::Hash mixed_case_hash(mixed_case_hex); - require(mixed_case_hash.bytes[0] == 0xAF, "mixed-case hash string parsed incorrectly"); + require( + mixed_case_hash.bytes[0] == 0xAF, + "mixed-case hash string parsed incorrectly"); require_throws( [] { merkle::Hash(std::string(64, 'z')); }, @@ -267,9 +269,14 @@ namespace merkle::Tree copy_assigned(hash_with_byte(0xEE)); copy_assigned = source; - require(copy_assigned.root() == expected_root, "copy assignment root mismatch"); - require(copy_assigned.min_index() == expected_min, "copy assignment min mismatch"); - require(copy_assigned.max_index() == expected_max, "copy assignment max mismatch"); + require( + copy_assigned.root() == expected_root, "copy assignment root mismatch"); + require( + copy_assigned.min_index() == expected_min, + "copy assignment min mismatch"); + require( + copy_assigned.max_index() == expected_max, + "copy assignment max mismatch"); require( copy_assigned.leaf(expected_min) == source_hashes[expected_min], "copy-assigned leaf mismatch"); diff --git a/test/demo_tree.cpp b/test/demo_tree.cpp index 05d6f87..3ec9fcb 100644 --- a/test/demo_tree.cpp +++ b/test/demo_tree.cpp @@ -41,9 +41,9 @@ int main() std::cout << "P" << std::setw(2) << std::setfill('0') << i << ": " << path->to_string(PRINT_HASH_SIZE) << " " << '\n'; if (!path->verify(root)) - { - throw std::runtime_error("root hash mismatch"); - } + { + throw std::runtime_error("root hash mismatch"); + } const std::vector chk = *path; } @@ -52,9 +52,9 @@ int main() mt.serialise(buffer); merkle::Tree dmt(buffer); if (mt.root() != dmt.root()) - { - throw std::runtime_error("root hash mismatch"); - } + { + throw std::runtime_error("root hash mismatch"); + } std::cout << '\n'; } @@ -74,7 +74,6 @@ int main() /// SNIPPET_END: OpenSSL-SHA256 } #endif - } catch (std::exception& ex) { diff --git a/test/flush.cpp b/test/flush.cpp index f3e9d28..bd78746 100644 --- a/test/flush.cpp +++ b/test/flush.cpp @@ -36,8 +36,8 @@ int main() for (size_t k = 0; k < num_trees && !timed_out(timeout, test_start_time); k++) { - const auto num_leaves = - static_cast(1 + (std::rand() / (double)RAND_MAX) * max_num_leaves); + const auto num_leaves = static_cast( + 1 + (std::rand() / (double)RAND_MAX) * max_num_leaves); total_leaves += num_leaves; auto hashes = make_hashes(num_leaves); diff --git a/test/partial_serialisation.cpp b/test/partial_serialisation.cpp index 411a2ae..d75d575 100644 --- a/test/partial_serialisation.cpp +++ b/test/partial_serialisation.cpp @@ -1,16 +1,15 @@ // Copyright (c) Microsoft Corporation. // Licensed under the MIT License. +#include "util.h" + #include #include #include #include #include -#include - #include - -#include "util.h" +#include constexpr size_t PRINT_HASH_SIZE = 3; @@ -41,7 +40,8 @@ int main() for (size_t k = 0; k < num_trees && !timed_out(timeout, test_start_time); k++) { - const auto num_leaves = static_cast(1 + (std::rand() / (double)RAND_MAX) * max_num_leaves); + const auto num_leaves = static_cast( + 1 + (std::rand() / (double)RAND_MAX) * max_num_leaves); const auto num_subtrees = static_cast( 1 + (std::rand() / (double)RAND_MAX) * max_num_subtrees); total_leaves += num_leaves; diff --git a/test/past_paths.cpp b/test/past_paths.cpp index 6553b91..137510f 100644 --- a/test/past_paths.cpp +++ b/test/past_paths.cpp @@ -1,15 +1,14 @@ // Copyright (c) Microsoft Corporation. // Licensed under the MIT License. +#include "util.h" + #include #include #include #include - #include -#include "util.h" - constexpr size_t PSZ = 3; std::shared_ptr past_root_spec( @@ -58,8 +57,10 @@ int main() bool is_timed_out = false; for (size_t l = 0; l < num_trees && !is_timed_out; l++) { - const auto num_leaves = static_cast(1 + (std::rand() / (double)RAND_MAX) * max_num_leaves); - const auto num_paths = static_cast(1 + (std::rand() / (double)RAND_MAX) * max_num_paths); + const auto num_leaves = static_cast( + 1 + (std::rand() / (double)RAND_MAX) * max_num_leaves); + const auto num_paths = static_cast( + 1 + (std::rand() / (double)RAND_MAX) * max_num_paths); total_leaves += num_leaves; total_paths += num_paths; @@ -118,12 +119,9 @@ int main() auto pp_root = past_path->root(); #ifdef MERKLECPP_TRACE_ENABLED - MERKLECPP_TOUT << "Past root: " << past_root->to_string(PSZ) - << '\n'; - MERKLECPP_TOUT << "Past path: " << past_path->to_string(PSZ) - << '\n'; - MERKLECPP_TOUT << "Computed root: " << pp_root->to_string(PSZ) - << '\n'; + MERKLECPP_TOUT << "Past root: " << past_root->to_string(PSZ) << '\n'; + MERKLECPP_TOUT << "Past path: " << past_path->to_string(PSZ) << '\n'; + MERKLECPP_TOUT << "Computed root: " << pp_root->to_string(PSZ) << '\n'; #endif if (*past_path_spec != *past_path) diff --git a/test/past_root.cpp b/test/past_root.cpp index f03600d..848b5fc 100644 --- a/test/past_root.cpp +++ b/test/past_root.cpp @@ -1,16 +1,15 @@ // Copyright (c) Microsoft Corporation. // Licensed under the MIT License. +#include "util.h" + #include #include #include #include #include -#include - #include - -#include "util.h" +#include constexpr size_t PRINT_HASH_SIZE = 3; @@ -80,7 +79,8 @@ int main() k++) { std::map past_roots; - const auto num_leaves = static_cast(1 + (std::rand() / (double)RAND_MAX) * max_num_leaves); + const auto num_leaves = static_cast( + 1 + (std::rand() / (double)RAND_MAX) * max_num_leaves); total_leaves += num_leaves; auto hashes = make_hashes(num_leaves); @@ -112,8 +112,7 @@ int main() if (*pr != kv.second) { std::cout << pr->to_string(PRINT_HASH_SIZE) - << " != " << kv.second.to_string(PRINT_HASH_SIZE) - << '\n'; + << " != " << kv.second.to_string(PRINT_HASH_SIZE) << '\n'; throw std::runtime_error("past root hash mismatch"); } } diff --git a/test/paths.cpp b/test/paths.cpp index 77624fb..a6fb74f 100644 --- a/test/paths.cpp +++ b/test/paths.cpp @@ -1,15 +1,14 @@ // Copyright (c) Microsoft Corporation. // Licensed under the MIT License. +#include "util.h" + #include #include #include #include - #include -#include "util.h" - constexpr size_t PRINT_HASH_SIZE = 3; int main() @@ -37,8 +36,10 @@ int main() for (size_t l = 0; l < num_trees && !timed_out(timeout, test_start_time); l++) { - const auto num_leaves = static_cast(1 + (std::rand() / (double)RAND_MAX) * max_num_leaves); - const auto num_paths = static_cast(1 + (std::rand() / (double)RAND_MAX) * max_num_paths); + const auto num_leaves = static_cast( + 1 + (std::rand() / (double)RAND_MAX) * max_num_leaves); + const auto num_paths = static_cast( + 1 + (std::rand() / (double)RAND_MAX) * max_num_paths); total_leaves += num_leaves; total_paths += num_paths; @@ -55,7 +56,9 @@ int main() for (size_t p = 0; p < num_paths && !timed_out(timeout, test_start_time); p++) { - const auto i = static_cast((std::rand() / static_cast(RAND_MAX)) * static_cast(num_leaves - 1)); + const auto i = static_cast( + (std::rand() / static_cast(RAND_MAX)) * + static_cast(num_leaves - 1)); auto path = mt.path(i); if (!path->verify(root)) { @@ -65,7 +68,8 @@ int main() path->serialise(serialised_path); if (path->serialised_size() != serialised_path.size()) { - throw std::runtime_error("serialised_size() != serialised_path.size()"); + throw std::runtime_error( + "serialised_size() != serialised_path.size()"); } } } diff --git a/test/retract.cpp b/test/retract.cpp index 30c6973..52b49ce 100644 --- a/test/retract.cpp +++ b/test/retract.cpp @@ -1,15 +1,14 @@ // Copyright (c) Microsoft Corporation. // Licensed under the MIT License. +#include "util.h" + #include #include #include #include - #include -#include "util.h" - constexpr size_t PRINT_HASH_SIZE = 3; int main() @@ -37,7 +36,8 @@ int main() for (size_t k = 0; k < num_trees && !timed_out(timeout, test_start_time); k++) { - const auto num_leaves = static_cast(1 + (std::rand() / (double)RAND_MAX) * max_num_leaves); + const auto num_leaves = static_cast( + 1 + (std::rand() / (double)RAND_MAX) * max_num_leaves); total_leaves += num_leaves; auto hashes = make_hashes(num_leaves); @@ -55,7 +55,8 @@ int main() mt.retract_to(mt.max_index()); if (mt.max_index() != max_before) { - throw std::runtime_error("retract_to(max_index()) changed max_index"); + throw std::runtime_error( + "retract_to(max_index()) changed max_index"); } } diff --git a/test/serialisation.cpp b/test/serialisation.cpp index 8956ae6..1dd2abc 100644 --- a/test/serialisation.cpp +++ b/test/serialisation.cpp @@ -1,16 +1,15 @@ // Copyright (c) Microsoft Corporation. // Licensed under the MIT License. +#include "util.h" + #include #include #include #include #include -#include - #include - -#include "util.h" +#include constexpr size_t PRINT_HASH_SIZE = 3; @@ -38,7 +37,8 @@ int main() for (size_t k = 0; k < num_trees && !timed_out(timeout, test_start_time); k++) { - const auto num_leaves = static_cast(1 + (std::rand() / (double)RAND_MAX) * max_num_leaves); + const auto num_leaves = static_cast( + 1 + (std::rand() / (double)RAND_MAX) * max_num_leaves); total_leaves += num_leaves; auto hashes = make_hashes(num_leaves); @@ -80,10 +80,8 @@ int main() mt.serialised_size() != mt2.serialised_size() || mt.size() != mt2.size()) { - std::cout << "before:" << '\n' - << mt.to_string(PRINT_HASH_SIZE) << '\n'; - std::cout << "after:" << '\n' - << mt2.to_string(PRINT_HASH_SIZE) << '\n'; + std::cout << "before:" << '\n' << mt.to_string(PRINT_HASH_SIZE) << '\n'; + std::cout << "after:" << '\n' << mt2.to_string(PRINT_HASH_SIZE) << '\n'; throw std::runtime_error("tree properties mismatch"); } diff --git a/test/tiles_level2.cpp b/test/tiles_level2.cpp index f0e38ab..e1ea541 100644 --- a/test/tiles_level2.cpp +++ b/test/tiles_level2.cpp @@ -21,11 +21,11 @@ namespace fs = std::filesystem; using merkle::Hash; +using merkle::tiles::TILE_WIDTH; using merkle::tiles::TileHashSource; using merkle::tiles::TileRef; using merkle::tiles::TileStore; using merkle::tiles::TileWriter; -using merkle::tiles::TILE_WIDTH; static void expect(bool cond, const std::string& what) { @@ -72,8 +72,7 @@ int main() // 65536 full L0 tiles + 256 full L1 tiles + 1 full L2 tile. expect( - stats.full_written == - (uint64_t)TILE_WIDTH * TILE_WIDTH + TILE_WIDTH + 1, + stats.full_written == (uint64_t)TILE_WIDTH * TILE_WIDTH + TILE_WIDTH + 1, "level2: tile counts"); expect(store.has_full_tile(2, 0), "level2: L2 tile present"); expect(!store.has_full_tile(2, 1), "level2: no second L2 tile"); @@ -84,9 +83,10 @@ int main() const TileHashSource src(store, n); - // Each level-2 entry j is the root of level-1 tile j, which rolls up level-0 - // tiles, which are the leaves verbatim. Cross-check the writer's roll-up, - // resolve's level-2 read, and the leaf chain on a sample of indices. + // Each level-2 entry j is the root of level-1 tile j, which rolls up + // level-0 tiles, which are the leaves verbatim. Cross-check the writer's + // roll-up, resolve's level-2 read, and the leaf chain on a sample of + // indices. for (const uint64_t j : {(uint64_t)0, (uint64_t)1, (uint64_t)200, (uint64_t)255}) { diff --git a/test/tiles_store.cpp b/test/tiles_store.cpp index a23a3a7..04848f1 100644 --- a/test/tiles_store.cpp +++ b/test/tiles_store.cpp @@ -3,6 +3,8 @@ #include "util.h" +#include +#include #include #include #include @@ -13,6 +15,7 @@ #include #include #include +#include #include namespace fs = std::filesystem; @@ -42,13 +45,26 @@ static std::string rel(const merkle::tiles::TileStore& store, const fs::path& p) return p.lexically_relative(store.root()).generic_string(); } +static bool any_tmp_files(const fs::path& dir) +{ + if (!fs::exists(dir)) + { + return false; + } + return std::any_of( + fs::recursive_directory_iterator(dir), + fs::recursive_directory_iterator(), + [](const auto& entry) { + return entry.path().filename().string().find(".tmp") != std::string::npos; + }); +} + // Overwrites a file with exactly the given bytes (to simulate corruption). static void overwrite_file(const fs::path& p, const std::vector& bytes) { std::ofstream f(p, std::ios::binary | std::ios::trunc); f.write( - reinterpret_cast(bytes.data()), - (std::streamsize)bytes.size()); + reinterpret_cast(bytes.data()), (std::streamsize)bytes.size()); } int main() @@ -140,13 +156,18 @@ int main() // Truncated tile: fewer bytes than a full tile. overwrite_file(store.tile_path(full_ref), std::vector(hsz, 0)); + expect(!store.has_full_tile(0, 0), "truncated tile is not durable"); expect_throws( [&] { (void)store.read_tile(full_ref); }, "truncated tile rejected"); + store.write_tile(full_ref, full); + expect(store.has_full_tile(0, 0), "truncated tile rewritten"); + expect(store.read_tile(full_ref) == full, "rewritten tile round-trip"); // Oversized tile: more bytes than a full tile. overwrite_file( store.tile_path(full_ref), std::vector((merkle::tiles::TILE_WIDTH + 1) * hsz, 0)); + expect(!store.has_full_tile(0, 0), "oversized tile is not durable"); expect_throws( [&] { (void)store.read_tile(full_ref); }, "oversized tile rejected"); @@ -174,6 +195,34 @@ int main() "decode_entries oversized length rejected"); } + // 3d. Concurrent same-tile writes use unique temp files and leave no + // temporary files behind after success. + { + const TileRef concurrent_ref{0, 42}; + std::atomic ok{true}; + std::vector threads; + for (size_t i = 0; i < 8; i++) + { + threads.emplace_back([&] { + try + { + store.write_tile(concurrent_ref, full); + } + catch (...) + { + ok = false; + } + }); + } + for (auto& thread : threads) + { + thread.join(); + } + expect(ok, "concurrent writes succeeded"); + expect(store.read_tile(concurrent_ref) == full, "concurrent tile valid"); + expect(!any_tmp_files(dir), "no temp files left after writes"); + } + std::cout << "tiles_store: OK" << '\n'; std::error_code ec; diff --git a/test/tiles_tree.cpp b/test/tiles_tree.cpp index 7706e5a..f009d90 100644 --- a/test/tiles_tree.cpp +++ b/test/tiles_tree.cpp @@ -8,6 +8,7 @@ #include #include #include +#include #include #include #include @@ -20,6 +21,12 @@ using merkle::tiles::MemoryHashSource; using merkle::tiles::ProofEngine; using merkle::tiles::TiledTree; +class ProofEngineProbe : public ProofEngine +{ +public: + using ProofEngine::largest_pow2_lt; +}; + static void expect(bool cond, const std::string& what) { if (!cond) @@ -159,6 +166,34 @@ int main() } std::cout << "memory source: OK" << '\n'; + // ---- Part 1b: hostile arithmetic inputs are rejected without UB or + // overflow loops. + { + merkle::Tree tree; + tree.insert(hashes[0]); + Hash out; + expect(!tree.subtree_root(64, 0, out), "subtree_root rejects level 64"); + expect(!tree.subtree_root(100, 0, out), "subtree_root rejects level 100"); + expect( + !tree.subtree_root(1, std::numeric_limits::max(), out), + "subtree_root rejects overflowing index"); + + expect(ProofEngineProbe::largest_pow2_lt(2) == 1, "pow2_lt 2"); + expect( + ProofEngineProbe::largest_pow2_lt((uint64_t)1 << 63) == + ((uint64_t)1 << 62), + "pow2_lt 2^63"); + expect( + ProofEngineProbe::largest_pow2_lt(((uint64_t)1 << 63) + 1) == + ((uint64_t)1 << 63), + "pow2_lt 2^63+1"); + expect( + ProofEngineProbe::largest_pow2_lt( + std::numeric_limits::max()) == ((uint64_t)1 << 63), + "pow2_lt uint64 max"); + std::cout << "hostile arithmetic inputs: OK" << '\n'; + } + // ---- Part 2: TiledTree flush, proofs over tiles + memory. const uint64_t n1 = 1000; // first flush size const uint64_t N = 1500; // final size @@ -333,7 +368,8 @@ int main() { rtt.append(hashes[i]); } - rtt.flush(); // covered = 768; target = floor((768 - 300) / 256) * 256 = 256 + rtt.flush(); // covered = 768; target = floor((768 - 300) / 256) * 256 = + // 256 expect(rtt.flushed_size() == 768, "margin: flushed size 768"); expect( rtt.tree_ref().min_index() == 256, diff --git a/test/tiles_writer.cpp b/test/tiles_writer.cpp index f7f9e5f..d100361 100644 --- a/test/tiles_writer.cpp +++ b/test/tiles_writer.cpp @@ -82,8 +82,7 @@ int main() expect(store.has_full_tile(0, 0), "A L0 full tile"); expect( - fs::file_size(store.tile_path(TileRef{0, 0})) == - 256U * Hash().size(), + fs::file_size(store.tile_path(TileRef{0, 0})) == 256U * Hash().size(), "A L0 full tile size"); expect(!store.has_full_tile(1, 0), "A no L1 full tile"); expect(!any_partial_dirs(store), "A no partial tiles"); @@ -133,8 +132,7 @@ int main() const auto l1 = store.read_tile(TileRef{1, 0}); expect(l1.size() == 256, "B L1 full width"); expect(l1[0] == rollup(store.read_tile(TileRef{0, 0})), "B L1[0]"); - expect( - l1[255] == rollup(store.read_tile(TileRef{0, 255})), "B L1[255]"); + expect(l1[255] == rollup(store.read_tile(TileRef{0, 255})), "B L1[255]"); std::cout << "B (size 70000): OK" << '\n'; } diff --git a/test/time_large_trees.cpp b/test/time_large_trees.cpp index f5ebfa1..3ab8b3f 100644 --- a/test/time_large_trees.cpp +++ b/test/time_large_trees.cpp @@ -42,8 +42,9 @@ int main() mt.root(); auto stop = std::chrono::high_resolution_clock::now(); const double seconds = - static_cast(std::chrono::duration_cast(stop - start) - .count()) / + static_cast( + std::chrono::duration_cast(stop - start) + .count()) / 1e9; std::cout << "NEW: " << mt.statistics.to_string() << " in " << seconds << " sec" << '\n'; @@ -66,8 +67,10 @@ int main() mt384.root(); auto stop384 = std::chrono::high_resolution_clock::now(); const double seconds384 = - static_cast(std::chrono::duration_cast(stop384 - start384) - .count()) / + static_cast( + std::chrono::duration_cast( + stop384 - start384) + .count()) / 1e9; std::cout << "SHA384: " << mt384.statistics.to_string() << " in " << seconds384 << " sec" << '\n'; @@ -90,8 +93,10 @@ int main() mt512.root(); auto stop512 = std::chrono::high_resolution_clock::now(); const double seconds512 = - static_cast(std::chrono::duration_cast(stop512 - start512) - .count()) / + static_cast( + std::chrono::duration_cast( + stop512 - start512) + .count()) / 1e9; std::cout << "SHA512: " << mt512.statistics.to_string() << " in " << seconds512 << " sec" << '\n'; @@ -121,8 +126,9 @@ int main() } mt_get_root(ec_mt, ec_root); stop = std::chrono::high_resolution_clock::now(); - const double ec_seconds = std::chrono::duration_cast(stop - start) - .count() / + const double ec_seconds = + std::chrono::duration_cast(stop - start) + .count() / 1e9; std::cout << "EC :" << " num_insert=" << ec_hashes.size() diff --git a/test/unit_tests.cpp b/test/unit_tests.cpp index e6f6967..da0dc46 100644 --- a/test/unit_tests.cpp +++ b/test/unit_tests.cpp @@ -149,7 +149,7 @@ TEST_CASE("PathT equality") const auto path0a = tree.path(0); const auto path0b = tree.path(0); // same path extracted twice - const auto path1 = tree.path(1); // path to a different leaf + const auto path1 = tree.path(1); // path to a different leaf // Two paths to the same leaf should be equal REQUIRE(*path0a == *path0b); @@ -166,7 +166,7 @@ TEST_CASE("PathT equality") h3.bytes[0] = 3; tree_diff.insert(h3); - const auto path_orig = tree.path(0); // h0 leaf, element has h1 + const auto path_orig = tree.path(0); // h0 leaf, element has h1 const auto path_diff = tree_diff.path(0); // h0 leaf, element has h3 REQUIRE_FALSE(*path_orig == *path_diff); REQUIRE(*path_orig != *path_diff); diff --git a/test/util.h b/test/util.h index 2abfc77..196f8b9 100644 --- a/test/util.h +++ b/test/util.h @@ -38,9 +38,9 @@ inline std::vector> make_hashesT( inline size_t random_index(merkle::Tree& mt) { - return (size_t)( - mt.min_index() + - (std::rand() / (double)RAND_MAX) * (mt.max_index() - mt.min_index())); + return (size_t)(mt.min_index() + + (std::rand() / (double)RAND_MAX) * + (mt.max_index() - mt.min_index())); } inline double get_timeout() From aae40956fc03682cd797e13761983a1ec2cc7382 Mon Sep 17 00:00:00 2001 From: achamayou Date: Thu, 2 Jul 2026 13:19:05 +0000 Subject: [PATCH 18/30] Fix CI audit findings --- CMakeLists.txt | 4 +- doc/conf.py | 29 +++++---- merklecpp.h | 149 ++++++++++++++++++++++++-------------------- merklecpp_tiles.h | 15 ++--- test/CMakeLists.txt | 12 ++-- test/coverage.cpp | 13 ++++ test/unit_tests.cpp | 6 +- 7 files changed, 126 insertions(+), 102 deletions(-) diff --git a/CMakeLists.txt b/CMakeLists.txt index 3e35781..e59b7b2 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -1,9 +1,9 @@ # Copyright (c) Microsoft Corporation. # Licensed under the MIT License. -project(merklecpp LANGUAGES CXX C ASM) - cmake_minimum_required(VERSION 3.11) + +project(merklecpp LANGUAGES CXX C ASM) set(CMAKE_CXX_STANDARD 17) set(MERKLECPP_DIR ${CMAKE_CURRENT_SOURCE_DIR}) diff --git a/doc/conf.py b/doc/conf.py index 5a7ef0e..6d84c31 100644 --- a/doc/conf.py +++ b/doc/conf.py @@ -12,17 +12,18 @@ # import os import sys -sys.path.insert(0, os.path.abspath('.')) + +sys.path.insert(0, os.path.abspath(".")) # -- Project information ----------------------------------------------------- -project = 'merklecpp' -copyright = '2021, Microsoft' -author = 'Christoph M. Wintersteiger' +project = "merklecpp" +copyright = "2021, Microsoft" +author = "Christoph M. Wintersteiger" # The full version, including alpha/beta/rc tags -release = '1.0.0' +release = "1.0.0" # -- General configuration --------------------------------------------------- @@ -31,12 +32,12 @@ # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom # ones. extensions = [ - "breathe", - "sphinx.ext.githubpages", + "breathe", + "sphinx.ext.githubpages", ] # Add any paths that contain templates here, relative to this directory. -templates_path = ['_templates'] +templates_path = ["_templates"] # List of patterns, relative to source directory, that match files and # directories to ignore when looking for source files. @@ -49,17 +50,15 @@ # The theme to use for HTML and HTML Help pages. See the documentation for # a list of builtin themes. # -html_theme = 'pydata_sphinx_theme' +html_theme = "pydata_sphinx_theme" -# Add any paths that contain custom static files (such as style sheets) here, -# relative to this directory. They are copied after the builtin static files, -# so a file named "default.css" will overwrite the builtin "default.css". -html_static_path = ['_static'] +# Add custom static paths here when the documentation needs local assets. +html_static_path = [] # Setup the breathe extension breathe_projects = {"merklecpp": "xml"} breathe_default_project = "merklecpp" -breathe_domain_by_extension = {"h" : "cpp"} +breathe_domain_by_extension = {"h": "cpp"} def setup(self): @@ -70,4 +69,4 @@ def setup(self): breathe_projects["merklecpp"] = str(srcdir / breathe_projects["merklecpp"]) if not os.environ.get("SKIP_DOXYGEN"): - subprocess.run(["doxygen"], cwd=srcdir / "..", check=True) \ No newline at end of file + subprocess.run(["doxygen"], cwd=srcdir / "..", check=True) diff --git a/merklecpp.h b/merklecpp.h index 274d92a..c4114ab 100644 --- a/merklecpp.h +++ b/merklecpp.h @@ -296,10 +296,14 @@ namespace merkle { public: /// @brief Path direction - using Direction = enum { PATH_LEFT, PATH_RIGHT }; + enum Direction + { + PATH_LEFT, + PATH_RIGHT + }; /// @brief Path element - using Element = struct + struct Element { /// @brief The hash of the path element HashT hash; @@ -799,9 +803,9 @@ namespace merkle /// @param hash Hash to insert void insert(const Hash& hash) { - MERKLECPP_TRACE(MERKLECPP_TOUT << "> insert " - << hash.to_string(TRACE_HASH_SIZE) - << std::endl;); + MERKLECPP_TRACE( + MERKLECPP_TOUT << "> insert " << hash.to_string(TRACE_HASH_SIZE) + << std::endl;); uninserted_leaf_nodes.push_back(Node::make(hash)); statistics.num_insert++; } @@ -843,10 +847,10 @@ namespace merkle walk_to(index, false, [this](Node*& n, bool go_right) { if (go_right && n->left) { - MERKLECPP_TRACE(MERKLECPP_TOUT - << " - conflate " - << n->left->hash.to_string(TRACE_HASH_SIZE) - << std::endl;); + MERKLECPP_TRACE( + MERKLECPP_TOUT << " - conflate " + << n->left->hash.to_string(TRACE_HASH_SIZE) + << std::endl;); if (n->left && n->left->dirty) { hash(n->left); @@ -901,10 +905,10 @@ namespace merkle n->dirty = true; if (go_left && n->right) { - MERKLECPP_TRACE(MERKLECPP_TOUT - << " - eliminate " - << n->right->hash.to_string(TRACE_HASH_SIZE) - << std::endl;); + MERKLECPP_TRACE( + MERKLECPP_TOUT << " - eliminate " + << n->right->hash.to_string(TRACE_HASH_SIZE) + << std::endl;); bool is_root = n == _root; Node* old_left = n->left; @@ -924,18 +928,18 @@ namespace merkle if (is_root) { - MERKLECPP_TRACE(MERKLECPP_TOUT - << " - new root: " - << n->hash.to_string(TRACE_HASH_SIZE) - << std::endl;); + MERKLECPP_TRACE( + MERKLECPP_TOUT + << " - new root: " << n->hash.to_string(TRACE_HASH_SIZE) + << std::endl;); assert(_root == n); } assert(n->invariant()); - MERKLECPP_TRACE(MERKLECPP_TOUT - << " - after elimination: " << std::endl - << to_string(TRACE_HASH_SIZE) << std::endl;); + MERKLECPP_TRACE( + MERKLECPP_TOUT << " - after elimination: " << std::endl + << to_string(TRACE_HASH_SIZE) << std::endl;); return false; } return true; @@ -1008,9 +1012,9 @@ namespace merkle statistics.num_root++; compute_root(); assert(_root && !_root->dirty); - MERKLECPP_TRACE(MERKLECPP_TOUT - << " - root: " << _root->hash.to_string(TRACE_HASH_SIZE) - << std::endl;); + MERKLECPP_TRACE( + MERKLECPP_TOUT << " - root: " << _root->hash.to_string(TRACE_HASH_SIZE) + << std::endl;); return _root->hash; } @@ -1080,12 +1084,12 @@ namespace merkle { walk_stack.push_back(cur); } - MERKLECPP_TRACE(MERKLECPP_TOUT - << " - at " << cur->hash.to_string(TRACE_HASH_SIZE) - << " (" << cur->size << "/" << (unsigned)cur->height - << ")" - << " (" << (go_right ? "R" : "L") << ")" - << std::endl;); + MERKLECPP_TRACE( + MERKLECPP_TOUT << " - at " << cur->hash.to_string(TRACE_HASH_SIZE) + << " (" << cur->size << "/" << (unsigned)cur->height + << ")" + << " (" << (go_right ? "R" : "L") << ")" + << std::endl;); if (cur->height == height) { if (!f(cur, go_right)) @@ -1140,8 +1144,9 @@ namespace merkle /// tree to @p as_of and then extracting the path of @p index. std::shared_ptr past_path(size_t index, size_t as_of) { - MERKLECPP_TRACE(MERKLECPP_TOUT << "> past_path from " << index - << " as of " << as_of << std::endl;); + MERKLECPP_TRACE( + MERKLECPP_TOUT << "> past_path from " << index << " as of " << as_of + << std::endl;); statistics.num_past_paths++; if ( @@ -1180,24 +1185,24 @@ namespace merkle bool const go_right_i = ((it_i >> (8 * sizeof(it_i) - 1)) & 0x01) != 0U; bool const go_right_a = ((it_a >> (8 * sizeof(it_a) - 1)) & 0x01) != 0U; - MERKLECPP_TRACE(MERKLECPP_TOUT - << " - at " << (unsigned)height << ": " - << cur_i->hash.to_string(TRACE_HASH_SIZE) << " (" - << cur_i->size << "/" << (unsigned)cur_i->height - << "/" << (go_right_i ? "R" : "L") << ")" - << " / " << cur_a->hash.to_string(TRACE_HASH_SIZE) - << " (" << cur_a->size << "/" - << (unsigned)cur_a->height << "/" - << (go_right_a ? "R" : "L") << ")" << std::endl;); + MERKLECPP_TRACE( + MERKLECPP_TOUT << " - at " << (unsigned)height << ": " + << cur_i->hash.to_string(TRACE_HASH_SIZE) << " (" + << cur_i->size << "/" << (unsigned)cur_i->height << "/" + << (go_right_i ? "R" : "L") << ")" + << " / " << cur_a->hash.to_string(TRACE_HASH_SIZE) + << " (" << cur_a->size << "/" + << (unsigned)cur_a->height << "/" + << (go_right_a ? "R" : "L") << ")" << std::endl;); if (!fork_node && go_right_i != go_right_a) { assert(cur_i == cur_a); assert(!go_right_i && go_right_a); - MERKLECPP_TRACE(MERKLECPP_TOUT - << " - split at " - << cur_i->hash.to_string(TRACE_HASH_SIZE) - << std::endl;); + MERKLECPP_TRACE( + MERKLECPP_TOUT << " - split at " + << cur_i->hash.to_string(TRACE_HASH_SIZE) + << std::endl;); fork_node = cur_i; } @@ -1417,8 +1422,8 @@ namespace merkle compute_root(); - MERKLECPP_TRACE(MERKLECPP_TOUT << to_string(TRACE_HASH_SIZE) - << std::endl;); + MERKLECPP_TRACE( + MERKLECPP_TOUT << to_string(TRACE_HASH_SIZE) << std::endl;); std::vector extras; walk_to(min_index(), false, [&extras](Node*& n, bool go_right) { @@ -1442,8 +1447,9 @@ namespace merkle /// @param bytes The vector of bytes to serialise to void serialise(size_t from, size_t to, std::vector& bytes) { - MERKLECPP_TRACE(MERKLECPP_TOUT << "> serialise from " << from << " to " - << to << std::endl;); + MERKLECPP_TRACE( + MERKLECPP_TOUT << "> serialise from " << from << " to " << to + << std::endl;); validate_partial_range(from, to); @@ -1460,8 +1466,8 @@ namespace merkle compute_root(); - MERKLECPP_TRACE(MERKLECPP_TOUT << to_string(TRACE_HASH_SIZE) - << std::endl;); + MERKLECPP_TRACE( + MERKLECPP_TOUT << to_string(TRACE_HASH_SIZE) << std::endl;); std::vector extras; walk_to(from, false, [&extras](Node*& n, bool go_right) { @@ -1516,19 +1522,24 @@ namespace merkle // Restore extra hashes on the left edge of the tree if ((it & 0x01) != 0U) { + const uint8_t restored_height = level_no + 1; + if (restored_height >= std::numeric_limits::digits) + { + throw std::runtime_error("invalid serialised tree height"); + } Hash h(bytes, position); MERKLECPP_TRACE(MERKLECPP_TOUT << "+";); auto n = Node::make(h); - n->height = level_no + 1; - n->size = (1 << n->height) - 1; + n->height = restored_height; + n->size = (static_cast(1) << n->height) - 1; assert(n->invariant()); level.insert(level.begin(), n); } - MERKLECPP_TRACE(for (auto& n - : level) MERKLECPP_TOUT - << " " << n->hash.to_string(TRACE_HASH_SIZE); - MERKLECPP_TOUT << std::endl;); + MERKLECPP_TRACE( + for (auto& n : level) MERKLECPP_TOUT + << " " << n->hash.to_string(TRACE_HASH_SIZE); + MERKLECPP_TOUT << std::endl;); // Rebuild the level for (size_t i = 0; i < level.size(); i += 2) @@ -1953,9 +1964,9 @@ namespace merkle { while (true) { - MERKLECPP_TRACE(MERKLECPP_TOUT << " @ " - << n->hash.to_string(TRACE_HASH_SIZE) - << std::endl;); + MERKLECPP_TRACE( + MERKLECPP_TOUT << " @ " << n->hash.to_string(TRACE_HASH_SIZE) + << std::endl;); assert(n->invariant()); if (n->is_full()) @@ -2026,10 +2037,10 @@ namespace merkle if (!complete && !result->is_full()) { - MERKLECPP_TRACE(MERKLECPP_TOUT - << " X save " - << result->hash.to_string(TRACE_HASH_SIZE) - << std::endl;); + MERKLECPP_TRACE( + MERKLECPP_TOUT << " X save " + << result->hash.to_string(TRACE_HASH_SIZE) + << std::endl;); return result; } } @@ -2044,9 +2055,9 @@ namespace merkle /// @param n New leaf node to insert void insert_leaf(Node*& root, Node* n) { - MERKLECPP_TRACE(MERKLECPP_TOUT << " - insert_leaf " - << n->hash.to_string(TRACE_HASH_SIZE) - << std::endl;); + MERKLECPP_TRACE( + MERKLECPP_TOUT << " - insert_leaf " + << n->hash.to_string(TRACE_HASH_SIZE) << std::endl;); leaf_nodes.push_back(n); if (insertion_stack.empty() && !root) { @@ -2066,9 +2077,9 @@ namespace merkle { if (!uninserted_leaf_nodes.empty()) { - MERKLECPP_TRACE(MERKLECPP_TOUT - << "* insert_leaves " << leaf_nodes.size() << " +" - << uninserted_leaf_nodes.size() << std::endl;); + MERKLECPP_TRACE( + MERKLECPP_TOUT << "* insert_leaves " << leaf_nodes.size() << " +" + << uninserted_leaf_nodes.size() << std::endl;); // Potential future improvement: make this go fast when there are many // leaves to insert. for (auto& n : uninserted_leaf_nodes) diff --git a/merklecpp_tiles.h b/merklecpp_tiles.h index cd7c505..4d0ab4f 100644 --- a/merklecpp_tiles.h +++ b/merklecpp_tiles.h @@ -515,7 +515,7 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) # ifdef O_CLOEXEC flags |= O_CLOEXEC; # endif - int fd = ::open(path.c_str(), flags); + const int fd = ::open(path.c_str(), flags); if (fd < 0) { throw std::runtime_error( @@ -642,10 +642,10 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) const uint64_t full_tiles = entries / TILE_WIDTH; - if (!cursor_inited[level]) + if (cursor_inited[level] == 0) { next_full[level] = full_prefix_length(level); - cursor_inited[level] = true; + cursor_inited[level] = 1; } for (uint64_t n = next_full[level]; n < full_tiles; n++) @@ -677,7 +677,7 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) std::vector next_full; /// @brief Per-level flag indicating next_full has been initialised. - std::vector cursor_inited; + std::vector cursor_inited; /// @brief Number of complete level-@p level entries for a tree of @p /// size. @@ -694,7 +694,7 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) if (next_full.size() < needed) { next_full.resize(needed, 0); - cursor_inited.resize(needed, false); + cursor_inited.resize(needed, 0); } } @@ -747,8 +747,9 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) else { // Roll up the complete child full tile (256 complete entries). - out.push_back(perfect_root( - store.read_tile(TileRef{(uint8_t)(level - 1), g}))); + out.push_back( + perfect_root( + store.read_tile(TileRef{(uint8_t)(level - 1), g}))); } } return out; diff --git a/test/CMakeLists.txt b/test/CMakeLists.txt index edbadf4..80c34e1 100644 --- a/test/CMakeLists.txt +++ b/test/CMakeLists.txt @@ -37,6 +37,12 @@ add_merklecpp_test(tiles_entries tiles_entries.cpp) add_merklecpp_test(tiles_level2 tiles_level2.cpp) add_merklecpp_test(time_tiles time_tiles.cpp) +find_path(DOCTEST_DIR doctest.h) +if(NOT DOCTEST_DIR) + message(FATAL_ERROR "doctest.h not found; install doctest or set DOCTEST_DIR") +endif() +add_merklecpp_test(unit_tests unit_tests.cpp) + if(TARGET evercrypt.host) add_merklecpp_test(compare_evercrypt compare_evercrypt.cpp) endif() @@ -46,9 +52,3 @@ if(OPENSSL ) add_merklecpp_test(compare_hash_functions compare_hash_functions.cpp) endif() - -find_path(DOCTEST_DIR doctest.h) - -if(DOCTEST_DIR) - add_merklecpp_test(unit_tests unit_tests.cpp) -endif() diff --git a/test/coverage.cpp b/test/coverage.cpp index 21967b5..b5cf21b 100644 --- a/test/coverage.cpp +++ b/test/coverage.cpp @@ -257,6 +257,18 @@ namespace "partial serialised_size should reject retracted to index"); } + void test_deserialise_rejects_invalid_flushed_height() + { + std::vector buffer; + merkle::serialise_uint64_t(0, buffer); + merkle::serialise_uint64_t(uint64_t{1} << 63, buffer); + + merkle::Tree tree; + require_throws( + [&] { tree.deserialise(buffer); }, + "deserialise should reject impossible flushed subtree heights"); + } + void test_tree_assignment_and_moves() { const auto source_hashes = make_hashes(5); @@ -319,6 +331,7 @@ int main() test_hash_string_parsing(); test_path_metadata_and_equality(); test_tree_partial_serialisation_bounds(); + test_deserialise_rejects_invalid_flushed_height(); test_tree_assignment_and_moves(); } catch (const std::exception& ex) diff --git a/test/unit_tests.cpp b/test/unit_tests.cpp index da0dc46..c72cac1 100644 --- a/test/unit_tests.cpp +++ b/test/unit_tests.cpp @@ -175,7 +175,7 @@ TEST_CASE("PathT equality") TEST_CASE("TreeT to_string") { // Empty tree produces "" - merkle::Tree empty_tree; + const merkle::Tree empty_tree; const std::string empty_str = empty_tree.to_string(); REQUIRE(empty_str.find("") != std::string::npos); @@ -192,7 +192,7 @@ TEST_CASE("TreeT to_string") TEST_CASE("TreeT leaf bounds and uninserted leaves") { // leaf() out of bounds on empty tree - merkle::Tree empty_tree; + const merkle::Tree empty_tree; REQUIRE_THROWS(empty_tree.leaf(0)); // Access leaf before insertion is flushed (uninserted_leaf_nodes path) @@ -216,7 +216,7 @@ TEST_CASE("TreeT size with uninserted leaves") { merkle::Tree tree; // size() when tree has uninserted leaves triggers insert_leaves() - merkle::Tree::Hash h; + const merkle::Tree::Hash h; tree.insert(h); // size() forces lazy insertion const size_t sz = tree.size(); From 77ed58813b3e4f5bb96936ab4a750a6aea66c36c Mon Sep 17 00:00:00 2001 From: achamayou Date: Fri, 3 Jul 2026 10:48:56 +0000 Subject: [PATCH 19/30] Fetch doctest in CMake when missing --- test/CMakeLists.txt | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-) diff --git a/test/CMakeLists.txt b/test/CMakeLists.txt index 80c34e1..60ab6c5 100644 --- a/test/CMakeLists.txt +++ b/test/CMakeLists.txt @@ -39,7 +39,26 @@ add_merklecpp_test(time_tiles time_tiles.cpp) find_path(DOCTEST_DIR doctest.h) if(NOT DOCTEST_DIR) - message(FATAL_ERROR "doctest.h not found; install doctest or set DOCTEST_DIR") + set(DOCTEST_VERSION 2.4.11) + set(DOCTEST_DIR "${CMAKE_BINARY_DIR}/_deps/doctest-${DOCTEST_VERSION}") + set(DOCTEST_HEADER "${DOCTEST_DIR}/doctest.h") + + message(STATUS "doctest.h not found; downloading doctest v${DOCTEST_VERSION}") + file(MAKE_DIRECTORY "${DOCTEST_DIR}") + file( + DOWNLOAD + "https://raw.githubusercontent.com/doctest/doctest/v${DOCTEST_VERSION}/doctest/doctest.h" + "${DOCTEST_HEADER}" + EXPECTED_HASH SHA256=44faa038e9c3f9728efbda143748d01124ea0a27f4bf78f35a15d8fab2e039fb + STATUS DOCTEST_DOWNLOAD_STATUS + TLS_VERIFY ON + ) + + list(GET DOCTEST_DOWNLOAD_STATUS 0 DOCTEST_DOWNLOAD_CODE) + list(GET DOCTEST_DOWNLOAD_STATUS 1 DOCTEST_DOWNLOAD_MESSAGE) + if(NOT DOCTEST_DOWNLOAD_CODE EQUAL 0) + message(FATAL_ERROR "failed to download doctest.h: ${DOCTEST_DOWNLOAD_MESSAGE}") + endif() endif() add_merklecpp_test(unit_tests unit_tests.cpp) From b6886f8857961b89d2447e2cf79c9751cd90e924 Mon Sep 17 00:00:00 2001 From: achamayou Date: Fri, 3 Jul 2026 12:48:30 +0000 Subject: [PATCH 20/30] Skip concurrent tile write stress on Windows --- test/tiles_store.cpp | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/test/tiles_store.cpp b/test/tiles_store.cpp index 04848f1..f669a0d 100644 --- a/test/tiles_store.cpp +++ b/test/tiles_store.cpp @@ -195,6 +195,9 @@ int main() "decode_entries oversized length rejected"); } +// Windows file replacement semantics can reject racing same-path replaces even +// when the final tile remains valid, so keep this stress check POSIX-only. +#ifndef _WIN32 // 3d. Concurrent same-tile writes use unique temp files and leave no // temporary files behind after success. { @@ -222,6 +225,7 @@ int main() expect(store.read_tile(concurrent_ref) == full, "concurrent tile valid"); expect(!any_tmp_files(dir), "no temp files left after writes"); } +#endif std::cout << "tiles_store: OK" << '\n'; From 202255a48f268b074a4df7834dfa5bf24dc640e5 Mon Sep 17 00:00:00 2001 From: achamayou Date: Fri, 3 Jul 2026 14:25:29 +0000 Subject: [PATCH 21/30] Address tiled storage review feedback --- .github/workflows/ci.yml | 4 ++-- .github/workflows/codeql-analysis.yml | 2 +- CMakeLists.txt | 1 + README.md | 18 +++++++++++++++++ merklecpp_tiles.h | 24 ++++++++++++++++++++--- test/CMakeLists.txt | 7 +++++-- test/tiles_entries.cpp | 28 +++++++++++++++++++++++++++ test/tiles_store.cpp | 16 +++++++++++++++ 8 files changed, 92 insertions(+), 8 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index aa69062..3f3a2e6 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -51,9 +51,9 @@ jobs: working-directory: ${{github.workspace}}/build/${{ matrix.build_type }} run: | if [ "$RUNNER_OS" == "Linux" ]; then - cmake $GITHUB_WORKSPACE -DCMAKE_CXX_COMPILER=${{ matrix.compiler }} -DCMAKE_BUILD_TYPE=${{ matrix.build_type }} -DTESTS=ON -DOPENSSL=ON -DCLANG_TIDY=ON -DCMAKE_EXPORT_COMPILE_COMMANDS=ON + cmake $GITHUB_WORKSPACE -DCMAKE_CXX_COMPILER=${{ matrix.compiler }} -DCMAKE_BUILD_TYPE=${{ matrix.build_type }} -DTESTS=ON -DLONG_TESTS=ON -DOPENSSL=ON -DCLANG_TIDY=ON -DCMAKE_EXPORT_COMPILE_COMMANDS=ON else - cmake $GITHUB_WORKSPACE -DCMAKE_BUILD_TYPE=${{ matrix.build_type }} -DTESTS=ON + cmake $GITHUB_WORKSPACE -DCMAKE_BUILD_TYPE=${{ matrix.build_type }} -DTESTS=ON -DLONG_TESTS=ON fi - name: Build diff --git a/.github/workflows/codeql-analysis.yml b/.github/workflows/codeql-analysis.yml index e3ecf9f..0bf9149 100644 --- a/.github/workflows/codeql-analysis.yml +++ b/.github/workflows/codeql-analysis.yml @@ -43,7 +43,7 @@ jobs: - name: Configure merklecpp working-directory: ${{github.workspace}}/build - run: cmake -DCMAKE_BUILD_TYPE=Debug -DTESTS=ON $GITHUB_WORKSPACE + run: cmake -DCMAKE_BUILD_TYPE=Debug -DTESTS=ON -DLONG_TESTS=ON $GITHUB_WORKSPACE - name: Build merklecpp working-directory: ${{github.workspace}}/build diff --git a/CMakeLists.txt b/CMakeLists.txt index e59b7b2..78122cd 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -14,6 +14,7 @@ option(EVERCRYPT "enable comparison with EverCrypt Merkle trees" OFF) option(OPENSSL "enable OpenSSL" OFF) option(TRACE "enable debug traces" OFF) option(CLANG_TIDY "enable clang-tidy checks during build" OFF) +option(LONG_TESTS "enable long-running tests" OFF) if(CLANG_TIDY) find_program(CLANG_TIDY_PROGRAM clang-tidy) diff --git a/README.md b/README.md index 04f93a2..9be2918 100644 --- a/README.md +++ b/README.md @@ -60,6 +60,24 @@ and [doc/design/tlog-tiles.md](doc/design/tlog-tiles.md) for the full design, file/directory layout, and the proof algorithms. +## Building and testing + +Enable the test suite with CMake's `TESTS` option: + + cmake -S . -B build -DTESTS=ON + cmake --build build + ctest --test-dir build + +Some tile coverage is intentionally long-running. `LONG_TESTS` is off by +default for local builds; turn it on when you want the full tile stress suite, +including level-2 tile coverage and tile proof timing: + + cmake -S . -B build -DTESTS=ON -DLONG_TESTS=ON + +The repository CI enables `LONG_TESTS` so pull requests continue to exercise the +full tiled-storage matrix. + + ## Contributing This project welcomes contributions and suggestions. Most contributions require you to agree to a diff --git a/merklecpp_tiles.h b/merklecpp_tiles.h index 4d0ab4f..2f2c20d 100644 --- a/merklecpp_tiles.h +++ b/merklecpp_tiles.h @@ -199,7 +199,21 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) /// @brief Whether a full entry bundle exists on disk. [[nodiscard]] bool has_entry_bundle(uint64_t index) const { - return std::filesystem::exists(entries_path(index)); + std::error_code ec; + const auto path = entries_path(index); + if (!std::filesystem::is_regular_file(path, ec) || ec) + { + return false; + } + try + { + (void)decode_entries(read_file(path), TILE_WIDTH); + return true; + } + catch (const std::exception&) + { + return false; + } } /// @brief Writes a full entry bundle to disk atomically. @@ -254,14 +268,14 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) size_t pos = 0; for (size_t i = 0; i < count; i++) { - if (pos + 2 > bytes.size()) + if (bytes.size() - pos < 2) { throw std::runtime_error("truncated entry bundle"); } const auto len = (uint16_t)(((uint16_t)bytes[pos] << 8) | bytes[pos + 1]); pos += 2; - if (pos + len > bytes.size()) + if (len > bytes.size() - pos) { throw std::runtime_error("truncated entry bundle"); } @@ -270,6 +284,10 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) bytes.begin() + static_cast(pos + len)); pos += len; } + if (pos != bytes.size()) + { + throw std::runtime_error("trailing bytes in entry bundle"); + } return out; } diff --git a/test/CMakeLists.txt b/test/CMakeLists.txt index 60ab6c5..35773fb 100644 --- a/test/CMakeLists.txt +++ b/test/CMakeLists.txt @@ -34,8 +34,11 @@ add_merklecpp_test(tiles_writer tiles_writer.cpp) add_merklecpp_test(tiles_proofs tiles_proofs.cpp) add_merklecpp_test(tiles_tree tiles_tree.cpp) add_merklecpp_test(tiles_entries tiles_entries.cpp) -add_merklecpp_test(tiles_level2 tiles_level2.cpp) -add_merklecpp_test(time_tiles time_tiles.cpp) + +if(LONG_TESTS) + add_merklecpp_test(tiles_level2 tiles_level2.cpp) + add_merklecpp_test(time_tiles time_tiles.cpp) +endif() find_path(DOCTEST_DIR doctest.h) if(NOT DOCTEST_DIR) diff --git a/test/tiles_entries.cpp b/test/tiles_entries.cpp index 92b44ba..8312f84 100644 --- a/test/tiles_entries.cpp +++ b/test/tiles_entries.cpp @@ -7,6 +7,7 @@ #include #include #include +#include #include #include #include @@ -29,6 +30,13 @@ static void expect(bool cond, const std::string& what) } } +static void overwrite_file(const fs::path& p, const std::vector& bytes) +{ + std::ofstream f(p, std::ios::binary | std::ios::trunc); + f.write( + reinterpret_cast(bytes.data()), (std::streamsize)bytes.size()); +} + int main() { const auto seed = std::time(nullptr); @@ -50,6 +58,19 @@ int main() expect(enc.size() == 8 + 0 + 1 + 3 + 300, "encoded size"); const auto dec = TileStore::decode_entries(enc, entries.size()); expect(dec == entries, "encode/decode round-trip"); + + auto trailing = enc; + trailing.push_back(0xFF); + bool trailing_threw = false; + try + { + (void)TileStore::decode_entries(trailing, entries.size()); + } + catch (const std::exception&) + { + trailing_threw = true; + } + expect(trailing_threw, "decode rejects trailing bytes"); } const auto hashes = make_hashes(70000); @@ -106,6 +127,13 @@ int main() expect(writer.write_up_to(256, entry_at).full_written == 1, "inc s1"); const auto s2 = writer.write_up_to(256, entry_at); expect(s2.full_written == 0, "inc immutable rerun"); + overwrite_file(store.entries_path(0), {0x00, 0x05, 0x01}); + expect(!store.has_entry_bundle(0), "inc corrupt bundle not durable"); + EntryBundleWriter resumed(store); + expect( + resumed.write_up_to(256, entry_at).full_written == 1, + "inc rewrite corrupt"); + expect(store.has_entry_bundle(0), "inc rewritten bundle durable"); const auto s3 = writer.write_up_to(600, entry_at); expect(s3.full_written == 1, "inc s3 full"); expect(store.has_entry_bundle(1), "inc bundle 1"); diff --git a/test/tiles_store.cpp b/test/tiles_store.cpp index f669a0d..da6a9f2 100644 --- a/test/tiles_store.cpp +++ b/test/tiles_store.cpp @@ -178,21 +178,37 @@ int main() entries[i] = {(uint8_t)i, 0x7F}; } store.write_entry_bundle(0, entries); + expect(store.has_entry_bundle(0), "bundle durable before truncation"); expect( store.read_entry_bundle(0).size() == merkle::tiles::TILE_WIDTH, "bundle valid before truncation"); // Claims a 5-byte entry but supplies only one trailing byte. overwrite_file(store.entries_path(0), {0x00, 0x05, 0x01}); + expect(!store.has_entry_bundle(0), "truncated bundle is not durable"); expect_throws( [&] { (void)store.read_entry_bundle(0); }, "truncated entry bundle rejected"); + // A syntactically complete bundle with trailing bytes is not durable. + auto encoded = merkle::tiles::TileStore::encode_entries(entries); + encoded.push_back(0xFF); + overwrite_file(store.entries_path(0), encoded); + expect(!store.has_entry_bundle(0), "trailing bundle is not durable"); + expect_throws( + [&] { (void)store.read_entry_bundle(0); }, + "trailing entry bundle rejected"); + // decode_entries directly: a length prefix that overruns the buffer. expect_throws( [] { (void)merkle::tiles::TileStore::decode_entries({0xFF, 0xFF, 0x00}, 1); }, "decode_entries oversized length rejected"); + expect_throws( + [] { + (void)merkle::tiles::TileStore::decode_entries({0x00, 0x00, 0xFF}, 1); + }, + "decode_entries trailing bytes rejected"); } // Windows file replacement semantics can reject racing same-path replaces even From ade7854a24378a12110cbe8469baaf46aee8b710 Mon Sep 17 00:00:00 2001 From: achamayou Date: Sat, 11 Jul 2026 08:11:25 +0000 Subject: [PATCH 22/30] Fix tiled tree move semantics Rebind the tile writer to the destination store during move construction, and disable unsafe copy and assignment operations. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/tiles-guide.md | 3 +++ merklecpp_tiles.h | 14 +++++++++++++ test/tiles_tree.cpp | 49 +++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 66 insertions(+) diff --git a/doc/tiles-guide.md b/doc/tiles-guide.md index 294caef..7045399 100644 --- a/doc/tiles-guide.md +++ b/doc/tiles-guide.md @@ -75,6 +75,9 @@ assert(inclusion->verify(root)); auto consistency = log.consistency_proof(/*m=*/100, /*n=*/n); ``` +`TiledTree` can be move-constructed, but it cannot be copied or assigned. Move +construction keeps its writer bound to the destination tree's tile store. + `flush()` is incremental: each call writes only the full tiles that became complete since the previous call. Full tiles are immutable — written once and never rewritten — and the incomplete frontier is never tiled (it stays in diff --git a/merklecpp_tiles.h b/merklecpp_tiles.h index 2f2c20d..e53f495 100644 --- a/merklecpp_tiles.h +++ b/merklecpp_tiles.h @@ -1317,6 +1317,20 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) config(std::move(config)), store(this->config.prefix), writer(store) {} + TiledTreeT(const TiledTreeT&) = delete; + TiledTreeT& operator=(const TiledTreeT&) = delete; + + /// @brief Moves a tiled tree, rebinding its writer to the moved store. + TiledTreeT(TiledTreeT&& other) : + config(std::move(other.config)), + store(std::move(other.store)), + writer(store), + tree(std::move(other.tree)), + tiles_size(std::exchange(other.tiles_size, 0)) + {} + + TiledTreeT& operator=(TiledTreeT&&) = delete; + /// @brief Appends a leaf hash. void append(const Hash& leaf_hash) { diff --git a/test/tiles_tree.cpp b/test/tiles_tree.cpp index f009d90..d8e03f7 100644 --- a/test/tiles_tree.cpp +++ b/test/tiles_tree.cpp @@ -12,6 +12,7 @@ #include #include #include +#include #include #include @@ -21,6 +22,11 @@ using merkle::tiles::MemoryHashSource; using merkle::tiles::ProofEngine; using merkle::tiles::TiledTree; +static_assert(!std::is_copy_constructible_v); +static_assert(!std::is_copy_assignable_v); +static_assert(std::is_move_constructible_v); +static_assert(!std::is_move_assignable_v); + class ProofEngineProbe : public ProofEngine { public: @@ -149,6 +155,49 @@ int main() std::cout << "empty tree: OK" << '\n'; } + // ---- Part 0b: moving a tree rebinds its writer to the destination store. + { + TiledTree::Config move_cfg; + move_cfg.prefix = base / "tt_move_expected"; + TiledTree source(move_cfg); + for (uint64_t i = 0; i < 300; i++) + { + source.append(hashes[i]); + } + source.flush(); + + const fs::path work = base / "tt_move_cwd"; + fs::create_directories(work); + const fs::path previous_cwd = fs::current_path(); + fs::current_path(work); + try + { + TiledTree moved(std::move(source)); + expect(moved.flushed_size() == 256, "move: flushed size retained"); + expect( + moved.store_ref().root() == move_cfg.prefix, + "move: destination store retained"); + for (uint64_t i = 300; i < 512; i++) + { + moved.append(hashes[i]); + } + expect(moved.flush().full_written == 1, "move: next tile written"); + expect( + fs::is_regular_file(move_cfg.prefix / "tile/0/001"), + "move: tile written to configured store"); + expect( + !fs::exists(work / "tile/0/001"), + "move: no tile written relative to current directory"); + } + catch (...) + { + fs::current_path(previous_cwd); + throw; + } + fs::current_path(previous_cwd); + std::cout << "move construction: OK" << '\n'; + } + // ---- Part 1: memory-source proofs (exercises TreeT::subtree_root). for (const uint64_t n : {(uint64_t)1, From 37d67166c1b6097a60830f5cf415e603f2ee513e Mon Sep 17 00:00:00 2001 From: achamayou Date: Sat, 11 Jul 2026 10:09:32 +0000 Subject: [PATCH 23/30] Mark tiled tree move noexcept Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- merklecpp_tiles.h | 2 +- test/tiles_tree.cpp | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/merklecpp_tiles.h b/merklecpp_tiles.h index e53f495..8eaab41 100644 --- a/merklecpp_tiles.h +++ b/merklecpp_tiles.h @@ -1321,7 +1321,7 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) TiledTreeT& operator=(const TiledTreeT&) = delete; /// @brief Moves a tiled tree, rebinding its writer to the moved store. - TiledTreeT(TiledTreeT&& other) : + TiledTreeT(TiledTreeT&& other) noexcept : config(std::move(other.config)), store(std::move(other.store)), writer(store), diff --git a/test/tiles_tree.cpp b/test/tiles_tree.cpp index d8e03f7..ec55cc5 100644 --- a/test/tiles_tree.cpp +++ b/test/tiles_tree.cpp @@ -24,7 +24,7 @@ using merkle::tiles::TiledTree; static_assert(!std::is_copy_constructible_v); static_assert(!std::is_copy_assignable_v); -static_assert(std::is_move_constructible_v); +static_assert(std::is_nothrow_move_constructible_v); static_assert(!std::is_move_assignable_v); class ProofEngineProbe : public ProofEngine From 90c2b987ed6be7dab6fb71baa8da3f8ab73a5c9e Mon Sep 17 00:00:00 2001 From: achamayou Date: Sat, 11 Jul 2026 10:16:41 +0000 Subject: [PATCH 24/30] Clarify full-tile-only storage Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- audit.md | 9 +---- doc/design/tlog-tiles.md | 85 +++++++++++++++++++--------------------- doc/tiles-guide.md | 14 +++---- merklecpp_tiles.h | 46 +++++++++++----------- test/tiles_writer.cpp | 42 ++++++++------------ 5 files changed, 87 insertions(+), 109 deletions(-) diff --git a/audit.md b/audit.md index c64cd96..fe6f044 100644 --- a/audit.md +++ b/audit.md @@ -57,7 +57,6 @@ and (c) several **robustness / hostile-input / test / doc** gaps. | T6 | Low | Tests | **Fixed:** empty tree and `65537` boundary coverage added | | D1 | Medium | Docs | **Fixed:** design doc now matches the per-source tile cache | | D2 | Low | Docs | **Fixed:** durability wording now describes synced temp-file atomic replace and bad-size tile rewrite | -| D3 | Low | Docs | Partial-tile / spec material retained as background; correct but worth a sharper "not implemented" caveat | --- @@ -237,10 +236,6 @@ when that option was removed. implemented per-source tile cache. * **D2 (Low)** — Fixed: `doc/design/tlog-tiles.md` and the code comment now describe synced unique-temp writes, atomic replace, and bad-size tile rewrite. -* **D3 (Low)** — Partial-tile spec material remains as background (`§2.1`, `§4`). - It is consistently caveated as the upstream format and "not produced or read - here", so this is acceptable; a one-line "the implementation has no notion of a - partial tile" near each occurrence would remove any doubt. --- @@ -288,6 +283,4 @@ Recorded so the breadth of the review is auditable: ## Recommended priority All implementation, test, and documentation findings from this audit have now -been addressed except **D3**, which remains a low-priority wording polish item: -the design doc still includes partial-tile spec background, but consistently -caveats that this implementation does not write or read partial tiles. +been addressed. diff --git a/doc/design/tlog-tiles.md b/doc/design/tlog-tiles.md index 448b95a..fc01805 100644 --- a/doc/design/tlog-tiles.md +++ b/doc/design/tlog-tiles.md @@ -16,9 +16,8 @@ tiles, from the in-memory tree, or from a combination of the two. 1. Write the tree to disk as a set of immutable, cacheable **tile files** using the [tlog-tiles](https://c2sp.org/tlog-tiles) file/directory layout and tile geometry (256-wide tiles, 8 tree levels per tile, path encoding). Only - **full, balanced** tiles are produced: the incomplete frontier is **never - tiled** (no partial tiles are written or read) and is instead kept in the - in-memory tree until it grows into a full tile. + **full, balanced** tiles are produced. The remaining frontier stays in the + in-memory tree until it crosses the next full-tile boundary. 2. Produce tiles **progressively**: each batch/flush writes only the newly completed tiles, and the write integrates with the existing `flush_to()` "compaction" so that flushed (evicted) subtrees are durably persisted before @@ -32,6 +31,13 @@ tiles, from the in-memory tree, or from a combination of the two. 4. Keep everything **header-only** and templated, matching the existing `TreeT` style. +### Full-tile-only policy + +Partial tiles are out of scope: merklecpp never emits or reads them. During a +flush, a tile becomes eligible only after all 256 entries are complete and +final. Entries beyond that boundary remain in memory. Once published, a full +tile is immutable and is never rewritten. + ### Hard constraints (from the request) - **Hashing logic is unchanged.** No modification to `HASH_FUNCTION`, to node @@ -58,12 +64,11 @@ tiles, from the in-memory tree, or from a combination of the two. A tiled log exposes the Merkle tree as a set of static resources: -- **Tiles** at `/tile//[.p/]`, `application/octet-stream`. +- **Tiles** at `/tile//`, `application/octet-stream`. - `` = level, decimal `0..63`, no leading zeros. - `` = tile index within the level, encoded as zero-padded 3-digit path elements where **all but the last element are prefixed with `x`**. Example: `1234067` → `x001/x234/067`. - - `.p/` is present only for **partial** tiles; `` is the width `1..255`. - A **full tile** is exactly **256 hashes** wide (`256 * HASH_SIZE` bytes). The spec fixes `HASH_SIZE = 32` (SHA-256); here it is the template parameter. - Level 0 hashes are **leaf hashes**. At level `L ≥ 1`, each hash is the Merkle @@ -73,19 +78,11 @@ A tiled log exposes the Merkle tree as a set of static resources: - The `n`-th tile at level `l` contains, for `i = 0..255`: `MTH(D[(n*256+i)*256^l : (n*256+i+1)*256^l])`. Its **start index** is `n*256^(l+1)`, its **end index** `(n+1)*256^(l+1)`. -- Some rightmost tiles are **partial**. The partial tile at level `l` for tree - size `s` has `floor(s / 256^l) mod 256` entries. **Empty tiles MUST NOT be - served.** Partial-tile entries are still complete `256^l`-leaf subtree roots; - only the *count* is partial. - - > **merklecpp deviation.** This implementation never writes or reads partial - > *tiles*. The incomplete frontier (everything past the last full tile) is held - > in the in-memory tree, not on disk, so the on-disk tile set is the full-tile - > prefix only. Entry bundles (below) follow the same rule: only full bundles - > are written, and the incomplete tail stays with the application. +- The frontier beyond the last full tile is held in the in-memory tree, so the + on-disk tile set is always a full-tile prefix. - **Entry bundles** at `/tile/entries/`: big-endian `uint16` - length-prefixed raw entries, 256 per full bundle (full bundles only — see the - deviation above). (See [§6.8](#68-entry-bundles-optional).) + length-prefixed raw entries, 256 per full bundle. (See + [section 6.8](#68-entry-bundles-optional).) - **Pruning**: a log keeps a *minimum index*; tiles/bundles whose end index is `≤ minimum index` may be denied. This maps cleanly onto merklecpp's `flush_to()` / `min_index()`. @@ -115,8 +112,8 @@ A tiled log exposes the Merkle tree as a set of static resources: ### 2.3 Compatibility statement The tile geometry, path encoding, and the inclusion / consistency proof -**algorithms** are exactly those of tlog-tiles / RFC 6962 (we emit only the -full tiles, never the partial tiles). +**algorithms** are exactly those of tlog-tiles / RFC 6962. The implementation +emits only immutable full tiles. The **hash values stored in tiles** are produced by the tree's existing `HASH_FUNCTION`. Because @@ -155,11 +152,11 @@ tree's `HASH_FUNCTION`; `MTH` denotes the Merkle Tree Hash computed with it. aligned entries is `perfect_root` of that run. Hence any RFC 6962 subtree root at tree level `k = 8L + r` is obtained from **one** level-`L` tile by hashing `2^r` consecutive entries (a single entry when `r = 0`). -- **Full-tile prefix (no partials).** Only complete level-`L` tiles are written: +- **Full-tile prefix.** Only complete level-`L` tiles are written: `full_tiles_L = floor( floor(s / 256^L) / 256 )`, together covering the leaf prefix `covered = floor(s / 256) * 256`. The incomplete frontier `[covered, s)` - is not tiled; it is served from the in-memory tree. A subtree that falls in - what would have been a partial tile is resolved by **descending to the highest + is not tiled; it is served from the in-memory tree. When no higher-level full + tile contains a subtree, it is resolved by **descending to the highest available full tile and rolling up** (ultimately from full level-0 tiles). This yields a single read primitive that both proof types are built on: @@ -170,7 +167,7 @@ subtree_root(level k, index j) = MTH(D[j·2^k : (j+1)·2^k]) # within `covered first = j << r ; n = first / 256 ; off = first % 256 if a full level-L tile n exists (first + 2^r within full_tiles_L · 256): return perfect_root( tile(L, n)[off : off + 2^r] ) # single entry if r==0 - else: # would-be partial: split & descend + else: # higher-level tile unavailable: descend return HASH_FUNCTION( subtree_root(k-1, 2j), subtree_root(k-1, 2j+1) ) ``` @@ -225,8 +222,6 @@ Resource paths: - Full tile: `tile//` (the only tiles this implementation writes) -- Partial tile (tlog-tiles format; **not produced or read here**): - `tile//.p/` - Entry bundle: `tile/entries/` (full bundles only) Tile byte format: the 256 entries concatenated, each `HASH_SIZE` raw bytes @@ -357,8 +352,8 @@ public: crash should leave either the old tile or the new complete tile visible; a wrong-size tile is not treated as a durable full tile and is rewritten by the writer. -- Full tiles are written once and never rewritten (immutability). No partial - tiles are produced, so every file under `tile//` is write-once. +- Full tiles are written once and never rewritten (immutability), so every file + under `tile//` is write-once. - A small in-process cache of recently read tiles avoids repeated I/O during proof generation. @@ -369,9 +364,9 @@ class TileWriterT { public: explicit TileWriterT(TileStoreT& store); - // Persist all newly-complete full tiles (all levels 0..63). No partial tiles - // are written. Incremental: only writes tiles not already present. Higher - // levels are always rolled up, since proof generation relies on them. + // Persist all newly-complete full tiles (all levels 0..63). Incremental: + // only writes tiles not already present. Higher levels are always rolled up, + // since proof generation relies on them. // `leaf_at` supplies level-0 leaf hashes for [0, size) (e.g. Tree::leaf). void write_up_to(uint64_t size, const std::function& leaf_at); @@ -386,8 +381,8 @@ for L = 0, 1, 2, ... while entries_L = size >> (8*L) is > 0: for N in [first_unwritten_full(L) .. full_L): # new full tiles only entries = [ entry(L, N*256 + i) for i in 0..255 ] store.write_tile({L, N}, entries) - # the rightmost, incomplete (partial) entries are NOT written; they stay in - # the in-memory tree until they complete a full tile + # rightmost incomplete entries are not written; they stay in memory until + # they complete a full tile entry(L, g): if L == 0: return leaf_at(g) @@ -580,8 +575,8 @@ Given the invariant, every leaf and every perfect subtree is resolvable: - `mth_range` resolves a perfect subtree directly when it lies wholly in tiles (`end ≤ covered`) or wholly in memory (`start ≥ min_index`); otherwise it splits and recurses, terminating at resolvable pieces (leaves at worst). A - subtree within `covered` whose level has no full tile (it would have been the - partial) is resolved by descending to the highest available full tile. + subtree within `covered` whose level has no completed full tile is resolved by + descending to the highest available full tile. Hence inclusion and consistency proofs are always producible after compaction, from full tiles alone (for the tiled prefix), from memory alone (when nothing @@ -612,12 +607,12 @@ to the application, exactly as the spec leaves it to log ecosystems. ## 9. Worked examples (used as test vectors) **Size 256:** exactly one full level-0 tile (`tile/0/000`). No level-1 tile (its -single entry is the un-tiled frontier root, kept in memory) and no partials. +single entry is the un-tiled frontier root, kept in memory). **Size 70 000:** 273 full level-0 tiles (`covered = 69 888`) + one full level-1 tile (`tile/1/000`, rolling up level-0 tiles 0..255) = **274 full tiles**. The -incomplete frontier at every level (the would-be partials) is not written; the -remaining 112 leaves and the higher-level roll-ups stay in the in-memory tree. +incomplete frontier at every level is not written; the remaining 112 leaves and +the higher-level roll-ups stay in the in-memory tree. These exact counts are asserted in tests. **Index encoding:** `1234067 → x001/x234/067`; `1000 → x001/000`; `255 → 255`. @@ -638,8 +633,8 @@ index-encoding vectors; tile byte round-trip. **Phase 2 — Write path.** `TileWriterT::write_up_to` (level-0 from `leaf_at`, roll-ups, incremental cursor; full tiles only — the frontier is never tiled). -*Tests:* sizes 256 & 70 000 produce exactly the expected full-tile set with no -partial directories; full-tile immutability (re-running writes nothing new). +*Tests:* sizes 256 & 70 000 produce exactly the expected full-tile set; +full-tile immutability (re-running writes nothing new). **Phase 3 — Hash sources & proof engine.** `TileHashSource`, `mth_range`, `ProofEngineT::root/inclusion_proof/consistency_proof/verify_consistency`. @@ -672,8 +667,8 @@ compatible": `past_path`. This is stronger than `verify()` and pins byte-compatibility. - **Root agreement.** `ProofEngineT::root(size)` == `tree.root()` / `tree.past_root(size-1)` for many random sizes. -- **Spec vectors.** Exact full-tile counts for sizes 256 and 70 000; the absence - of any partial-tile directory; index-encoding strings (incl. `x001/x234/067`). +- **Spec vectors.** Exact full-tile counts and on-disk file sets for sizes 256 + and 70 000; index-encoding strings (incl. `x001/x234/067`). - **Flush/compaction.** Cross-check proofs for flushed indices against a twin tree that was never flushed (same inserts) — roots must match and proofs must verify. @@ -694,10 +689,10 @@ compatible": the consumer's choice and out of scope. - **Filesystem dependency.** Tile I/O needs ``/``; isolated in the companion header so the core stays dependency-free. -- **No partial tiles.** The incomplete frontier is never tiled, so there are no - partial-tile rewrites to race a reader; every tile is write-once and immutable. - The cost is that a stand-alone tile reader cannot serve the frontier — that is - the in-memory tree's job (or the application must keep it elsewhere). +- **Immutable full tiles.** A tile is emitted only after all of its entries are + final, and every emitted tile is write-once. A stand-alone tile reader cannot + serve the frontier; that is the in-memory tree's job (or the application must + keep it elsewhere). - **`flush_to` alignment.** Compaction flushes only to a 256-multiple (minus retention) to uphold the coverage invariant; it also keeps at least one resident leaf, since `flush_to` cannot drain the whole tree. Enforced inside diff --git a/doc/tiles-guide.md b/doc/tiles-guide.md index 7045399..bb842d5 100644 --- a/doc/tiles-guide.md +++ b/doc/tiles-guide.md @@ -79,9 +79,9 @@ auto consistency = log.consistency_proof(/*m=*/100, /*n=*/n); construction keeps its writer bound to the destination tree's tile store. `flush()` is incremental: each call writes only the full tiles that became -complete since the previous call. Full tiles are immutable — written once and -never rewritten — and the incomplete frontier is never tiled (it stays in -memory until it grows into a full tile). +complete since the previous call. Full tiles are immutable: written once after +all 256 entries are final and never rewritten. The remaining frontier stays in +memory until it crosses the next full-tile boundary. Tile files are written through unique temporary files, synced, then published with an atomic replace. On POSIX systems the parent directory is also synced @@ -204,7 +204,7 @@ for (auto& leaf : batch) tree.insert(leaf); merkle::tiles::TileStore store("/var/log/mylog"); merkle::tiles::TileWriter writer(store); -// Write all newly-complete full tiles (the incomplete frontier is not tiled). +// Write all newly-complete full tiles; keep the remaining frontier in memory. auto stats = writer.write_up_to( tree.num_leaves(), [&](uint64_t i) -> const merkle::Hash& { return tree.leaf(i); }); @@ -269,8 +269,8 @@ Under the configured `prefix`: ``` Tile indices use the tlog-tiles path encoding: zero-padded 3-digit groups with -all but the last prefixed by `x` (e.g. index `1234067` → `x001/x234/067`). All -tiles are full (256-wide) and immutable; the incomplete frontier is never tiled. -See +all but the last prefixed by `x` (e.g. index `1234067` -> `x001/x234/067`). Every +tile is full (256-wide), final, and immutable. Entries beyond the last full-tile +boundary remain in memory. See [`design/tlog-tiles.md`](design/tlog-tiles.md) for the full specification of the geometry and proof algorithms. diff --git a/merklecpp_tiles.h b/merklecpp_tiles.h index 8eaab41..5f11f64 100644 --- a/merklecpp_tiles.h +++ b/merklecpp_tiles.h @@ -32,12 +32,11 @@ # include #endif -// Tiled storage for merklecpp trees, following the C2SP tlog-tiles layout -// (https://c2sp.org/tlog-tiles). The tile geometry, path encoding and partial -// tile rules match the specification; the hash values stored in tiles are -// produced by the tree's existing HASH_FUNCTION, so tile-derived proofs are -// byte-identical to those produced by merkle::TreeT (see -// doc/design/tlog-tiles.md). +// Tiled storage for merklecpp trees, following the full-tile geometry and path +// encoding of the C2SP tlog-tiles layout (https://c2sp.org/tlog-tiles). Only +// complete, immutable tiles are stored. Their hash values are produced by the +// tree's existing HASH_FUNCTION, so tile-derived proofs are byte-identical to +// those produced by merkle::TreeT (see doc/design/tlog-tiles.md). namespace merkle // NOLINT(modernize-concat-nested-namespaces) { @@ -601,10 +600,9 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) /// @tparam HASH_FUNCTION The tree's node hash function /// @note Only balanced subtrees are tiled: a level-L entry is the root of a /// complete 2**(8L)-leaf subtree. Only full tiles (256 such entries) are - /// written; they are therefore immutable and written exactly once. The - /// incomplete frontier is deliberately never tiled (no partial tiles are - /// produced): it is retained in memory by the owning tree until it grows - /// into a full tile. + /// written; they are therefore immutable and written exactly once. Entries + /// beyond the last full-tile boundary remain in memory until a later flush + /// completes the next tile. template < size_t HASH_SIZE, void HASH_FUNCTION( @@ -638,9 +636,10 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) /// [0, size); only ever queried for leaves of complete subtrees. /// @return Counts of tiles written /// @note Incremental: full tiles already on disk are immutable and are - /// never rewritten. The incomplete frontier is never tiled. Tiles are - /// always rolled up at every level (0..63), so the on-disk set always - /// contains the higher-level roll-ups that proof generation relies on. + /// never rewritten. Entries that do not complete a tile are not written. + /// Tiles are always rolled up at every level (0..63), so the on-disk set + /// always contains the higher-level roll-ups that proof generation relies + /// on. Stats write_up_to(uint64_t size, const LeafFn& leaf_at) { Stats stats; @@ -805,10 +804,10 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) /// @brief Resolves subtree roots from tlog-tiles tile files. /// @note @p available_size is rounded down to a whole number of full tiles: - /// only complete, durably-written full tiles are read (no partial tiles). - /// A complete subtree within that full-tile prefix is resolvable; anything - /// reaching into the incomplete frontier yields false so that a proof - /// builder can fall back to another source (e.g. an in-memory tree). + /// only complete, durably-written full tiles are read. A complete subtree + /// within that full-tile prefix is resolvable; anything reaching into the + /// incomplete frontier yields false so that a proof builder can fall back + /// to another source (e.g. an in-memory tree). template < size_t HASH_SIZE, void HASH_FUNCTION( @@ -861,9 +860,9 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) /// @brief Resolves a complete subtree known to lie within the full-tile /// prefix, reading the highest-level full tile that holds it (and rolling - /// up); descends to lower full tiles where a higher-level tile is not - /// full (it would otherwise have been the unwritten partial). Terminates - /// because full level-0 tiles always cover the prefix. + /// up); descends to lower full tiles when a higher-level full tile has not + /// completed. Terminates because full level-0 tiles always cover the + /// prefix. void resolve(uint8_t level, uint64_t index, Hash& out) const { if (level <= TILE_HEIGHT) @@ -1501,10 +1500,9 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) /// stores leaf hashes, while the raw entries (and the leaf-hash derivation /// linking each entry to its level-0 tile hash) are the application's /// responsibility. Only full bundles (TILE_WIDTH entries) are written; they - /// are immutable and written exactly once. The incomplete tail is never - /// bundled (no partial bundles are produced): it stays with the application - /// until it grows into a full bundle, mirroring the un-tiled Merkle - /// frontier. + /// are immutable and written exactly once. The incomplete tail stays with + /// the application until it grows into a full bundle, mirroring the + /// un-tiled Merkle frontier. template < size_t HASH_SIZE, void HASH_FUNCTION( diff --git a/test/tiles_writer.cpp b/test/tiles_writer.cpp index d100361..3f4ff49 100644 --- a/test/tiles_writer.cpp +++ b/test/tiles_writer.cpp @@ -29,24 +29,18 @@ static void expect(bool cond, const std::string& what) } } -// True if any partial-tile directory (".p") exists anywhere in the store: the -// writer must never produce one. -static bool any_partial_dirs(const TileStore& store) +static size_t tile_file_count(const TileStore& store) { const fs::path tiles = store.root() / "tile"; if (!fs::exists(tiles)) { - return false; + return 0; } const fs::recursive_directory_iterator it(tiles); - return std::any_of(begin(it), end(it), [](const fs::directory_entry& e) { - if (!e.is_directory()) - { - return false; - } - const std::string name = e.path().filename().string(); - return name.size() >= 2 && name.substr(name.size() - 2) == ".p"; - }); + return (size_t)std::count_if( + begin(it), end(it), [](const fs::directory_entry& e) { + return e.is_regular_file(); + }); } // Roll up a full level-0 tile and compare with a level-1 tile entry. @@ -67,9 +61,9 @@ int main() try { - // ---- A. size 256: exactly one full L0 tile, no partials and no L1 tile - // (the L1 root is not yet a full tile; it stays in memory). The - // level-1 entry it would hold equals the real merkle::Tree root. + // ---- A. size 256: exactly one full L0 tile and no L1 tile (the L1 root is + // not yet a full tile, so it stays in memory). The level-1 entry it + // would hold equals the real merkle::Tree root. { const auto hashes = make_hashes(256); const auto leaf_at = [&](uint64_t i) -> const Hash& { return hashes[i]; }; @@ -85,7 +79,7 @@ int main() fs::file_size(store.tile_path(TileRef{0, 0})) == 256U * Hash().size(), "A L0 full tile size"); expect(!store.has_full_tile(1, 0), "A no L1 full tile"); - expect(!any_partial_dirs(store), "A no partial tiles"); + expect(tile_file_count(store) == 1, "A exact tile file count"); // Level-0 tile is the leaf hashes verbatim. const auto l0 = store.read_tile(TileRef{0, 0}); @@ -126,7 +120,7 @@ int main() expect(!fs::exists(store.root() / "tile" / "2"), "B no level 2"); expect(!fs::exists(store.root() / "tile" / "3"), "B no level 3"); - expect(!any_partial_dirs(store), "B no partial tiles"); + expect(tile_file_count(store) == 274, "B exact tile file count"); // Higher-level entries are roll-ups of the complete child tiles. const auto l1 = store.read_tile(TileRef{1, 0}); @@ -137,8 +131,7 @@ int main() std::cout << "B (size 70000): OK" << '\n'; } - // ---- C. incremental writes: immutability and idempotency; never a - // partial. + // ---- C. incremental writes preserve immutability and idempotency. { const auto hashes = make_hashes(1024); const auto leaf_at = [&](uint64_t i) -> const Hash& { return hashes[i]; }; @@ -161,14 +154,13 @@ int main() expect(store.has_full_tile(0, 1), "C L0 tile 1"); expect(!store.has_full_tile(0, 2), "C no L0 tile 2"); expect(!store.has_full_tile(1, 0), "C no L1 full tile"); - expect(!any_partial_dirs(store), "C no partial tiles"); + expect(tile_file_count(store) == 2, "C exact tile file count"); std::cout << "C (incremental): OK" << '\n'; } // ---- D. crossing into a full level-1 tile: the roll-up appears as a full - // tile, prior full tiles are never rewritten, and no partial is - // ever produced. + // tile and prior full tiles are never rewritten. { const auto hashes = make_hashes(65536); const auto leaf_at = [&](uint64_t i) -> const Hash& { return hashes[i]; }; @@ -180,7 +172,7 @@ int main() const auto s1 = writer.write_up_to(65535, leaf_at); expect(s1.full_written == 255, "D s1 255 full L0"); expect(!store.has_full_tile(1, 0), "D no L1 before completion"); - expect(!any_partial_dirs(store), "D no partial before completion"); + expect(tile_file_count(store) == 255, "D initial tile file count"); // Completing the 256th L0 tile yields one new L0 tile and one full L1. const auto s2 = writer.write_up_to(65536, leaf_at); @@ -189,7 +181,7 @@ int main() expect(store.has_full_tile(1, 0), "D L1 tile 0"); expect(!store.has_full_tile(1, 1), "D no L1 tile 1"); expect(!store.has_full_tile(2, 0), "D no L2 tile"); - expect(!any_partial_dirs(store), "D no partial tiles"); + expect(tile_file_count(store) == 257, "D final tile file count"); const auto l1 = store.read_tile(TileRef{1, 0}); expect(l1[0] == rollup(store.read_tile(TileRef{0, 0})), "D L1[0]"); @@ -237,7 +229,7 @@ int main() store.read_tile(TileRef{1, 0})[0] == rollup(store.read_tile(TileRef{0, 0})), "E L1[0] rolled up from resumed L0"); - expect(!any_partial_dirs(store), "E no partial tiles"); + expect(tile_file_count(store) == 274, "E exact tile file count"); // A third fresh writer confirms full idempotence after a resume. TileStore store3(dir); From 74e63c06b378225a78c9a904f77c540fcd0eb077 Mon Sep 17 00:00:00 2001 From: achamayou Date: Sat, 11 Jul 2026 10:43:04 +0000 Subject: [PATCH 25/30] Seal interrupted tile flushes Document the caller-synchronized concurrency contract for tiled storage and proof caches. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/tlog-tiles.md | 73 +++++++++++++++++++++------- doc/tiles-guide.md | 41 +++++++++++----- merklecpp_tiles.h | 62 +++++++++++++++++++----- test/tiles_tree.cpp | 102 +++++++++++++++++++++++++++++++++++++-- 4 files changed, 234 insertions(+), 44 deletions(-) diff --git a/doc/design/tlog-tiles.md b/doc/design/tlog-tiles.md index fc01805..1200f14 100644 --- a/doc/design/tlog-tiles.md +++ b/doc/design/tlog-tiles.md @@ -38,6 +38,14 @@ flush, a tile becomes eligible only after all 256 entries are complete and final. Entries beyond that boundary remain in memory. Once published, a full tile is immutable and is never rewritten. +### Thread-safety policy + +The tiled-storage API provides no internal synchronization. Every object is +single-threaded unless its caller serializes all access. This requirement also +applies to separate objects that share a store prefix and to methods declared +`const`, because proof reads update the `TileHashSourceT` LRU cache. Locking and +reader/writer coordination are deliberately left to the application. + ### Hard constraints (from the request) - **Hashing logic is unchanged.** No modification to `HASH_FUNCTION`, to node @@ -356,6 +364,8 @@ public: under `tile//` is write-once. - A small in-process cache of recently read tiles avoids repeated I/O during proof generation. +- `TileStoreT` does not synchronize access. Callers must serialize all + operations on a store and across stores that share a prefix. ### 6.4 Write path — `TileWriterT` (progressive, on compaction) @@ -398,13 +408,19 @@ entry(L, g): Compaction integration (the "on compaction" story): ```cpp -// In TiledTreeT::flush(): write durable full tiles, THEN free memory. -writer.write_up_to(size, [&](uint64_t i) -> const Hash& { return tree.leaf(i); }); +// In TiledTreeT::flush(): seal, write durable full tiles, THEN free memory. uint64_t covered = (size / TILE_WIDTH) * TILE_WIDTH; // full level-0 coverage -flushed_size = covered; // only full tiles durable +immutable_size = max(immutable_size, covered); // before any write +writer.write_up_to(size, [&](uint64_t i) -> const Hash& { return tree.leaf(i); }); +flushed_size = covered; // after all levels succeed // compact() drops only [min_index, covered); the frontier [covered, size) stays. ``` +If `write_up_to` throws, `immutable_size` remains advanced because a full tile +may already be visible, while `flushed_size` remains at its previous successful +boundary. The in-memory tree is not compacted. The caller fixes the I/O error +and retries with the same tree state; existing finalized tiles are reused. + ### 6.5 `HashSource` — tiles, memory, or both ```cpp @@ -425,6 +441,9 @@ struct HashSource { // concept (duck-typed or virtual) (`≥ min_index`). - `CombinedHashSource{mem, tiles}` — try memory first (no I/O), then tiles (configurable order). This is the "combination of tiles and in-memory tree". +- `TileHashSource` mutates its LRU cache during `const` reads. It and every + `ProofEngine` that refers to it require caller-provided synchronization when + shared between threads. ### 6.6 `ProofEngineT` — inclusion & consistency @@ -508,6 +527,8 @@ public: void append(const Hash& leaf_hash); // tree.insert uint64_t size() const; // tree.num_leaves Hash root(); // tree.root + uint64_t flushed_size() const; // successful tile boundary + uint64_t immutable_size() const; // rollback boundary // Write newly-complete full tiles. Compaction (dropping already-tiled // leaves from memory) happens only if compact_on_flush. @@ -518,6 +539,9 @@ public: // available from the tiles. uint64_t compact(); + // Roll back only beyond immutable_size(). + void retract_to(size_t index); + // Proofs over tiles ∪ resident tree (works for flushed indices). std::shared_ptr inclusion_proof(uint64_t index, uint64_t size); std::vector consistency_proof(uint64_t m, uint64_t n); @@ -526,6 +550,9 @@ public: }; ``` +`TiledTreeT` performs no internal locking. The caller must serialize every +operation on a shared instance, including proof calls. + Stateless alternative for callers with their own storage: free functions `write_tiles(tree, store, size, opts)` and a `ProofEngineT` built on a `CombinedHashSource`, so the wrapper is optional sugar. @@ -547,25 +574,33 @@ entry bundles are an **application-owned** add-on, included for completeness: ## 7. Progressive production & compaction -The pairing of tile writing with `flush_to` gives the central correctness -invariant: +The pairing of tile writing with `flush_to` gives two central correctness +invariants: -> **Coverage invariant.** Never flush past the durably-written full-level-0 -> coverage: `min_index() ≤ covered`, where -> `covered = floor(size / 256) * 256` after `write_up_to(size, …)`. +> **Compaction invariant.** Never compact past the last fully successful flush: +> `min_index() <= flushed_size`. +> +> **Immutability invariant.** Never roll back below a full-tile boundary that a +> flush may have published: `size >= immutable_size`. Per flush: 1. `append(...)` new leaf hashes; compute `root()`. -2. `write_up_to(size, leaf_at)` — persist newly-complete **full** tiles at all - levels (incremental). The incomplete frontier is **not** tiled, so - `flushed_size = covered = floor(size / 256) * 256`. -3. *(optional)* `compact()` → `flush_to(covered − retention_margin)` — reclaim +2. Compute `covered = floor(size / 256) * 256` and advance `immutable_size` to + `covered` before any write can publish a full tile. +3. `write_up_to(size, leaf_at)` - persist newly-complete **full** tiles at all + levels (incremental). +4. After every level succeeds, set `flushed_size = covered`. +5. *(optional)* `compact()` -> `flush_to(covered - retention_margin)` - reclaim memory, only when `compact_on_flush` is set (or `compact()` is called explicitly). It drops only the full-tile-covered prefix, so the un-tiled frontier always stays resident. By default nothing is dropped and the tree stays whole. +If step 3 fails, steps 4 and 5 do not run. `immutable_size` stays advanced to +prevent stale-tile rollback, while `flushed_size` stays at the last complete +all-level write so proofs and compaction do not trust an incomplete flush. + Given the invariant, every leaf and every perfect subtree is resolvable: - leaf `i < covered` ⇒ in a full level-0 tile; leaf `i ≥ min_index` ⇒ resident. @@ -698,11 +733,15 @@ compatible": resident leaf, since `flush_to` cannot drain the whole tree. Enforced inside `TiledTreeT::compact`. - **Rollback vs. immutable tiles.** Tiles are write-once, so rolling the tree - back (`retract_to`) over already-tiled entries would leave stale, never- - rewritten tiles and is forbidden: `TiledTreeT::retract_to` throws if the - resulting size is below `flushed_size()`, permitting only rollback of the - un-tiled (post-flush) frontier. Retracting the underlying tree directly - via `tree_ref()` bypasses this guard and must be avoided. + back (`retract_to`) over a range that a flush may have published would leave + stale, never-rewritten tiles. `TiledTreeT::retract_to` therefore throws if the + resulting size is below `immutable_size()`. A failed flush may advance + `immutable_size()` without advancing `flushed_size()`; retry with the same + tree state. Retracting the underlying tree directly via `tree_ref()` bypasses + this guard and must be avoided. +- **No internal synchronization.** Every tiled-storage object and shared store + prefix requires external serialization. This includes `const` proof reads, + which update the tile cache. - **Very large indices.** Index math uses `uint64_t`; encoding handles multi-group indices. Level bound `≤ 63` per spec (8 suffices for `2^64`). - **Open question — `subtree_root` in core vs. `past_path`-derived memory diff --git a/doc/tiles-guide.md b/doc/tiles-guide.md index bb842d5..f0e5ed6 100644 --- a/doc/tiles-guide.md +++ b/doc/tiles-guide.md @@ -14,6 +14,7 @@ page is a practical how-to. ## Contents - [Requirements and a note on hashing](#requirements-and-a-note-on-hashing) +- [Thread safety](#thread-safety) - [Quick start: `TiledTree`](#quick-start-tiledtree) - [Flushing and compaction](#flushing-and-compaction) - [Rollback](#rollback) @@ -44,6 +45,17 @@ page is a practical how-to. produces; they are not RFC 6962 unless you instantiate your tree with an RFC 6962 hash function (not required, and not the goal here). +## Thread safety + +The tiled-storage API provides no internal synchronization. Treat every +`TileStore`, `TileWriter`, `TileHashSource`, `ProofEngine`, `TiledTree`, and +`EntryBundleWriter` instance as single-threaded. + +If an object or store prefix is shared between threads, the caller must +serialize every operation. This includes methods declared `const`: proof +generation updates the `TileHashSource` LRU cache. The library deliberately +does not add locks or otherwise coordinate concurrent readers and writers. + ## Quick start: `TiledTree` `TiledTree` is the high-level wrapper: append leaf hashes, flush them to disk @@ -85,9 +97,8 @@ memory until it crosses the next full-tile boundary. Tile files are written through unique temporary files, synced, then published with an atomic replace. On POSIX systems the parent directory is also synced -after the rename. A wrong-size tile file is not considered durable by the -writer, so a later flush rewrites it instead of treating mere file existence as -success. +after the rename. A wrong-size file at a tile path is not a published tile, so +the writer does not treat mere file existence as success. ## Flushing and compaction @@ -113,6 +124,14 @@ merkle::tiles::TiledTree log(cfg); - Proofs for dropped leaves are still produced — they are served from the tiles and transparently combined with the resident frontier. +`flushed_size()` is the boundary completed successfully at every required tile +level, and it is the only boundary used for proof reads and compaction. +`immutable_size()` is the rollback boundary. A flush seals that boundary before +it starts writing, because an error can occur after a full tile becomes visible. +If a flush throws, `immutable_size()` may advance while `flushed_size()` does +not. Keep the same tree contents, correct the I/O failure, and retry `flush()`; +finalized tiles are reused rather than rewritten. + ```cpp log.compact(); // free memory now uint64_t resident_from = log.tree_ref().min_index(); @@ -120,19 +139,19 @@ uint64_t resident_from = log.tree_ref().min_index(); ## Rollback -Tiles are immutable, so you may only roll back entries that have **not yet been -committed to tiles** (i.e. appended since the last flush). `retract_to` -enforces this: +Tiles are immutable, so you may only roll back entries beyond the boundary +returned by `immutable_size()`. `retract_to` enforces this: ```cpp log.retract_to(index); // keep leaves [0, index], drop the rest ``` -- Allowed when the resulting size is `>= flushed_size()` — i.e. only the - un-tiled frontier is removed. `flushed_size()` is the full-tile-covered prefix - (a multiple of 256), so everything appended since the last full tile can be - rolled back. -- Throws otherwise (rolling back committed entries would leave stale tiles). +- Allowed when the resulting size is `>= immutable_size()`. +- Throws otherwise, because a flush may already have published an immutable + full tile for that range. +- After a successful flush, `immutable_size() == flushed_size()`. After an + interrupted flush, `immutable_size()` may be larger until the same tree state + is flushed successfully. - `retract_to` mirrors `merkle::Tree::retract_to`: `index` is the new *last* leaf, so the resulting size is `index + 1`. diff --git a/merklecpp_tiles.h b/merklecpp_tiles.h index 5f11f64..2d35ddf 100644 --- a/merklecpp_tiles.h +++ b/merklecpp_tiles.h @@ -37,6 +37,10 @@ // complete, immutable tiles are stored. Their hash values are produced by the // tree's existing HASH_FUNCTION, so tile-derived proofs are byte-identical to // those produced by merkle::TreeT (see doc/design/tlog-tiles.md). +// +// Thread safety: types in this header do not synchronize access internally. +// Callers must serialize all operations on shared objects and store prefixes, +// including const proof operations that update the tile cache. namespace merkle // NOLINT(modernize-concat-nested-namespaces) { @@ -97,6 +101,8 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) /// @tparam HASH_SIZE Size of each hash in bytes /// @tparam HASH_FUNCTION The tree's node hash function (carried for use by /// later components; tile I/O itself does not hash). + /// @warning No internal synchronization is provided. Callers must serialize + /// access to a store and to all stores that share its prefix. template < size_t HASH_SIZE, void HASH_FUNCTION( @@ -603,6 +609,8 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) /// written; they are therefore immutable and written exactly once. Entries /// beyond the last full-tile boundary remain in memory until a later flush /// completes the next tile. + /// @warning No internal synchronization is provided. Callers must serialize + /// access to a writer and its store. template < size_t HASH_SIZE, void HASH_FUNCTION( @@ -808,6 +816,9 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) /// within that full-tile prefix is resolvable; anything reaching into the /// incomplete frontier yields false so that a proof builder can fall back /// to another source (e.g. an in-memory tree). + /// @warning No internal synchronization is provided. Even const operations + /// update the internal LRU cache, so callers must serialize all access to a + /// shared source. template < size_t HASH_SIZE, void HASH_FUNCTION( @@ -937,6 +948,8 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) /// HASH_FUNCTION, so an inclusion proof is byte-identical to the one /// produced by merkle::TreeT::path()/past_path() and verifies with /// PathT::verify(). + /// @warning Thread safety is inherited from the supplied HashSource. Callers + /// must serialize operations when the source is shared. template < size_t HASH_SIZE, void HASH_FUNCTION( @@ -1282,6 +1295,8 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) /// Config::compact_on_flush, or call compact() explicitly; it never drops /// the un-tiled frontier. Proofs are served from the combination of the /// resident tree (frontier) and the full tiles (compacted past). + /// @warning No internal synchronization is provided. Callers must serialize + /// all access to a shared tree, including proof operations. template < size_t HASH_SIZE, void HASH_FUNCTION( @@ -1325,7 +1340,8 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) store(std::move(other.store)), writer(store), tree(std::move(other.tree)), - tiles_size(std::exchange(other.tiles_size, 0)) + tiles_size(std::exchange(other.tiles_size, 0)), + sealed_size(std::exchange(other.sealed_size, 0)) {} TiledTreeT& operator=(TiledTreeT&&) = delete; @@ -1348,13 +1364,24 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) return tree.root(); } - /// @brief The number of leaves durably written to full tiles (always a - /// multiple of TILE_WIDTH; the incomplete frontier is not tiled). + /// @brief The number of leaves covered by the last fully successful flush. + /// @note This is always a multiple of TILE_WIDTH. It advances only after + /// every required tile level has been written successfully, and controls + /// proof reads and compaction. [[nodiscard]] uint64_t flushed_size() const { return tiles_size; } + /// @brief The rollback seal for ranges a flush may have published. + /// @note A flush seals its full-tile boundary before writing. If the write + /// fails, this may exceed flushed_size(); keep the same tree contents and + /// retry the flush. + [[nodiscard]] uint64_t immutable_size() const + { + return sealed_size; + } + /// @brief Access to the underlying tree. Tree& tree_ref() { @@ -1370,9 +1397,11 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) /// @brief Writes newly-complete full tiles to disk; compacts only if /// Config::compact_on_flush is set. /// @return Counts of the full tiles written by this flush - /// @note Only full (balanced) tiles are written, so flushed_size() - /// advances to the largest multiple of TILE_WIDTH <= size(); the - /// incomplete frontier stays resident in memory. + /// @note The full-tile boundary is made immutable before any tile write. + /// Only after every required tile level succeeds does flushed_size() + /// advance to that boundary. On failure, immutable_size() may advance + /// while flushed_size() does not; the tree remains resident and the flush + /// can be retried without rewriting finalized tiles. Stats flush() { Stats stats; @@ -1382,10 +1411,16 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) return stats; } + const uint64_t covered = (n / TILE_WIDTH) * TILE_WIDTH; + if (covered > sealed_size) + { + sealed_size = covered; + } + stats = writer.write_up_to(n, [this](uint64_t i) -> const Hash& { return tree.leaf((size_t)i); }); - tiles_size = (n / TILE_WIDTH) * TILE_WIDTH; + tiles_size = covered; if (config.compact_on_flush) { @@ -1427,17 +1462,17 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) /// @brief Rolls the tree back so that @p index becomes the last leaf, /// removing all leaves after it (same semantics as TreeT::retract_to). /// @note Only full tiles are immutable: this throws if the resulting size - /// would be smaller than flushed_size() (the full-tile prefix). The - /// un-tiled frontier (>= flushed_size()) may be freely rolled back. + /// would be smaller than immutable_size(). A failed flush may advance + /// immutable_size() without advancing flushed_size(). /// (Retracting the underlying tree directly via tree_ref() bypasses this /// guard and can leave stale tiles -- do not do that.) void retract_to(size_t index) { - if ((uint64_t)index + 1 < tiles_size) + if ((uint64_t)index + 1 < sealed_size) { throw std::runtime_error( - "TiledTree::retract_to: cannot roll back entries already committed " - "to immutable tiles (resulting size < flushed size)"); + "TiledTree::retract_to: cannot roll back entries sealed for " + "immutable tiles (resulting size < immutable size)"); } tree.retract_to(index); } @@ -1476,6 +1511,7 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) Writer writer; Tree tree; uint64_t tiles_size = 0; + uint64_t sealed_size = 0; /// @brief Builds a proof engine over the combined resident-tree /// (frontier) and full-tile (flushed past) source, and invokes @p fn with @@ -1503,6 +1539,8 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) /// are immutable and written exactly once. The incomplete tail stays with /// the application until it grows into a full bundle, mirroring the /// un-tiled Merkle frontier. + /// @warning No internal synchronization is provided. Callers must serialize + /// access to a writer and its store. template < size_t HASH_SIZE, void HASH_FUNCTION( diff --git a/test/tiles_tree.cpp b/test/tiles_tree.cpp index ec55cc5..3dbcb1c 100644 --- a/test/tiles_tree.cpp +++ b/test/tiles_tree.cpp @@ -141,6 +141,7 @@ int main() expect(ett.size() == 0, "empty: size 0"); expect(ett.flush().full_written == 0, "empty: flush writes nothing"); expect(ett.flushed_size() == 0, "empty: flushed size 0"); + expect(ett.immutable_size() == 0, "empty: immutable size 0"); expect(ett.compact() == 0, "empty: compact no-op"); bool ethrew = false; try @@ -174,6 +175,7 @@ int main() { TiledTree moved(std::move(source)); expect(moved.flushed_size() == 256, "move: flushed size retained"); + expect(moved.immutable_size() == 256, "move: immutable size retained"); expect( moved.store_ref().root() == move_cfg.prefix, "move: destination store retained"); @@ -451,7 +453,7 @@ int main() bool threw = false; try { - rtt.retract_to(700); // size 701 < flushed_size 768 + rtt.retract_to(700); // size 701 < immutable_size 768 } catch (const std::exception&) { @@ -536,7 +538,7 @@ int main() // Only the immutable full-tile prefix [0,256) is protected: rolling // back into it is refused, while rolling back within the un-tiled - // frontier (>= flushed_size()) is allowed. + // frontier (>= immutable_size()) is allowed. expect(rb.flushed_size() == 256, "rb flushed to full-tile prefix"); bool threw = false; @@ -559,7 +561,7 @@ int main() threw = true; } expect(threw, "rb retract just below tiled prefix throws"); - rb.retract_to(255); // size 256 == flushed_size(): drops only frontier + rb.retract_to(255); // size 256 == immutable_size(): drops only frontier expect(rb.size() == 256, "rb retract to tiled prefix allowed"); } @@ -580,7 +582,8 @@ int main() { rb.append(hashes[i]); } - rb.retract_to(1099); // frontier rollback (>= flushed_size()) is allowed + // Frontier rollback (>= immutable_size()) is allowed. + rb.retract_to(1099); expect(rb.size() == 1100, "rb compact frontier retract ok"); merkle::Tree exp_tree; @@ -604,6 +607,97 @@ int main() } expect(threw, "rb compact retract below tiled throws"); } + + // 3d. A failed flush may publish an immutable full tile before a later + // write fails. The attempted full-tile boundary is sealed against + // rollback, while flushed_size advances only after a successful retry. + { + TiledTree::Config cfg; + cfg.prefix = base / "rb_interrupted"; + TiledTree interrupted(cfg); + for (uint64_t i = 0; i < 512; i++) + { + interrupted.append(hashes[i]); + } + + const fs::path blocker = cfg.prefix / "tile/0/001"; + fs::create_directories(blocker); + bool flush_threw = false; + try + { + interrupted.flush(); + } + catch (const std::exception&) + { + flush_threw = true; + } + + expect(flush_threw, "interrupted flush throws"); + expect( + fs::is_regular_file(cfg.prefix / "tile/0/000"), + "interrupted flush published first full tile"); + expect( + interrupted.flushed_size() == 0, + "interrupted flush does not advance flushed size"); + expect( + interrupted.immutable_size() == 512, + "interrupted flush seals attempted boundary"); + expect( + interrupted.compact() == 0, + "interrupted flush does not permit compaction"); + + TiledTree recovered(std::move(interrupted)); + expect( + recovered.immutable_size() == 512, + "interrupted flush seal survives move"); + + bool retract_threw = false; + try + { + recovered.retract_to(0); + } + catch (const std::exception&) + { + retract_threw = true; + } + expect( + retract_threw, + "interrupted flush rejects rollback across attempted tiles"); + expect(recovered.size() == 512, "interrupted rollback changes nothing"); + + fs::remove(blocker); + expect( + recovered.flush().full_written == 1, + "interrupted flush retry writes only missing tile"); + expect( + recovered.flushed_size() == 512, + "interrupted flush retry advances flushed size"); + expect( + recovered.immutable_size() == 512, + "interrupted flush retry preserves immutable size"); + recovered.compact(); + + merkle::Tree expected; + for (uint64_t i = 0; i < 512; i++) + { + expected.insert(hashes[i]); + } + const Hash expected_root = expected.root(); + expect( + recovered.root() == expected_root, + "interrupted flush root matches reference"); + for (const uint64_t i : + {(uint64_t)0, (uint64_t)255, (uint64_t)256, (uint64_t)511}) + { + const auto proof = recovered.inclusion_proof(i, 512); + expect( + *proof == *expected.path(i), + "interrupted flush inclusion matches reference"); + expect( + proof->verify(expected_root), + "interrupted flush inclusion verifies"); + } + } } std::cout << "rollback: OK" << '\n'; From ce881a71427b7de7c96aab1d6972cc2ff1db3697 Mon Sep 17 00:00:00 2001 From: achamayou Date: Sat, 11 Jul 2026 16:09:55 +0000 Subject: [PATCH 26/30] Reject existing TiledTree data Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- README.md | 6 +++++ doc/design/tlog-tiles.md | 7 +++++ doc/tiles-guide.md | 9 +++++++ merklecpp_tiles.h | 55 ++++++++++++++++++++++++++++++++++++++-- test/tiles_tree.cpp | 33 ++++++++++++++++++++++++ 5 files changed, 108 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 9be2918..efadb1e 100644 --- a/README.md +++ b/README.md @@ -54,6 +54,12 @@ byte-identical to one from `merkle::Tree::path()` and verifies with the same auto consistency = log.consistency_proof(/*m=*/100, /*n=*/log.size()); +`TiledTree` creates a new tiled tree: the configured directory may exist, but +its `tile` subdirectory must be absent or empty. It deliberately rejects +existing tile data because tile files alone do not identify or restore the +tree that produced them. Applications with externally persisted tree state can +use the lower-level `TileStore` and `TileWriter` APIs to resume a store. + See the [tiled storage guide](doc/tiles-guide.md) for a how-to covering flushing, compaction, rollback, proofs, and the lower-level building blocks, and [doc/design/tlog-tiles.md](doc/design/tlog-tiles.md) for the full design, diff --git a/doc/design/tlog-tiles.md b/doc/design/tlog-tiles.md index 1200f14..493146f 100644 --- a/doc/design/tlog-tiles.md +++ b/doc/design/tlog-tiles.md @@ -583,6 +583,13 @@ invariants: > **Immutability invariant.** Never roll back below a full-tile boundary that a > flush may have published: `size >= immutable_size`. +`TiledTree` is fresh-only: its configured directory may exist, but the `tile` +subdirectory must be absent or empty. Tile files do not carry the size, root, +hash identity, or ownership information needed to reopen a tree safely, so the +wrapper rejects an existing tile namespace rather than adopting it. The +lower-level `TileWriter` supports resume for applications that persist and +validate the matching tree state themselves. + Per flush: 1. `append(...)` new leaf hashes; compute `root()`. diff --git a/doc/tiles-guide.md b/doc/tiles-guide.md index f0e5ed6..43d13ce 100644 --- a/doc/tiles-guide.md +++ b/doc/tiles-guide.md @@ -90,6 +90,15 @@ auto consistency = log.consistency_proof(/*m=*/100, /*n=*/n); `TiledTree` can be move-constructed, but it cannot be copied or assigned. Move construction keeps its writer bound to the destination tree's tile store. +`TiledTree` always creates a new tiled tree. The configured directory may +already exist, but its `tile` subdirectory must be absent or empty. Construction +throws rather than adopting existing tiles because those files do not identify +the tree that produced them or contain enough state to restore its size and +root. If your application persists and validates that state separately, use +the lower-level `TileStore` and `TileWriter` APIs; `TileWriter` intentionally +resumes existing full tiles and therefore trusts the caller to supply the same +tree and hash function. + `flush()` is incremental: each call writes only the full tiles that became complete since the previous call. Full tiles are immutable: written once after all 256 entries are final and never rewritten. The remaining frontier stays in diff --git a/merklecpp_tiles.h b/merklecpp_tiles.h index 2d35ddf..8a653da 100644 --- a/merklecpp_tiles.h +++ b/merklecpp_tiles.h @@ -611,6 +611,9 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) /// completes the next tile. /// @warning No internal synchronization is provided. Callers must serialize /// access to a writer and its store. + /// @warning A writer trusts existing full tiles as output from the same + /// tree and hash function. Callers resuming a store must establish that + /// ownership and restore the matching tree state. template < size_t HASH_SIZE, void HASH_FUNCTION( @@ -1295,6 +1298,11 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) /// Config::compact_on_flush, or call compact() explicitly; it never drops /// the un-tiled frontier. Proofs are served from the combination of the /// resident tree (frontier) and the full tiles (compacted past). + /// @note TiledTree creates a new tiled tree and cannot reopen one from tile + /// files alone. Construction rejects a non-empty tile namespace because + /// the files do not identify their tree or record enough state to restore + /// it. Use TileWriter directly only when the caller owns and restores that + /// state. /// @warning No internal synchronization is provided. Callers must serialize /// all access to a shared tree, including proof operations. template < @@ -1314,7 +1322,9 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) /// @brief Configuration for a tiled tree. struct Config { - /// @brief Root directory for the tiles. + /// @brief Root directory for a new tiled tree. + /// @note The directory itself may exist, but its tile subdirectory must + /// be absent or empty. std::filesystem::path prefix; /// @brief Number of most-recent leaves to keep resident when @@ -1329,7 +1339,9 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) explicit TiledTreeT(Config config) : config(std::move(config)), store(this->config.prefix), writer(store) - {} + { + require_empty_tile_namespace(); + } TiledTreeT(const TiledTreeT&) = delete; TiledTreeT& operator=(const TiledTreeT&) = delete; @@ -1513,6 +1525,45 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) uint64_t tiles_size = 0; uint64_t sealed_size = 0; + void require_empty_tile_namespace() const + { + const auto tile_root = store.root() / "tile"; + std::error_code ec; + const bool exists = std::filesystem::exists(tile_root, ec); + if (ec) + { + throw std::runtime_error( + "TiledTree: cannot inspect tile namespace " + tile_root.string() + + ": " + ec.message()); + } + if (!exists) + { + return; + } + + const bool is_directory = std::filesystem::is_directory(tile_root, ec); + if (ec) + { + throw std::runtime_error( + "TiledTree: cannot inspect tile namespace " + tile_root.string() + + ": " + ec.message()); + } + const bool is_empty = + is_directory && std::filesystem::is_empty(tile_root, ec); + if (ec) + { + throw std::runtime_error( + "TiledTree: cannot inspect tile namespace " + tile_root.string() + + ": " + ec.message()); + } + if (!is_empty) + { + throw std::runtime_error( + "TiledTree: tile namespace is not empty; reopening an existing " + "tiled tree is not supported"); + } + } + /// @brief Builds a proof engine over the combined resident-tree /// (frontier) and full-tile (flushed past) source, and invokes @p fn with /// it. diff --git a/test/tiles_tree.cpp b/test/tiles_tree.cpp index 3dbcb1c..05d91ce 100644 --- a/test/tiles_tree.cpp +++ b/test/tiles_tree.cpp @@ -200,6 +200,39 @@ int main() std::cout << "move construction: OK" << '\n'; } + // ---- Part 0c: a TiledTree is fresh-only. It accepts an existing root + // directory, but rejects a tile namespace that may belong to another + // tree rather than silently adopting its immutable files. + { + TiledTree::Config existing_cfg; + existing_cfg.prefix = base / "tt_existing"; + fs::create_directories(existing_cfg.prefix / "tile"); + { + TiledTree first(existing_cfg); + for (uint64_t i = 0; i < 256; i++) + { + first.append(hashes[i]); + } + expect( + first.flush().full_written == 1, + "existing prefix: first tree writes tile"); + } + + bool construction_threw = false; + try + { + const TiledTree second(existing_cfg); + } + catch (const std::exception&) + { + construction_threw = true; + } + expect( + construction_threw, + "existing prefix: second tree rejects existing tiles"); + std::cout << "existing tile namespace: OK" << '\n'; + } + // ---- Part 1: memory-source proofs (exercises TreeT::subtree_root). for (const uint64_t n : {(uint64_t)1, From 0ea87da5507873ef65b71d7acefe560c0ff1c799 Mon Sep 17 00:00:00 2001 From: achamayou Date: Sat, 11 Jul 2026 16:20:22 +0000 Subject: [PATCH 27/30] Preserve exact rollback boundary Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- audit.md | 10 ++++---- doc/design/tlog-tiles.md | 24 +++++++++--------- doc/tiles-guide.md | 9 ++++--- merklecpp_tiles.h | 24 +++++++++--------- test/tiles_tree.cpp | 53 ++++++++++++++++++++++++++++++++++------ 5 files changed, 82 insertions(+), 38 deletions(-) diff --git a/audit.md b/audit.md index fe6f044..d61ed0b 100644 --- a/audit.md +++ b/audit.md @@ -261,11 +261,11 @@ Recorded so the breadth of the review is auditable: already-written child tiles; the `next_full` cursor + `has_full_tile` skip avoid duplicate writes and never skip a tile; atomic temp+rename keeps the on-disk prefix contiguous, so the monotonic `full_prefix_length` binary search is valid. -* **`compact` / `retract_to`** — `target` is clamped to a `TILE_WIDTH` multiple - ≤ `tiles_size`, so the frontier is always retained and `min_index ≤ tiles_size` - (no coverage gap); `target ≥ n` keeps ≥ 1 resident leaf; the `index + 1 < - tiles_size` guard protects the immutable prefix while allowing rollback to - exactly `flushed_size`. +* **`compact` / `retract_to`** - `target` is normally aligned to a `TILE_WIDTH` + multiple and, for a nonzero `tiles_size`, is capped below it, so the frontier + and final tiled leaf remain resident. The `index + 1 < sealed_size` guard + protects the immutable prefix while allowing rollback to exactly + `immutable_size`. * **`TileStoreT` I/O** — `decode_entries`/`read_tile` bounds checks are correct and overflow-free (fuzzed clean under ASan/UBSan); paths are built only from integer indices + the caller's `prefix`, so path traversal is not realistic; diff --git a/doc/design/tlog-tiles.md b/doc/design/tlog-tiles.md index 493146f..3ecc42f 100644 --- a/doc/design/tlog-tiles.md +++ b/doc/design/tlog-tiles.md @@ -413,7 +413,7 @@ uint64_t covered = (size / TILE_WIDTH) * TILE_WIDTH; // full level-0 cover immutable_size = max(immutable_size, covered); // before any write writer.write_up_to(size, [&](uint64_t i) -> const Hash& { return tree.leaf(i); }); flushed_size = covered; // after all levels succeed -// compact() drops only [min_index, covered); the frontier [covered, size) stays. +// compact() retains the last covered leaf (when any) and the entire frontier. ``` If `write_up_to` throws, `immutable_size` remains advanced because a full tile @@ -577,8 +577,8 @@ entry bundles are an **application-owned** add-on, included for completeness: The pairing of tile writing with `flush_to` gives two central correctness invariants: -> **Compaction invariant.** Never compact past the last fully successful flush: -> `min_index() <= flushed_size`. +> **Compaction invariant.** Retain the final leaf of the last fully successful +> flush: `flushed_size == 0 || min_index() < flushed_size`. > > **Immutability invariant.** Never roll back below a full-tile boundary that a > flush may have published: `size >= immutable_size`. @@ -598,11 +598,11 @@ Per flush: 3. `write_up_to(size, leaf_at)` - persist newly-complete **full** tiles at all levels (incremental). 4. After every level succeeds, set `flushed_size = covered`. -5. *(optional)* `compact()` -> `flush_to(covered - retention_margin)` - reclaim - memory, only when `compact_on_flush` is set (or `compact()` is called - explicitly). It drops only the full-tile-covered prefix, so the un-tiled - frontier always stays resident. By default nothing is dropped and the tree - stays whole. +5. *(optional)* `compact()` computes an aligned retention target, capped below + nonzero `covered`, then calls `flush_to(target)`. This reclaims memory only + when `compact_on_flush` is set (or `compact()` is called explicitly), while + retaining the final tiled leaf and the entire un-tiled frontier. By default + nothing is dropped and the tree stays whole. If step 3 fails, steps 4 and 5 do not run. `immutable_size` stays advanced to prevent stale-tile rollback, while `flushed_size` stays at the last complete @@ -735,10 +735,10 @@ compatible": final, and every emitted tile is write-once. A stand-alone tile reader cannot serve the frontier; that is the in-memory tree's job (or the application must keep it elsewhere). -- **`flush_to` alignment.** Compaction flushes only to a 256-multiple (minus - retention) to uphold the coverage invariant; it also keeps at least one - resident leaf, since `flush_to` cannot drain the whole tree. Enforced inside - `TiledTreeT::compact`. +- **`flush_to` alignment.** Compaction normally flushes to a 256-multiple + derived from retention. When that target equals `flushed_size`, it stops one + leaf earlier so `TreeT` can still retract to exactly that size. This one-leaf + overlap is enforced inside `TiledTreeT::compact`. - **Rollback vs. immutable tiles.** Tiles are write-once, so rolling the tree back (`retract_to`) over a range that a flush may have published would leave stale, never-rewritten tiles. `TiledTreeT::retract_to` therefore throws if the diff --git a/doc/tiles-guide.md b/doc/tiles-guide.md index 43d13ce..6753f5a 100644 --- a/doc/tiles-guide.md +++ b/doc/tiles-guide.md @@ -127,9 +127,10 @@ merkle::tiles::TiledTree log(cfg); - `compact_on_flush` (default `false`): when set, `flush()` calls `compact()` for you. - `compact()` can also be called explicitly at any time. It drops from memory - only the leaves already covered by a **durably written full tile**, keeping - `retention_margin` recent leaves resident. It returns the new minimum - (smallest still-resident) leaf index. + only leaves already covered by a **durably written full tile**, keeping at + least `retention_margin` recent leaves resident. It also retains the final + tiled leaf so rollback to exactly `immutable_size()` remains representable. + It returns the new minimum (smallest still-resident) leaf index. - Proofs for dropped leaves are still produced — they are served from the tiles and transparently combined with the resident frontier. @@ -158,6 +159,8 @@ log.retract_to(index); // keep leaves [0, index], drop the rest - Allowed when the resulting size is `>= immutable_size()`. - Throws otherwise, because a flush may already have published an immutable full tile for that range. +- The exact `immutable_size()` boundary remains available after compaction; + compaction retains the final tiled leaf needed by the in-memory tree. - After a successful flush, `immutable_size() == flushed_size()`. After an interrupted flush, `immutable_size()` may be larger until the same tree state is flushed successfully. diff --git a/merklecpp_tiles.h b/merklecpp_tiles.h index 8a653da..6836175 100644 --- a/merklecpp_tiles.h +++ b/merklecpp_tiles.h @@ -1329,6 +1329,8 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) /// @brief Number of most-recent leaves to keep resident when /// compacting (i.e. never dropped from memory). + /// @note Compaction may retain one additional tiled boundary leaf so + /// rollback to exactly immutable_size() remains possible. uint64_t retention_margin = 0; /// @brief If set, flush() compacts after writing tiles, dropping @@ -1441,28 +1443,28 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) return stats; } - /// @brief Drops from the in-memory tree every leaf already covered by a - /// durably-written full tile, keeping retention_margin recent leaves. + /// @brief Drops old leaves covered by durably-written full tiles, keeping + /// retention_margin recent leaves and one tiled boundary leaf. /// @return The new minimum (smallest still-resident) leaf index /// @note Only leaves covered by a full tile are dropped, so the un-tiled /// frontier is always retained in memory and inclusion/consistency proofs - /// remain available (the past from tiles, the frontier from memory). Has - /// no effect until tiling has produced full tiles. + /// remain available (the past from tiles, the frontier from memory). The + /// leaf at flushed_size() - 1 also remains resident so retract_to() can + /// represent a tree whose size is exactly immutable_size(). Has no effect + /// until tiling has produced full tiles. uint64_t compact() { - const uint64_t n = tree.num_leaves(); const uint64_t covered = (tiles_size / TILE_WIDTH) * TILE_WIDTH; uint64_t target = covered > config.retention_margin ? covered - config.retention_margin : 0; target = (target / TILE_WIDTH) * TILE_WIDTH; - // merklecpp's flush_to keeps at least one resident leaf, so never flush - // the whole tree (this can only arise when the size is an exact - // multiple of TILE_WIDTH and nothing is retained); the extra resident - // leaf is harmless (it is also covered by a full tile). - if (n > 0 && target >= n) + // TreeT cannot retract below min_index(). Keep the final tiled leaf + // resident so rollback to a size of exactly immutable_size() remains + // representable after compaction. + if (covered > 0 && target == covered) { - target = n - 1; + target--; } if (target > tree.min_index()) { diff --git a/test/tiles_tree.cpp b/test/tiles_tree.cpp index 05d91ce..8676081 100644 --- a/test/tiles_tree.cpp +++ b/test/tiles_tree.cpp @@ -303,7 +303,8 @@ int main() dtt.flush(); expect(dtt.tree_ref().min_index() == 0, "default: nothing dropped"); dtt.compact(); - expect(dtt.tree_ref().min_index() == 768, "compact() drops to 768"); + expect( + dtt.tree_ref().min_index() == 767, "compact() keeps boundary leaf"); } TiledTree::Config cfg; @@ -318,7 +319,7 @@ int main() } tt.flush(); // flushes full tiles; flushed_size = covered = 768 expect(tt.flushed_size() == 768, "flushed size"); - expect(tt.tree_ref().min_index() == 768, "compacted to 768"); + expect(tt.tree_ref().min_index() == 767, "compacted with boundary overlap"); for (uint64_t i = n1; i < N; i++) { @@ -387,7 +388,7 @@ int main() // work (this also confirms the writer never reads a flushed leaf). tt.flush(); // flushes full tiles; flushed_size = covered = 1280 expect(tt.flushed_size() == 1280, "second flushed size"); - expect(tt.tree_ref().min_index() == 1280, "flushed to 1280"); + expect(tt.tree_ref().min_index() == 1279, "second boundary leaf retained"); for (const uint64_t i : {(uint64_t)0, @@ -598,7 +599,43 @@ int main() expect(rb.size() == 256, "rb retract to tiled prefix allowed"); } - // 3c. Rollback interacts correctly with compaction (flushed + tiled + // 3c. Compaction keeps the final tiled leaf resident, so rollback to a + // tree size of exactly immutable_size() remains representable. + { + TiledTree::Config cfg; + cfg.prefix = base / "rb_exact_boundary"; + cfg.compact_on_flush = true; + TiledTree rb(cfg); + for (uint64_t i = 0; i < 1000; i++) + { + rb.append(hashes[i]); + } + rb.flush(); + expect(rb.flushed_size() == 768, "rb boundary flushed size"); + expect(rb.immutable_size() == 768, "rb boundary immutable size"); + expect( + rb.tree_ref().min_index() == 767, + "rb boundary final tiled leaf retained"); + + rb.retract_to(767); + expect(rb.size() == 768, "rb exact immutable boundary allowed"); + + merkle::Tree expected; + for (uint64_t i = 0; i < 768; i++) + { + expected.insert(hashes[i]); + } + const Hash expected_root = expected.root(); + expect( + rb.root() == expected_root, "rb boundary root matches reference"); + const auto proof = rb.inclusion_proof(0, 768); + expect( + *proof == *expected.path(0), + "rb boundary inclusion matches reference"); + expect(proof->verify(expected_root), "rb boundary inclusion verifies"); + } + + // 3d. Rollback interacts correctly with compaction (flushed + tiled // past). { TiledTree::Config cfg; @@ -609,8 +646,10 @@ int main() { rb.append(hashes[i]); } - rb.flush(); // flushed_size = covered = 768; compact to 768 - expect(rb.tree_ref().min_index() == 768, "rb compact flushed to 768"); + rb.flush(); // flushed_size = covered = 768; retain boundary leaf 767 + expect( + rb.tree_ref().min_index() == 767, + "rb compact boundary leaf retained"); for (uint64_t i = 1000; i < 1200; i++) { rb.append(hashes[i]); @@ -641,7 +680,7 @@ int main() expect(threw, "rb compact retract below tiled throws"); } - // 3d. A failed flush may publish an immutable full tile before a later + // 3e. A failed flush may publish an immutable full tile before a later // write fails. The attempted full-tile boundary is sealed against // rollback, while flushed_size advances only after a successful retry. { From 9bfdb90e7331ff7d0987389a4f7ae4637117c795 Mon Sep 17 00:00:00 2001 From: achamayou Date: Sat, 11 Jul 2026 16:29:32 +0000 Subject: [PATCH 28/30] Harden tile store filesystem writes Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- audit.md | 31 +++++++------- doc/design/tlog-tiles.md | 8 ++-- doc/tiles-guide.md | 7 ++-- merklecpp_tiles.h | 90 ++++++++++++++++++++++++++++++++++------ test/tiles_store.cpp | 51 +++++++++++++++++++++++ 5 files changed, 153 insertions(+), 34 deletions(-) diff --git a/audit.md b/audit.md index d61ed0b..b72672c 100644 --- a/audit.md +++ b/audit.md @@ -32,10 +32,10 @@ and (c) several **robustness / hostile-input / test / doc** gaps. > tree (`tiles_tree` Part 0). > > **Update (hardening commit).** The remaining implementation findings are now -> addressed: **C2/D2** by synced unique-temp writes, parent-directory sync on -> POSIX, and size-validated tile presence; **C3/C4** by overflow-safe public -> arithmetic guards; **C5** by unique temp names and cleanup; and **P1/D1** by a -> per-source tile read cache plus corrected docs. +> addressed: **C2/D2** by synced unique-temp writes, durable creation of every +> directory ancestor on POSIX, and size-validated tile presence; **C3/C4** by +> overflow-safe public arithmetic guards; **C5** by unique temp names and +> cleanup; and **P1/D1** by a per-source tile read cache plus corrected docs. --- @@ -44,7 +44,7 @@ and (c) several **robustness / hostile-input / test / doc** gaps. | # | Severity | Area | Summary | |---|----------|------|---------| | C1 | Medium | Correctness (config) | **Fixed:** the `write_higher_levels` option was removed, so required roll-up tiles are always written | -| C2 | Medium | Durability | **Fixed:** `write_file_atomically` syncs unique temp files, atomically replaces, syncs POSIX parent dirs, and bad-size tiles are rewritten | +| C2 | Medium | Durability | **Fixed:** `write_file_atomically` syncs unique temp files, atomically replaces, durably creates POSIX directory ancestors, and rewrites bad-size tiles | | C3 | Low | Robustness (public API) | **Fixed:** `TreeT::subtree_root` validates `level`/`index` before shifting and uses overflow-safe range checks | | C4 | Low | Robustness (hostile input) | **Fixed:** `largest_pow2_lt` uses an overflow-safe loop condition for `size > 2^63` | | C5 | Low | Concurrency / cleanup | **Fixed:** temp file names are unique per process/time/counter and cleaned up on error | @@ -108,17 +108,19 @@ remove/guard the option or document the incompatibility. ### C2 — `write_file_atomically` is atomic but not durable; the store can wedge after a crash (Medium) > **Status:** Fixed by the hardening commit. `write_file_atomically` now writes a -> unique temp file, syncs it, publishes it with atomic replace, syncs the POSIX -> parent directory, and removes the temp file on error. `has_full_tile` also -> requires the exact full-tile byte size, so a truncated/oversized tile is -> rewritten instead of treated as durable. +> unique temp file, syncs it, publishes it with atomic replace, syncs every +> newly created directory link and the destination directory on POSIX, and +> removes the temp file on error. `has_full_tile` also requires the exact +> full-tile byte size, so a truncated/oversized tile is rewritten instead of +> treated as durable. The original implementation used stream flush plus rename and treated file existence as the idempotence signal. A crash could therefore leave a present but -wrong-size tile that future writes skipped and future reads rejected. The current -implementation syncs the temp file before publishing, syncs POSIX parent -directories after publishing, and makes `has_full_tile` require the exact full -tile size so wrong-size tiles are rewritten. +wrong-size tile that future writes skipped and future reads rejected. The +current implementation syncs the temp file before publishing, durably creates +each POSIX directory ancestor, syncs the destination directory after +publishing, and makes `has_full_tile` require the exact full tile size so +wrong-size tiles are rewritten. ### C3 — `TreeT::subtree_root` does not validate its arguments (Low) @@ -235,7 +237,8 @@ when that option was removed. * **D1 (Medium)** — Fixed: `doc/design/tlog-tiles.md` now describes the implemented per-source tile cache. * **D2 (Low)** — Fixed: `doc/design/tlog-tiles.md` and the code comment now - describe synced unique-temp writes, atomic replace, and bad-size tile rewrite. + describe synced unique-temp writes, durable directory creation, atomic + replace, and bad-size tile rewrite. --- diff --git a/doc/design/tlog-tiles.md b/doc/design/tlog-tiles.md index 3ecc42f..c6d7fd7 100644 --- a/doc/design/tlog-tiles.md +++ b/doc/design/tlog-tiles.md @@ -356,10 +356,10 @@ public: ``` - Writes use a unique temporary file, sync the file contents, publish with an - atomic replace, and on POSIX sync the parent directory after the rename. A - crash should leave either the old tile or the new complete tile visible; a - wrong-size tile is not treated as a durable full tile and is rewritten by the - writer. + atomic replace, and on POSIX sync every newly created directory link plus the + destination directory after the rename. A crash should leave either the old + tile or the new complete tile visible; a wrong-size tile is not treated as a + durable full tile and is rewritten by the writer. - Full tiles are written once and never rewritten (immutability), so every file under `tile//` is write-once. - A small in-process cache of recently read tiles avoids repeated I/O during diff --git a/doc/tiles-guide.md b/doc/tiles-guide.md index 6753f5a..7327522 100644 --- a/doc/tiles-guide.md +++ b/doc/tiles-guide.md @@ -105,9 +105,10 @@ all 256 entries are final and never rewritten. The remaining frontier stays in memory until it crosses the next full-tile boundary. Tile files are written through unique temporary files, synced, then published -with an atomic replace. On POSIX systems the parent directory is also synced -after the rename. A wrong-size file at a tile path is not a published tile, so -the writer does not treat mere file existence as success. +with an atomic replace. On POSIX systems, each newly created directory is made +durable by syncing its parent, and the destination directory is synced after +the rename. A wrong-size file at a tile path is not a published tile, so the +writer does not treat mere file existence as success. ## Flushing and compaction diff --git a/merklecpp_tiles.h b/merklecpp_tiles.h index 6836175..2b9f8de 100644 --- a/merklecpp_tiles.h +++ b/merklecpp_tiles.h @@ -315,18 +315,12 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) /// @brief Writes a file atomically via a synced temporary file. /// @note Uses unique temp names, cleans them up on errors, and syncs the /// file before publishing it with an atomic replace. POSIX builds also - /// sync the parent directory after rename. + /// sync each newly created directory's parent and the destination + /// directory after rename. static void write_file_atomically( const std::filesystem::path& path, const std::vector& bytes) { - std::error_code ec; - std::filesystem::create_directories(path.parent_path(), ec); - if (ec) - { - throw std::runtime_error( - "cannot create directory " + path.parent_path().string() + ": " + - ec.message()); - } + create_directories_durably(path.parent_path()); const std::filesystem::path tmp = temp_path(path); TempFileGuard guard(tmp); @@ -336,6 +330,69 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) guard.dismiss(); } + template + static void create_directories_durably( + const std::filesystem::path& directory, const SyncParent& sync_parent) + { + if (directory.empty()) + { + return; + } + + std::error_code ec; + const bool exists = std::filesystem::exists(directory, ec); + if (ec) + { + throw std::runtime_error( + "cannot inspect directory " + directory.string() + ": " + + ec.message()); + } + if (exists) + { + const bool is_directory = + std::filesystem::is_directory(directory, ec); + if (ec) + { + throw std::runtime_error( + "cannot inspect directory " + directory.string() + ": " + + ec.message()); + } + if (!is_directory) + { + throw std::runtime_error( + "cannot create directory " + directory.string() + + ": path exists and is not a directory"); + } + return; + } + + const auto parent = directory.parent_path(); + if (parent != directory) + { + create_directories_durably(parent, sync_parent); + } + + const bool created = std::filesystem::create_directory(directory, ec); + if (ec) + { + throw std::runtime_error( + "cannot create directory " + directory.string() + ": " + + ec.message()); + } + if (created) + { + sync_parent(parent.empty() ? std::filesystem::path(".") : parent); + } + } + + static void create_directories_durably( + const std::filesystem::path& directory) + { + create_directories_durably( + directory, + [](const std::filesystem::path& parent) { sync_directory(parent); }); + } + class TempFileGuard { public: @@ -392,6 +449,15 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) #endif } + static void require_write_progress( + size_t written, const std::filesystem::path& path) + { + if (written == 0) + { + throw std::runtime_error("short write: " + path.string()); + } + } + static void write_and_sync_file( const std::filesystem::path& path, const std::vector& bytes) { @@ -424,6 +490,7 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) throw std::runtime_error( system_error_message("error writing file " + path.string())); } + require_write_progress((size_t)done, path); written += done; } if (!FlushFileBuffers(handle)) @@ -473,10 +540,7 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) throw std::runtime_error( system_error_message("error writing file " + path.string())); } - if (done == 0) - { - throw std::runtime_error("short write: " + path.string()); - } + require_write_progress((size_t)done, path); written += (size_t)done; } if (::fsync(fd) != 0) diff --git a/test/tiles_store.cpp b/test/tiles_store.cpp index da6a9f2..0589570 100644 --- a/test/tiles_store.cpp +++ b/test/tiles_store.cpp @@ -59,6 +59,26 @@ static bool any_tmp_files(const fs::path& dir) }); } +class TileStoreProbe : public merkle::tiles::TileStore +{ +public: + using Store = merkle::tiles::TileStore; + + static std::vector create_and_record_directory_syncs( + const fs::path& directory) + { + std::vector synced; + Store::create_directories_durably( + directory, [&](const fs::path& parent) { synced.push_back(parent); }); + return synced; + } + + static void check_write_progress(size_t written) + { + Store::require_write_progress(written, "test"); + } +}; + // Overwrites a file with exactly the given bytes (to simulate corruption). static void overwrite_file(const fs::path& p, const std::vector& bytes) { @@ -105,6 +125,37 @@ int main() expect_eq( rel(store, store.entries_path(5)), "tile/entries/005", "entries full"); + // 2b. Every newly-created directory link is followed by a sync of its + // parent, from the first missing ancestor through the final directory. + { + const fs::path nested = dir / "durable" / "tile" / "1"; + const auto synced = + TileStoreProbe::create_and_record_directory_syncs(nested); + const std::vector expected = { + dir.parent_path(), dir, dir / "durable", dir / "durable" / "tile"}; + expect(synced == expected, "new directory parents synced in order"); + expect(fs::is_directory(nested), "nested directories created"); + expect( + TileStoreProbe::create_and_record_directory_syncs(nested).empty(), + "existing directories need no creation sync"); + } + + // 2c. A successful write must make progress. This shared guard prevents + // the Windows WriteFile loop from spinning if it reports zero bytes. + { + bool zero_threw = false; + try + { + TileStoreProbe::check_write_progress(0); + } + catch (const std::exception&) + { + zero_threw = true; + } + expect(zero_threw, "zero-byte write rejected"); + TileStoreProbe::check_write_progress(1); + } + const size_t hsz = Hash().size(); // 3a. Full tile byte round-trip. From 9fad334c38ffc28705e70b28db823ad781523b4b Mon Sep 17 00:00:00 2001 From: achamayou Date: Sat, 11 Jul 2026 19:25:11 +0000 Subject: [PATCH 29/30] Fix tiled storage recovery and durability Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/workflows/build-docs.yml | 5 +- audit.md | 115 ++++++++++---- doc/design/tlog-tiles.md | 37 +++-- doc/tiles-guide.md | 31 +++- merklecpp_tiles.h | 248 ++++++++++++++++++++----------- test/CMakeLists.txt | 9 ++ test/tiles_entries.cpp | 61 ++++++++ test/tiles_hashes.cpp | 103 +++++++++++++ test/tiles_store.cpp | 177 +++++++++++++++++++--- test/tiles_writer.cpp | 72 +++++++++ 10 files changed, 700 insertions(+), 158 deletions(-) create mode 100644 test/tiles_hashes.cpp diff --git a/.github/workflows/build-docs.yml b/.github/workflows/build-docs.yml index e3d617c..d6e7c4f 100644 --- a/.github/workflows/build-docs.yml +++ b/.github/workflows/build-docs.yml @@ -4,10 +4,11 @@ on: push: branches: - main + pull_request: workflow_dispatch: concurrency: - group: pages + group: pages-${{ github.ref }} cancel-in-progress: false permissions: read-all @@ -24,6 +25,7 @@ jobs: - name: Setup Pages id: pages uses: actions/configure-pages@45bfe0192ca1faeb007ade9deae92b16b8254a0d # v6.0.0 + if: github.event_name != 'pull_request' - name: Install dependencies run: | @@ -42,6 +44,7 @@ jobs: uses: actions/upload-pages-artifact@fc324d3547104276b827a68afc52ff2a11cc49c9 # v5.0.0 with: path: doc/build + if: github.event_name != 'pull_request' deploy: name: Deploy diff --git a/audit.md b/audit.md index b72672c..8e8ff1d 100644 --- a/audit.md +++ b/audit.md @@ -1,9 +1,9 @@ -# Audit: tiled storage & tile-backed proofs (`experiment/tiles`) +# Audit: tiled storage & tile-backed proofs (PR #48) -**Scope.** All changes on `experiment/tiles` relative to `main` (15 commits, -+3,900 / −2 lines): the new header `merklecpp_tiles.h`, the `TreeT::subtree_root` -addition in `merklecpp.h`, the `test/tiles_*.cpp` / `test/time_tiles.cpp` suite, -and `doc/design/tlog-tiles.md` + `doc/tiles-guide.md`. +**Scope.** All PR #48 changes relative to `main`: the new header +`merklecpp_tiles.h`, the `TreeT::subtree_root` addition in `merklecpp.h`, the +`test/tiles_*.cpp` / `test/time_tiles.cpp` suite, and +`doc/design/tlog-tiles.md` + `doc/tiles-guide.md`. **Method.** Four independent review passes: a manual read plus three specialist reviewers. Two reviewers built and ran the tile suite under GCC **and** Clang @@ -11,14 +11,12 @@ with **ASan + UBSan**, and one cross-checked **113,462** generated proofs agains the `merkle::TreeT` oracle across every tile boundary. Where a finding is marked *reproduced*, it was triggered in a running build, not merely read. -**Headline.** The core proof math is **sound**: inclusion proofs are byte-identical -to `TreeT::path()/past_path()`, consistency proofs round-trip through -`verify_consistency`, and both were checked exhaustively for small trees and by a -large oracle sweep. The shift in `subtree_root` that looks like 64-bit UB is in -fact **safe**. No correctness defect exists on the **default** code path. The -issues below are (a) a real **durability** gap whose docs overstate the guarantee, -(b) a **non-default config** (`write_higher_levels = false`) that crashes proofs, -and (c) several **robustness / hostile-input / test / doc** gaps. +**Headline.** The core proof math is **sound**: inclusion proofs are +byte-identical to `TreeT::path()/past_path()`, consistency proofs round-trip +through `verify_consistency`, and both were checked exhaustively for small trees +and by a large oracle sweep. Follow-up review found additional durability retry +and cursor-recovery defects outside the proof math; these are recorded below +and fixed. > **Update (follow-up commit).** Acting on this audit: the `write_higher_levels` > option has been **removed** entirely — higher-level roll-up tiles are now @@ -36,6 +34,14 @@ and (c) several **robustness / hostile-input / test / doc** gaps. > directory ancestor on POSIX, and size-validated tile presence; **C3/C4** by > overflow-safe public arithmetic guards; **C5** by unique temp names and > cleanup; and **P1/D1** by a per-source tile read cache plus corrected docs. +> +> **Update (fleet-review follow-up).** **C6** now re-confirms directory entries +> and destination directories before reusing visible files, so failed syncs are +> retried. **C7** replaces non-monotonic binary/exponential cursor discovery +> with a bounded contiguous scan for tiles and entry bundles. **C8** documents +> the concrete `tree_ref()` and `store_ref()` mutation hazards. Production-path +> fault injection, interior-corruption, sparse-index, SHA-384/SHA-512, timeout, +> and pull-request documentation checks cover the fixes. --- @@ -48,6 +54,9 @@ and (c) several **robustness / hostile-input / test / doc** gaps. | C3 | Low | Robustness (public API) | **Fixed:** `TreeT::subtree_root` validates `level`/`index` before shifting and uses overflow-safe range checks | | C4 | Low | Robustness (hostile input) | **Fixed:** `largest_pow2_lt` uses an overflow-safe loop condition for `size > 2^63` | | C5 | Low | Concurrency / cleanup | **Fixed:** temp file names are unique per process/time/counter and cleaned up on error | +| C6 | High | Durability retry | **Fixed:** visible files and directory chains are re-synced before writer reuse | +| C7 | Medium | Cursor recovery | **Fixed:** recovery scans the bounded contiguous prefix and repairs interior holes | +| C8 | Low | Public escape hatches | **Fixed:** `tree_ref()` and `store_ref()` warn about bookkeeping and proof corruption | | P1 | Low–Med | Performance | **Fixed:** `TileHashSource` keeps a small decoded tile cache for proof generation | | T1 | Medium | Tests | **Fixed:** corrupt/truncated tile and entry-bundle read tests added | | T2 | Medium | Tests | **Fixed:** level-2 coverage added via `tiles_level2` and large proof sweeps | @@ -55,8 +64,11 @@ and (c) several **robustness / hostile-input / test / doc** gaps. | T4 | Low–Med | Tests | **Fixed:** reopen/resume coverage added for `full_prefix_length` cursor recovery | | T5 | Low | Tests | **Fixed:** non-zero `retention_margin` compaction and guarded rollback are covered | | T6 | Low | Tests | **Fixed:** empty tree and `65537` boundary coverage added | +| T7 | Medium | Tests | **Fixed:** production-path directory-sync failures, interior corruption, and sparse indices are covered | +| T8 | Low | Tests | **Fixed:** SHA-384/SHA-512 tiled runtime coverage and CTest timeouts were added | | D1 | Medium | Docs | **Fixed:** design doc now matches the per-source tile cache | | D2 | Low | Docs | **Fixed:** durability wording now describes synced temp-file atomic replace and bad-size tile rewrite | +| D3 | Low | Docs | **Fixed:** stale API/cursor text was corrected and documentation now builds on pull requests | --- @@ -107,20 +119,21 @@ remove/guard the option or document the incompatibility. ### C2 — `write_file_atomically` is atomic but not durable; the store can wedge after a crash (Medium) -> **Status:** Fixed by the hardening commit. `write_file_atomically` now writes a -> unique temp file, syncs it, publishes it with atomic replace, syncs every -> newly created directory link and the destination directory on POSIX, and -> removes the temp file on error. `has_full_tile` also requires the exact -> full-tile byte size, so a truncated/oversized tile is rewritten instead of -> treated as durable. +> **Status:** Fixed by the hardening and fleet-review follow-up commits. +> `write_file_atomically` writes a unique synced temp file, publishes it with +> atomic replace, syncs every directory link and the destination directory on +> POSIX, and removes the temp file on error. Before reuse, an existing file's +> directory chain and destination directory are re-confirmed. Exact file +> validation means a malformed tile or bundle is rewritten. The original implementation used stream flush plus rename and treated file existence as the idempotence signal. A crash could therefore leave a present but wrong-size tile that future writes skipped and future reads rejected. The current implementation syncs the temp file before publishing, durably creates each POSIX directory ancestor, syncs the destination directory after -publishing, and makes `has_full_tile` require the exact full tile size so -wrong-size tiles are rewritten. +publishing, and re-confirms those syncs before an existing file is reused. +`has_full_tile` requires the exact full tile size, so wrong-size tiles are +rewritten. ### C3 — `TreeT::subtree_root` does not validate its arguments (Low) @@ -201,6 +214,43 @@ place (corruption-safe only because tile content is deterministic). On a write o **Fix.** Unique temp suffix (pid + counter/random) and `remove` it on the error path. +### C6 - failed directory syncs can be skipped on retry (High) + +> **Status:** Fixed. Directory-entry and directory-content sync state is only +> cached after a successful sync. A writer re-confirms this state before +> trusting any visible existing tile or entry bundle, including through a fresh +> `TileStore` instance. + +The earlier write-once check used valid file presence as its retry signal. If a +rename succeeded and the following directory sync failed, a retry could skip +the visible file and advance `flushed_size()` even though a crash could still +lose the directory entry. Similarly, a directory created before its parent +sync failed remained visible and was not synced again. + +Production-path fault injection now covers both interruption points and verifies +that retry repeats the failed sync before a tile is reused or published. + +### C7 - prefix recovery assumes filesystem presence is monotonic (Medium) + +> **Status:** Fixed. Fresh writers linearly inspect the contiguous prefix only +> up to the full-file count relevant to the requested size. + +Binary search cannot locate the first missing or malformed file when a later +file is valid. Exponential search can also overflow when crafted sparse files +exist at powers of two. The bounded ordered scan repairs the first hole, checks +later files individually in the write loop, and never examines irrelevant high +indices. Tiles and entry bundles have matching regressions. + +### C8 - mutable escape-hatch consequences were understated (Low) + +> **Status:** Fixed. API comments and the guide now describe the concrete +> invariants callers assume responsibility for. + +Direct retraction through `tree_ref()` can make `flushed_size()` and +`immutable_size()` exceed `size()` and can make `flushed_size()` regress. +Correctly sized but unrelated files written through `store_ref()` are trusted +by a later flush and can invalidate proofs after compaction. + --- ## Performance @@ -224,11 +274,13 @@ proofs against `TreeT::path/past_path/past_root`, round-trips consistency proofs through `verify_consistency`, exercises tamper/wrong-root rejection, and is **exhaustive** over `(m,k)` consistency pairs for `n ≤ 16`. The gaps: -The historical gaps listed in the first audit pass are now covered: corrupt and -wrong-size tile/bundle files, level-2 tiles, reopen/resume, non-zero retention -margin, guarded rollback below the flushed prefix, empty trees, and `65537` / -large-boundary proof cases. The `write_higher_levels=false` test became obsolete -when that option was removed. +The historical and follow-up gaps are now covered: corrupt and wrong-size +tile/bundle files, interior holes, sparse high indices, production-path +directory-sync failures, level-2 tiles, SHA-384/SHA-512 tiled trees, +reopen/resume, non-zero retention margin, guarded rollback below the flushed +prefix, empty trees, and `65537` / large-boundary proof cases. Long tests have +CTest timeouts. The `write_higher_levels=false` test became obsolete when that +option was removed. --- @@ -239,6 +291,9 @@ when that option was removed. * **D2 (Low)** — Fixed: `doc/design/tlog-tiles.md` and the code comment now describe synced unique-temp writes, durable directory creation, atomic replace, and bad-size tile rewrite. +* **D3 (Low)** - Fixed: the design uses the implemented `tree_ref()` / + `store_ref()` and lower-level `TileWriterT` APIs, describes bounded + in-memory cursor recovery and cache lifetime, and is built for pull requests. --- @@ -260,10 +315,10 @@ Recorded so the breadth of the review is auditable: `m`); `verify_consistency` matches the standard CT verifier (`m==n`, `m==0`, `m>n`, `is_pow2(m)` prepend, the `fn/sn` reduction, final `sn==0`). Round-trips for all `m/` is write-once. - A small in-process cache of recently read tiles avoids repeated I/O during @@ -402,8 +404,10 @@ entry(L, g): - Level 0 reads leaf hashes via the supplied `leaf_at` (e.g. `Tree::leaf(i)`), which exist **before** flushing. Higher levels roll up from the level-0 tiles just written — independent of tree internals and robust to prior flushes. -- `first_unwritten_full(L)` is tracked in the store (probe `has_full_tile`, or a - small persisted cursor) to keep each flush O(new tiles). +- `first_unwritten_full(L)` is cached in each writer. A fresh writer reconstructs + it with a bounded, ordered scan of the contiguous prefix up to `full_L`. + Scanning in order is required because malformed or externally created files + can make raw file presence non-monotonic. Compaction integration (the "on compaction" story): @@ -444,6 +448,9 @@ struct HashSource { // concept (duck-typed or virtual) - `TileHashSource` mutates its LRU cache during `const` reads. It and every `ProofEngine` that refers to it require caller-provided synchronization when shared between threads. +- `TiledTreeT` constructs these sources for each proof call, so its cache lasts + for one call. A caller that wants cross-call caching can retain a lower-level + `TileHashSource`. ### 6.6 `ProofEngineT` — inclusion & consistency @@ -464,6 +471,8 @@ public: // RFC 6962 consistency proof that size `m` is a prefix of size `n` (m<=n). std::vector consistency_proof(uint64_t m, uint64_t n) const; + std::vector consistency_proof_from_indices( + uint64_t first_index, uint64_t second_index) const; // Verifier (consistency is new to merklecpp; inclusion reuses PathT::verify). static bool verify_consistency(uint64_t m, uint64_t n, @@ -545,17 +554,19 @@ public: // Proofs over tiles ∪ resident tree (works for flushed indices). std::shared_ptr inclusion_proof(uint64_t index, uint64_t size); std::vector consistency_proof(uint64_t m, uint64_t n); + std::vector consistency_proof_from_indices(uint64_t i, uint64_t j); - Tree& tree(); // escape hatch + Tree& tree_ref(); // mutable escape hatch + Store& store_ref(); // mutable escape hatch }; ``` `TiledTreeT` performs no internal locking. The caller must serialize every operation on a shared instance, including proof calls. -Stateless alternative for callers with their own storage: free functions -`write_tiles(tree, store, size, opts)` and a `ProofEngineT` built on a -`CombinedHashSource`, so the wrapper is optional sugar. +Callers with their own storage can construct a `TileWriterT` and call +`write_up_to`, then build a `ProofEngineT` on a `CombinedHashSource`, so the +wrapper is optional sugar. ### 6.8 Entry bundles (optional) @@ -745,12 +756,16 @@ compatible": resulting size is below `immutable_size()`. A failed flush may advance `immutable_size()` without advancing `flushed_size()`; retry with the same tree state. Retracting the underlying tree directly via `tree_ref()` bypasses - this guard and must be avoided. + this guard, can make the size boundaries inconsistent or non-monotonic, and + must be avoided. Files written through `store_ref()` are trusted without + checking that they match the tree and can invalidate proofs after compaction. - **No internal synchronization.** Every tiled-storage object and shared store prefix requires external serialization. This includes `const` proof reads, which update the tile cache. - **Very large indices.** Index math uses `uint64_t`; encoding handles - multi-group indices. Level bound `≤ 63` per spec (8 suffices for `2^64`). + multi-group indices. Level bound `<= 63` per spec (8 suffices for `2^64`). + Resume scans are bounded by the requested tree size and cannot follow sparse + files beyond that range. - **Open question — `subtree_root` in core vs. `past_path`-derived memory source.** Recommend the tiny non-hashing accessor; falls back to zero-core- change if maintainers prefer. Either keeps hashing untouched. diff --git a/doc/tiles-guide.md b/doc/tiles-guide.md index 7327522..396dbaa 100644 --- a/doc/tiles-guide.md +++ b/doc/tiles-guide.md @@ -97,7 +97,9 @@ the tree that produced them or contain enough state to restore its size and root. If your application persists and validates that state separately, use the lower-level `TileStore` and `TileWriter` APIs; `TileWriter` intentionally resumes existing full tiles and therefore trusts the caller to supply the same -tree and hash function. +tree and hash function. A fresh writer scans the requested range in order, +stopping at the first missing or malformed file, so an interior hole is +rewritten rather than hidden by later files. `flush()` is incremental: each call writes only the full tiles that became complete since the previous call. Full tiles are immutable: written once after @@ -107,8 +109,10 @@ memory until it crosses the next full-tile boundary. Tile files are written through unique temporary files, synced, then published with an atomic replace. On POSIX systems, each newly created directory is made durable by syncing its parent, and the destination directory is synced after -the rename. A wrong-size file at a tile path is not a published tile, so the -writer does not treat mere file existence as success. +the rename. Before reusing a visible file, a writer also re-confirms its +directory chain and destination directory. This makes a retry repeat a failed +directory sync even when the rename or directory creation is already visible. +A wrong-size file at a tile path is not a published tile and is rewritten. ## Flushing and compaction @@ -168,9 +172,15 @@ log.retract_to(index); // keep leaves [0, index], drop the rest - `retract_to` mirrors `merkle::Tree::retract_to`: `index` is the new *last* leaf, so the resulting size is `index + 1`. -> ⚠️ Retracting the underlying tree directly via `log.tree_ref().retract_to(...)` -> bypasses this guard and can leave the tiles inconsistent with the tree. Use -> `TiledTree::retract_to` instead. +> **Warning:** Treat `tree_ref()` as an inspection escape hatch unless you also +> maintain every tiled-tree invariant yourself. Direct retraction bypasses the +> guard, can make `flushed_size()` and `immutable_size()` exceed `size()`, and +> can make `flushed_size()` regress. Use `TiledTree::retract_to` instead. + +`store_ref()` is similarly unsafe for mutation. A later flush trusts any +correctly sized tile written through it without checking that the hashes match +the in-memory tree. A mismatched tile can silently invalidate proofs after +compaction. ## Proofs @@ -243,6 +253,11 @@ auto stats = writer.write_up_to( // stats.full_written ``` +`TileWriter` keeps an in-memory next-file cursor. A new writer reconstructs it +by checking the contiguous prefix only up to the number of full files relevant +to the requested tree size. Existing files are re-confirmed as durably +published before reuse; malformed files and holes are rewritten. + ### Reading tiles and computing proofs A `HashSource` resolves the root of a complete subtree; pick where it reads from: @@ -268,7 +283,9 @@ tiles). For the live frontier, combine it with a `MemoryHashSource` — which is exactly what `TiledTree` does for you. `TiledTree` simply wires a `CombinedHashSource(MemoryHashSource, TileHashSource)` -into a `ProofEngine` for you. +into a `ProofEngine` for you. It creates these sources for each proof call, so +its tile cache is per-call. A long-lived lower-level `TileHashSource` retains +its cache across calls. ## Entry bundles (optional) diff --git a/merklecpp_tiles.h b/merklecpp_tiles.h index 2b9f8de..1e3ce90 100644 --- a/merklecpp_tiles.h +++ b/merklecpp_tiles.h @@ -14,8 +14,10 @@ #include #include #include +#include #include #include +#include #include #include #include @@ -52,6 +54,20 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) /// @brief Number of hashes in a full tile (2**TILE_HEIGHT). static constexpr uint16_t TILE_WIDTH = 256; + namespace detail + { + template + uint64_t contiguous_prefix_length(uint64_t limit, const Present& present) + { + uint64_t length = 0; + while (length < limit && present(length)) + { + length++; + } + return length; + } + } + /// @brief Encodes a tile index as tlog-tiles path elements. /// @param n The tile index /// @return The index as zero-padded, '/'-separated 3-digit groups, where @@ -97,6 +113,18 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) uint64_t index = 0; }; + template < + size_t HASH_SIZE, + void HASH_FUNCTION( + const HashT&, const HashT&, HashT&)> + class TileWriterT; + + template < + size_t HASH_SIZE, + void HASH_FUNCTION( + const HashT&, const HashT&, HashT&)> + class EntryBundleWriterT; + /// @brief Reads and writes tlog-tiles tile files on a local filesystem. /// @tparam HASH_SIZE Size of each hash in bytes /// @tparam HASH_FUNCTION The tree's node hash function (carried for use by @@ -109,6 +137,9 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) const HashT&, const HashT&, HashT&)> class TileStoreT { + friend class TileWriterT; + friend class EntryBundleWriterT; + public: /// @brief The type of hashes stored in tiles. using Hash = HashT; @@ -297,9 +328,46 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) } protected: + using DirectorySync = std::function; + + TileStoreT(std::filesystem::path prefix, DirectorySync directory_sync) : + prefix(std::move(prefix)), directory_sync(std::move(directory_sync)) + {} + /// @brief The root directory of the store. std::filesystem::path prefix; + DirectorySync directory_sync; + std::set durable_directory_entries; + std::set durable_directory_contents; + + [[nodiscard]] bool confirm_full_tile(uint8_t level, uint64_t index) + { + const auto path = tile_path(TileRef{level, index}); + if (!has_full_tile(level, index)) + { + return false; + } + confirm_file_durable(path); + return true; + } + + [[nodiscard]] bool confirm_entry_bundle(uint64_t index) + { + const auto path = entries_path(index); + if (!has_entry_bundle(index)) + { + return false; + } + confirm_file_durable(path); + return true; + } + + void begin_write_attempt() + { + durable_directory_contents.clear(); + } + /// @brief Reads an entire file into a byte vector. static std::vector read_file(const std::filesystem::path& path) { @@ -315,9 +383,10 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) /// @brief Writes a file atomically via a synced temporary file. /// @note Uses unique temp names, cleans them up on errors, and syncs the /// file before publishing it with an atomic replace. POSIX builds also - /// sync each newly created directory's parent and the destination - /// directory after rename. - static void write_file_atomically( + /// confirm each directory entry in the path and sync the destination + /// directory after rename. Existing files are only reused after these + /// syncs are re-confirmed by the current write attempt. + void write_file_atomically( const std::filesystem::path& path, const std::vector& bytes) { create_directories_durably(path.parent_path()); @@ -325,20 +394,34 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) const std::filesystem::path tmp = temp_path(path); TempFileGuard guard(tmp); write_and_sync_file(tmp, bytes); + const auto parent = directory_or_dot(path.parent_path()); + durable_directory_contents.erase(parent); replace_file(tmp, path); - sync_directory(path.parent_path()); + sync_directory_contents(parent); guard.dismiss(); } - template - static void create_directories_durably( - const std::filesystem::path& directory, const SyncParent& sync_parent) + static std::filesystem::path directory_or_dot( + const std::filesystem::path& directory) + { + return directory.empty() ? std::filesystem::path(".") : directory; + } + + void create_directories_durably( + const std::filesystem::path& requested_directory) { - if (directory.empty()) + const auto directory = directory_or_dot(requested_directory); + if (durable_directory_entries.count(directory) != 0) { return; } + auto parent = directory_or_dot(directory.parent_path()); + if (parent != directory) + { + create_directories_durably(parent); + } + std::error_code ec; const bool exists = std::filesystem::exists(directory, ec); if (ec) @@ -363,34 +446,48 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) "cannot create directory " + directory.string() + ": path exists and is not a directory"); } - return; } - - const auto parent = directory.parent_path(); - if (parent != directory) + else { - create_directories_durably(parent, sync_parent); + const bool created = std::filesystem::create_directory(directory, ec); + if (ec) + { + throw std::runtime_error( + "cannot create directory " + directory.string() + ": " + + ec.message()); + } + if (!created && !std::filesystem::is_directory(directory, ec)) + { + throw std::runtime_error( + "cannot create directory " + directory.string() + + (ec ? ": " + ec.message() : "")); + } } - const bool created = std::filesystem::create_directory(directory, ec); - if (ec) - { - throw std::runtime_error( - "cannot create directory " + directory.string() + ": " + - ec.message()); - } - if (created) + if (parent != directory) { - sync_parent(parent.empty() ? std::filesystem::path(".") : parent); + sync_directory_for_durability(parent); + durable_directory_contents.insert(parent); } + durable_directory_entries.insert(directory); } - static void create_directories_durably( - const std::filesystem::path& directory) + void confirm_file_durable(const std::filesystem::path& path) + { + const auto parent = directory_or_dot(path.parent_path()); + create_directories_durably(parent); + sync_directory_contents(parent); + } + + void sync_directory_contents(const std::filesystem::path& directory) { - create_directories_durably( - directory, - [](const std::filesystem::path& parent) { sync_directory(parent); }); + const auto path = directory_or_dot(directory); + if (durable_directory_contents.count(path) != 0) + { + return; + } + sync_directory_for_durability(path); + durable_directory_contents.insert(path); } class TempFileGuard @@ -592,7 +689,17 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) #endif } - static void sync_directory(const std::filesystem::path& path) + void sync_directory_for_durability(const std::filesystem::path& path) + { + if (directory_sync) + { + directory_sync(path); + return; + } + sync_directory_on_disk(path); + } + + static void sync_directory_on_disk(const std::filesystem::path& path) { #ifndef _WIN32 int flags = O_RDONLY; @@ -711,13 +818,15 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) /// [0, size); only ever queried for leaves of complete subtrees. /// @return Counts of tiles written /// @note Incremental: full tiles already on disk are immutable and are - /// never rewritten. Entries that do not complete a tile are not written. + /// never rewritten once validated and confirmed durable. Malformed files + /// are replaced. Entries that do not complete a tile are not written. /// Tiles are always rolled up at every level (0..63), so the on-disk set /// always contains the higher-level roll-ups that proof generation relies /// on. Stats write_up_to(uint64_t size, const LeafFn& leaf_at) { Stats stats; + store.begin_write_attempt(); // tlog-tiles defines levels 0..63; the loop stops early once a level // has no complete entries (see the entries == 0 break below). @@ -736,13 +845,13 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) if (cursor_inited[level] == 0) { - next_full[level] = full_prefix_length(level); + next_full[level] = full_prefix_length(level, full_tiles); cursor_inited[level] = 1; } for (uint64_t n = next_full[level]; n < full_tiles; n++) { - if (store.has_full_tile(level, n)) + if (store.confirm_full_tile(level, n)) { continue; // immutable: never rewrite an existing full tile } @@ -790,33 +899,12 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) } } - /// @brief Length of the contiguous prefix of full tiles already on disk. - [[nodiscard]] uint64_t full_prefix_length(uint8_t level) const + /// @brief Length of the confirmed contiguous prefix, bounded by @p limit. + [[nodiscard]] uint64_t full_prefix_length(uint8_t level, uint64_t limit) { - if (!store.has_full_tile(level, 0)) - { - return 0; - } - uint64_t lo = 0; // present - uint64_t hi = 1; - while (store.has_full_tile(level, hi)) - { - lo = hi; - hi <<= 1; - } - while (hi - lo > 1) - { - const uint64_t mid = lo + (hi - lo) / 2; - if (store.has_full_tile(level, mid)) - { - lo = mid; - } - else - { - hi = mid; - } - } - return lo + 1; + return detail::contiguous_prefix_length(limit, [&](uint64_t index) { + return store.confirm_full_tile(level, index); + }); } /// @brief Collects @p count consecutive level-@p level entries, each the @@ -1461,12 +1549,19 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) } /// @brief Access to the underlying tree. + /// @warning Mutating the tree directly bypasses tiled-tree bookkeeping. + /// In particular, direct retraction can make flushed_size() and + /// immutable_size() exceed size() and can make flushed_size() regress. + /// Use TiledTreeT operations whenever they are available. Tree& tree_ref() { return tree; } /// @brief Access to the underlying tile store. + /// @warning Files written or changed through this reference are trusted + /// by later flushes without checking that their hashes match this tree. + /// Mismatched files can silently invalidate proofs after compaction. Store& store_ref() { return store; @@ -1542,8 +1637,6 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) /// @note Only full tiles are immutable: this throws if the resulting size /// would be smaller than immutable_size(). A failed flush may advance /// immutable_size() without advancing flushed_size(). - /// (Retracting the underlying tree directly via tree_ref() bypasses this - /// guard and can leave stale tiles -- do not do that.) void retract_to(size_t index) { if ((uint64_t)index + 1 < sealed_size) @@ -1686,21 +1779,23 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) /// [0, size); only ever queried for entries of complete bundles. /// @return Counts of bundles written /// @note Incremental: full bundles already on disk are immutable and are - /// never rewritten. The incomplete tail is never bundled. + /// never rewritten once validated and confirmed durable. Malformed files + /// are replaced. The incomplete tail is never bundled. Stats write_up_to(uint64_t size, const EntryFn& entry_at) { Stats stats; + store.begin_write_attempt(); const uint64_t full = size / TILE_WIDTH; if (!cursor_inited) { - next_full = full_prefix_length(); + next_full = full_prefix_length(full); cursor_inited = true; } for (uint64_t n = next_full; n < full; n++) { - if (store.has_entry_bundle(n)) + if (store.confirm_entry_bundle(n)) { continue; // immutable: never rewrite an existing full bundle } @@ -1733,32 +1828,11 @@ namespace merkle // NOLINT(modernize-concat-nested-namespaces) return out; } - [[nodiscard]] uint64_t full_prefix_length() const + [[nodiscard]] uint64_t full_prefix_length(uint64_t limit) { - if (!store.has_entry_bundle(0)) - { - return 0; - } - uint64_t lo = 0; - uint64_t hi = 1; - while (store.has_entry_bundle(hi)) - { - lo = hi; - hi <<= 1; - } - while (hi - lo > 1) - { - const uint64_t mid = lo + (hi - lo) / 2; - if (store.has_entry_bundle(mid)) - { - lo = mid; - } - else - { - hi = mid; - } - } - return lo + 1; + return detail::contiguous_prefix_length(limit, [&](uint64_t index) { + return store.confirm_entry_bundle(index); + }); } }; diff --git a/test/CMakeLists.txt b/test/CMakeLists.txt index 35773fb..2a0572b 100644 --- a/test/CMakeLists.txt +++ b/test/CMakeLists.txt @@ -35,9 +35,18 @@ add_merklecpp_test(tiles_proofs tiles_proofs.cpp) add_merklecpp_test(tiles_tree tiles_tree.cpp) add_merklecpp_test(tiles_entries tiles_entries.cpp) +if(OPENSSL) + add_merklecpp_test(tiles_hashes tiles_hashes.cpp) +endif() + if(LONG_TESTS) add_merklecpp_test(tiles_level2 tiles_level2.cpp) add_merklecpp_test(time_tiles time_tiles.cpp) + set_tests_properties( + ${MERKLECPP_TEST_PREFIX}tiles_level2 + ${MERKLECPP_TEST_PREFIX}time_tiles + PROPERTIES TIMEOUT 900 + ) endif() find_path(DOCTEST_DIR doctest.h) diff --git a/test/tiles_entries.cpp b/test/tiles_entries.cpp index 8312f84..e48ae37 100644 --- a/test/tiles_entries.cpp +++ b/test/tiles_entries.cpp @@ -140,6 +140,67 @@ int main() expect(!store.has_entry_bundle(2), "inc no tail bundle"); } + // 5. Resume repairs an interior malformed bundle even when later files are + // valid. + { + const fs::path dir = base / "eholes"; + { + TileStore store(dir); + EntryBundleWriter writer(store); + expect( + writer.write_up_to(2048, entry_at).full_written == 8, + "holes initial bundles"); + overwrite_file(store.entries_path(3), {0x00, 0x05, 0x01}); + expect(!store.has_entry_bundle(3), "holes interior bundle malformed"); + expect(store.has_entry_bundle(7), "holes later bundle remains valid"); + } + + TileStore store(dir); + EntryBundleWriter writer(store); + expect( + writer.write_up_to(2048, entry_at).full_written == 1, + "holes rewrites interior bundle"); + std::vector> expected; + expected.reserve(merkle::tiles::TILE_WIDTH); + constexpr uint64_t hole_begin = (uint64_t)merkle::tiles::TILE_WIDTH * 3; + constexpr uint64_t hole_end = (uint64_t)merkle::tiles::TILE_WIDTH * 4; + for (uint64_t i = hole_begin; i < hole_end; i++) + { + expected.push_back(entry_at(i)); + } + expect( + store.read_entry_bundle(3) == expected, + "holes repaired bundle contents"); + } + + // 6. Recovery examines only bundle indices relevant to the requested size. + { + const fs::path dir = base / "esparse"; + const std::vector> bundle( + merkle::tiles::TILE_WIDTH, {0x42}); + { + TileStore store(dir); + store.write_entry_bundle(0, bundle); + for (uint64_t index = 1;; index <<= 1) + { + store.write_entry_bundle(index, bundle); + if (index == (uint64_t{1} << 63)) + { + break; + } + } + } + + TileStore store(dir); + EntryBundleWriter writer(store); + expect( + writer.write_up_to(merkle::tiles::TILE_WIDTH, entry_at).full_written == + 0, + "sparse bounded recovery"); + expect( + store.has_entry_bundle(0), "sparse requested bundle remains valid"); + } + std::cout << "tiles_entries: OK" << '\n'; std::error_code ec; diff --git a/test/tiles_hashes.cpp b/test/tiles_hashes.cpp new file mode 100644 index 0000000..4ac86a5 --- /dev/null +++ b/test/tiles_hashes.cpp @@ -0,0 +1,103 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. + +#include "util.h" + +#include +#include +#include +#include +#include +#include +#include +#include + +namespace fs = std::filesystem; + +static void expect(bool cond, const std::string& what) +{ + if (!cond) + { + throw std::runtime_error("check failed: " + what); + } +} + +template < + size_t HASH_SIZE, + typename Tree, + typename TiledTree, + typename ProofEngine> +static void exercise_tiled_hash(const fs::path& prefix, const std::string& name) +{ + const auto hashes = make_hashesT(300); + Tree reference; + typename TiledTree::Config config; + config.prefix = prefix; + config.compact_on_flush = true; + TiledTree tree(config); + + for (const auto& hash : hashes) + { + reference.insert(hash); + tree.append(hash); + } + + expect(tree.flush().full_written == 1, name + " full tile written"); + expect(tree.flushed_size() == 256, name + " flushed size"); + expect(tree.root() == reference.root(), name + " root"); + expect( + fs::file_size(prefix / "tile" / "0" / "000") == + (uintmax_t)merkle::tiles::TILE_WIDTH * HASH_SIZE, + name + " tile byte size"); + + const auto inclusion = tree.inclusion_proof(0, tree.size()); + expect(inclusion->verify(reference.root()), name + " inclusion proof"); + + const auto consistency = tree.consistency_proof(256, tree.size()); + expect( + ProofEngine::verify_consistency( + 256, + tree.size(), + *reference.past_root(255), + reference.root(), + consistency), + name + " consistency proof"); +} + +int main() +{ + const auto seed = std::time(nullptr); + std::srand((unsigned)seed); + std::cout << "seed=" << seed << '\n'; + + const fs::path base = fs::temp_directory_path() / + ("merklecpp_tiles_hashes_" + std::to_string((unsigned long long)seed) + + "_" + std::to_string(std::rand())); + + try + { + exercise_tiled_hash< + 48, + merkle::Tree384, + merkle::tiles::TiledTree384, + merkle::tiles::ProofEngine384>(base / "sha384", "SHA384"); + exercise_tiled_hash< + 64, + merkle::Tree512, + merkle::tiles::TiledTree512, + merkle::tiles::ProofEngine512>(base / "sha512", "SHA512"); + + std::cout << "tiles_hashes: OK" << '\n'; + std::error_code ec; + fs::remove_all(base, ec); + } + catch (const std::exception& ex) + { + std::cout << "Error: " << ex.what() << '\n'; + std::error_code ec; + fs::remove_all(base, ec); + return 1; + } + + return 0; +} diff --git a/test/tiles_store.cpp b/test/tiles_store.cpp index 0589570..6d45d01 100644 --- a/test/tiles_store.cpp +++ b/test/tiles_store.cpp @@ -12,6 +12,7 @@ #include #include #include +#include #include #include #include @@ -64,21 +65,54 @@ class TileStoreProbe : public merkle::tiles::TileStore public: using Store = merkle::tiles::TileStore; - static std::vector create_and_record_directory_syncs( - const fs::path& directory) + static void check_write_progress(size_t written) { - std::vector synced; - Store::create_directories_durably( - directory, [&](const fs::path& parent) { synced.push_back(parent); }); - return synced; + Store::require_write_progress(written, "test"); } +}; - static void check_write_progress(size_t written) +struct SyncFault +{ + fs::path fail_path; + size_t failures_remaining = 0; + size_t matching_calls_before_failure = 0; + std::vector calls; + + void sync(const fs::path& path) { - Store::require_write_progress(written, "test"); + const auto normalised = path.lexically_normal(); + calls.push_back(normalised); + if (failures_remaining > 0 && normalised == fail_path.lexically_normal()) + { + if (matching_calls_before_failure > 0) + { + matching_calls_before_failure--; + return; + } + failures_remaining--; + throw std::runtime_error("injected directory sync failure"); + } + } + + [[nodiscard]] size_t call_count(const fs::path& path) const + { + const auto normalised = path.lexically_normal(); + return (size_t)std::count(calls.begin(), calls.end(), normalised); } }; +class FaultInjectingTileStore : public merkle::tiles::TileStore +{ +public: + using Store = merkle::tiles::TileStore; + + FaultInjectingTileStore( + fs::path prefix, const std::shared_ptr& fault) : + Store( + std::move(prefix), [fault](const fs::path& path) { fault->sync(path); }) + {} +}; + // Overwrites a file with exactly the given bytes (to simulate corruption). static void overwrite_file(const fs::path& p, const std::vector& bytes) { @@ -112,6 +146,8 @@ int main() "encode 1234067"); merkle::tiles::TileStore store(dir); + const size_t hsz = Hash().size(); + const auto full = make_hashes(merkle::tiles::TILE_WIDTH); // 2. Resource path layout (full tiles and bundles only). expect_eq( @@ -125,19 +161,22 @@ int main() expect_eq( rel(store, store.entries_path(5)), "tile/entries/005", "entries full"); - // 2b. Every newly-created directory link is followed by a sync of its - // parent, from the first missing ancestor through the final directory. + // 2b. Production writes sync each directory link and the destination + // directory in order. { - const fs::path nested = dir / "durable" / "tile" / "1"; - const auto synced = - TileStoreProbe::create_and_record_directory_syncs(nested); + const fs::path prefix = dir / "durable"; + const auto fault = std::make_shared(); + FaultInjectingTileStore durable_store(prefix, fault); + durable_store.write_tile(TileRef{1, 0}, full); const std::vector expected = { - dir.parent_path(), dir, dir / "durable", dir / "durable" / "tile"}; - expect(synced == expected, "new directory parents synced in order"); - expect(fs::is_directory(nested), "nested directories created"); + dir.parent_path(), dir, prefix, prefix / "tile", prefix / "tile" / "1"}; expect( - TileStoreProbe::create_and_record_directory_syncs(nested).empty(), - "existing directories need no creation sync"); + fault->calls.size() >= expected.size() && + std::equal( + expected.begin(), + expected.end(), + fault->calls.end() - (std::ptrdiff_t)expected.size()), + "directory parents and destination synced in order"); } // 2c. A successful write must make progress. This shared guard prevents @@ -156,10 +195,102 @@ int main() TileStoreProbe::check_write_progress(1); } - const size_t hsz = Hash().size(); + // 2d. A failed destination-directory sync after rename is retried before + // another already-open writer trusts the visible tile. + { + const fs::path prefix = dir / "retry_destination"; + const fs::path destination = prefix / "tile" / "0"; + const auto fault = std::make_shared(); + fault->fail_path = destination; + constexpr uint64_t growing_size = (uint64_t)merkle::tiles::TILE_WIDTH * 2; + const auto growing = make_hashes((size_t)growing_size); + const auto leaf_at = [&](uint64_t i) -> const Hash& { + return growing[i]; + }; + + FaultInjectingTileStore retry_store(prefix, fault); + merkle::tiles::TileWriter retry_writer(retry_store); + expect( + retry_writer.write_up_to(merkle::tiles::TILE_WIDTH, leaf_at) + .full_written == 1, + "first writer publishes initial tile"); + + fault->matching_calls_before_failure = 1; + fault->failures_remaining = 1; + { + FaultInjectingTileStore failed_store(prefix, fault); + merkle::tiles::TileWriter writer(failed_store); + bool sync_threw = false; + try + { + (void)writer.write_up_to(growing_size, leaf_at); + } + catch (const std::exception&) + { + sync_threw = true; + } + expect(sync_threw, "destination sync failure propagated"); + expect( + failed_store.has_full_tile(0, 1), + "renamed tile remains visible after sync failure"); + } + + expect( + retry_writer.write_up_to(growing_size, leaf_at).full_written == 0, + "retry certifies visible tile without rewriting"); + expect( + fault->call_count(destination) == 4, + "destination directory sync retried"); + const std::vector second( + growing.begin() + (std::ptrdiff_t)merkle::tiles::TILE_WIDTH, + growing.end()); + expect( + retry_store.read_tile(TileRef{0, 1}) == second, + "retried tile remains intact"); + } + + // 2e. A failed ancestor-parent sync is retried even though the directory + // created before the failure remains visible. + { + const fs::path prefix = dir / "retry_ancestor"; + const auto fault = std::make_shared(); + fault->fail_path = prefix; + fault->failures_remaining = 1; + + { + FaultInjectingTileStore failed_store(prefix, fault); + merkle::tiles::TileWriter writer(failed_store); + const auto leaf_at = [&](uint64_t i) -> const Hash& { return full[i]; }; + bool sync_threw = false; + try + { + (void)writer.write_up_to(merkle::tiles::TILE_WIDTH, leaf_at); + } + catch (const std::exception&) + { + sync_threw = true; + } + expect(sync_threw, "ancestor sync failure propagated"); + expect( + fs::is_directory(prefix / "tile"), + "directory remains visible after parent sync failure"); + expect( + !failed_store.has_full_tile(0, 0), + "tile not published before ancestor sync succeeds"); + } + + FaultInjectingTileStore retry_store(prefix, fault); + merkle::tiles::TileWriter retry_writer(retry_store); + const auto leaf_at = [&](uint64_t i) -> const Hash& { return full[i]; }; + expect( + retry_writer.write_up_to(merkle::tiles::TILE_WIDTH, leaf_at) + .full_written == 1, + "retry writes tile after certifying existing ancestor"); + expect(fault->call_count(prefix) == 2, "ancestor parent sync retried"); + expect(retry_store.has_full_tile(0, 0), "retry publishes full tile"); + } // 3a. Full tile byte round-trip. - const auto full = make_hashes(merkle::tiles::TILE_WIDTH); const TileRef full_ref{0, 0}; store.write_tile(full_ref, full); expect(store.has_full_tile(0, 0), "has_full_tile after write"); @@ -266,7 +397,8 @@ int main() // when the final tile remains valid, so keep this stress check POSIX-only. #ifndef _WIN32 // 3d. Concurrent same-tile writes use unique temp files and leave no - // temporary files behind after success. + // temporary files behind after success. Each thread owns its store object; + // sharing one object requires caller-provided synchronization. { const TileRef concurrent_ref{0, 42}; std::atomic ok{true}; @@ -276,7 +408,8 @@ int main() threads.emplace_back([&] { try { - store.write_tile(concurrent_ref, full); + merkle::tiles::TileStore thread_store(dir); + thread_store.write_tile(concurrent_ref, full); } catch (...) { diff --git a/test/tiles_writer.cpp b/test/tiles_writer.cpp index 3f4ff49..05e847f 100644 --- a/test/tiles_writer.cpp +++ b/test/tiles_writer.cpp @@ -8,6 +8,7 @@ #include #include #include +#include #include #include #include @@ -49,6 +50,14 @@ static Hash rollup(const std::vector& leaves) return merkle::tiles::perfect_root<32, merkle::sha256_compress>(leaves); } +static void overwrite_file( + const fs::path& path, const std::vector& bytes) +{ + std::ofstream file(path, std::ios::binary | std::ios::trunc); + file.write( + reinterpret_cast(bytes.data()), (std::streamsize)bytes.size()); +} + int main() { const auto seed = std::time(nullptr); @@ -241,6 +250,69 @@ int main() std::cout << "E (writer resume): OK" << '\n'; } + // ---- F. Resume discovers the first hole rather than trusting later valid + // files as proof that the prefix is contiguous. + { + const auto hashes = make_hashes(2048); + const auto leaf_at = [&](uint64_t i) -> const Hash& { return hashes[i]; }; + const fs::path dir = base / "f"; + + { + TileStore store(dir); + TileWriter writer(store); + expect( + writer.write_up_to(2048, leaf_at).full_written == 8, + "F initial tiles"); + overwrite_file(store.tile_path(TileRef{0, 3}), {0}); + expect(!store.has_full_tile(0, 3), "F interior tile corrupt"); + expect(store.has_full_tile(0, 7), "F later tile remains valid"); + } + + TileStore store(dir); + TileWriter writer(store); + expect( + writer.write_up_to(2048, leaf_at).full_written == 1, + "F rewrites interior hole"); + const std::vector expected( + hashes.begin() + (std::ptrdiff_t)merkle::tiles::TILE_WIDTH * 3, + hashes.begin() + (std::ptrdiff_t)merkle::tiles::TILE_WIDTH * 4); + expect( + store.read_tile(TileRef{0, 3}) == expected, "F repaired tile contents"); + expect(tile_file_count(store) == 8, "F exact tile file count"); + + std::cout << "F (interior recovery): OK" << '\n'; + } + + // ---- G. Recovery is bounded by the requested tree size, so sparse files + // at geometrically increasing indices cannot overflow its search. + { + const fs::path dir = base / "g"; + const auto hashes = make_hashes(merkle::tiles::TILE_WIDTH); + { + TileStore store(dir); + store.write_tile(TileRef{0, 0}, hashes); + for (uint64_t index = 1;; index <<= 1) + { + store.write_tile(TileRef{0, index}, hashes); + if (index == (uint64_t{1} << 63)) + { + break; + } + } + } + + TileStore store(dir); + TileWriter writer(store); + const auto leaf_at = [&](uint64_t i) -> const Hash& { return hashes[i]; }; + expect( + writer.write_up_to(merkle::tiles::TILE_WIDTH, leaf_at).full_written == + 0, + "G bounded sparse recovery"); + expect(store.has_full_tile(0, 0), "G requested tile remains valid"); + + std::cout << "G (bounded sparse recovery): OK" << '\n'; + } + std::cout << "tiles_writer: OK" << '\n'; std::error_code ec; From 10fa37098e75bbf6e46fba9c3656a14ff66fb500 Mon Sep 17 00:00:00 2001 From: achamayou Date: Sun, 12 Jul 2026 11:02:20 +0000 Subject: [PATCH 30/30] Add illustrated tiled tree walkthrough Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/tiles-illustrated.md | 764 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 764 insertions(+) create mode 100644 doc/tiles-illustrated.md diff --git a/doc/tiles-illustrated.md b/doc/tiles-illustrated.md new file mode 100644 index 0000000..30bf65f --- /dev/null +++ b/doc/tiles-illustrated.md @@ -0,0 +1,764 @@ +# Tiled Merkle trees: an illustrated walkthrough + +This page builds a visual model of how an append-only Merkle tree moves from +memory into immutable tile files, and how proofs continue to work across both +places. + +> [!IMPORTANT] +> **Every example on this page uses an imaginary tile width of 32 entries +> purely to keep the diagrams readable.** +> +> The merklecpp implementation remains fixed at `TILE_WIDTH = 256` and +> `TILE_HEIGHT = 8`. A width of 32 is not a configuration option, and this page +> does not propose changing the code, file format, defaults, or examples +> elsewhere. Unless a section explicitly says "illustrative", use 256. + +## The scaled-down model + +A production tile contains 256 entries and spans 8 binary tree levels because +`256 = 2^8`. This page scales that geometry down to 32 entries and 5 levels +because `32 = 2^5`. + +| Property | This page only | Production merklecpp | +|---|---:|---:| +| Tile width | 32 entries | 256 entries | +| Tree levels spanned by one tile | 5 | 8 | +| Leaves covered by one full level-0 tile | 32 | 256 | +| Leaves covered by one level-1 entry | 32 | 256 | +| Leaves covered by one full level-1 tile | 1,024 | 65,536 | + +The scaling changes only the numbers in the drawings. The rules are the same: + +1. Only full tiles are written. +2. A level-0 tile contains leaf hashes. +3. A higher-level tile contains roots of complete tiles from the level below. +4. The incomplete right-hand frontier remains in memory. +5. Published tiles are immutable. +6. Proofs can resolve subtree roots from memory, tiles, or both. + +### Notation + +- `h7` is the hash of leaf 7. +- `R[a, b)` is the Merkle root of the half-open leaf range `[a, b)`. +- `tile/L/NNN` is tile index `NNN` at tile level `L`. +- "Resident" means the in-memory tree can still expand that range to answer + proof requests. +- "Compacted" means the in-memory tree retains enough summary hashes to keep + its root correct, but no longer retains all detail below that range. + +### Colors used below + +```mermaid +flowchart TB + T["Tile-backed hash or range"]:::tile + M["Resident in-memory hash or range"]:::memory + B["Available from both tiles and memory"]:::both + S["Compacted in-memory summary"]:::summary + X["Leaf being proved"]:::target + PT["Hash emitted in a proof
blue outline: from tiles"]:::proofTile + PM["Hash emitted in a proof
green outline: from memory"]:::proofMemory + + T ~~~ M + M ~~~ B + B ~~~ S + S ~~~ X + X ~~~ PT + PT ~~~ PM + + classDef tile fill:#dbeafe,stroke:#2563eb,color:#111827 + classDef memory fill:#dcfce7,stroke:#16a34a,color:#111827 + classDef both fill:#f3e8ff,stroke:#9333ea,color:#111827 + classDef summary fill:#e5e7eb,stroke:#6b7280,color:#111827 + classDef target fill:#fed7aa,stroke:#ea580c,stroke-width:3px,color:#111827 + classDef proofTile fill:#fef3c7,stroke:#2563eb,stroke-width:3px,color:#111827 + classDef proofMemory fill:#fef3c7,stroke:#16a34a,stroke-width:3px,color:#111827 +``` + +## What `flush()` and `compact()` each do + +Appending, flushing, and compacting are separate operations: + +```mermaid +flowchart TB + A["append(h)
Add a leaf hash to the in-memory tree"]:::memory + B["A complete 32-entry range now exists
but no file is written automatically"]:::memory + C["flush()
Write every newly complete full tile"]:::tile + D["The same range exists on disk and in memory
(the default after flush)"]:::both + E["compact()
Optionally discard old resident detail"]:::summary + F["Old detail is served from tiles;
the incomplete frontier stays in memory"]:::both + + A --> B + B --> C + C --> D + D --> E + E --> F + + classDef tile fill:#dbeafe,stroke:#2563eb,color:#111827 + classDef memory fill:#dcfce7,stroke:#16a34a,color:#111827 + classDef both fill:#f3e8ff,stroke:#9333ea,color:#111827 + classDef summary fill:#e5e7eb,stroke:#6b7280,color:#111827 +``` + +`flush()` does not compact by default. Setting `compact_on_flush = true` makes +the final two steps happen in one call, but the durability rule is unchanged: +compaction happens only after all required tile writes succeed. + +If a tile write fails, `immutable_size()` may advance past `flushed_size()` +because a published tile cannot be rolled back. Keep the same tree contents and +retry the flush. See +[Flushing and compaction](tiles-guide.md#flushing-and-compaction) for the full +interrupted-write contract. + +## What is inside a tile file? + +In the illustrative model, `tile/0/000` is the concatenation of 32 leaf hashes: + +```mermaid +flowchart TB + F["tile/0/000
32 serialized hashes"]:::tile + A["entries 0..7
h0 ... h7"]:::tile + B["entries 8..15
h8 ... h15"]:::tile + C["entries 16..23
h16 ... h23"]:::tile + D["entries 24..31
h24 ... h31"]:::tile + R["R[0, 32)
reconstructed by hashing the entries"]:::computed + N["Internal binary-tree nodes are reconstructed;
they are not separately stored in the file"]:::note + + F -->|first bytes| A + A -->|followed by| B + B -->|followed by| C + C -->|followed by| D + D --> R + R --> N + + classDef tile fill:#dbeafe,stroke:#2563eb,color:#111827 + classDef computed fill:#ede9fe,stroke:#7c3aed,color:#111827 + classDef note fill:#f9fafb,stroke:#9ca3af,color:#374151 +``` + +At level 1, each entry is already the root of 32 leaves: + +```mermaid +flowchart TB + L1["tile/1/000
32 serialized subtree roots"]:::tile + L0A["entry 0
root(tile/0/000) = R[0, 32)"]:::tile + L0B["entry 1
root(tile/0/001) = R[32, 64)"]:::tile + L0C["entries 2..30
..."]:::tile + L0Z["entry 31
root(tile/0/031) = R[992, 1024)"]:::tile + ROOT["R[0, 1024)
reconstructed from tile/1/000"]:::computed + + L1 --> L0A + L0A --> L0B + L0B --> L0C + L0C --> L0Z + L0Z --> ROOT + + classDef tile fill:#dbeafe,stroke:#2563eb,color:#111827 + classDef computed fill:#ede9fe,stroke:#7c3aed,color:#111827 +``` + +The production version of the second diagram needs 256 level-0 tile roots, so +its first full level-1 tile appears at 65,536 leaves rather than 1,024. + +## On-disk file layout + +After an illustrative 1,030-leaf tree is flushed, the full-tile boundary is +1,024: + +```text +prefix/ + tile/ + 0/ + 000 # h0 ... h31 + 001 # h32 ... h63 + ... + 031 # h992 ... h1023 + 1/ + 000 # R[0,32), R[32,64), ... R[992,1024) +``` + +Leaves `[1024, 1030)` do not appear in a tile file because they do not complete +another 32-entry tile. They remain in memory. + +```mermaid +flowchart TB + N["n = 1,030 leaves"]:::computed + C["covered = floor(1,030 / 32) * 32 = 1,024"]:::computed + L0["32 full level-0 files
tile/0/000 through tile/0/031"]:::tile + L1["1 full level-1 file
tile/1/000"]:::tile + M["6-leaf frontier
[1024, 1030) in memory"]:::memory + + N --> C + C --> L0 + L0 --> L1 + L1 --> M + + classDef tile fill:#dbeafe,stroke:#2563eb,color:#111827 + classDef memory fill:#dcfce7,stroke:#16a34a,color:#111827 + classDef computed fill:#ede9fe,stroke:#7c3aed,color:#111827 +``` + +The optional `tile/entries/` bundles are omitted here. They store raw +application entries, not Merkle tree nodes, and do not change proof generation. + +## Tree growth, one snapshot at a time + +The next snapshots assume `retention_margin = 0`. Where compaction is shown, +merklecpp still retains the final tiled leaf as a boundary leaf. This is why +the "both" range below is one leaf wide. + +### Snapshot A: 20 leaves + +No full 32-entry tile exists: + +```mermaid +flowchart TB + N["n = 20"]:::computed + C["full-tile boundary = 0"]:::computed + M["Memory only
[0, 20)"]:::memory + D["Disk
no tile files"]:::empty + + N --> C + C --> M + M --> D + + classDef memory fill:#dcfce7,stroke:#16a34a,color:#111827 + classDef computed fill:#ede9fe,stroke:#7c3aed,color:#111827 + classDef empty fill:#f9fafb,stroke:#9ca3af,color:#374151 +``` + +Calling `flush()` at this point writes nothing. Every root and proof is served +from the in-memory tree. + +### Snapshot B: 40 leaves, before the first flush + +The first 32 leaves form a complete tile, but tile creation is explicit: + +```mermaid +flowchart TB + N["n = 40"]:::computed + M0["[0, 32)
complete and eligible, still memory only"]:::memory + M1["[32, 40)
incomplete frontier, memory only"]:::memory + D["Disk
still empty until flush()"]:::empty + + N --> M0 + M0 --> M1 + M1 --> D + + classDef memory fill:#dcfce7,stroke:#16a34a,color:#111827 + classDef computed fill:#ede9fe,stroke:#7c3aed,color:#111827 + classDef empty fill:#f9fafb,stroke:#9ca3af,color:#374151 +``` + +### Snapshot B: 40 leaves, after `flush()` + +The default `flush()` writes the full prefix but does not remove it from +memory: + +```mermaid +flowchart TB + F["flush() succeeds
flushed_size() = 32"]:::computed + B["[0, 32)
tile/0/000 + resident memory"]:::both + M["[32, 40)
resident memory only"]:::memory + + F --> B + B --> M + + classDef memory fill:#dcfce7,stroke:#16a34a,color:#111827 + classDef both fill:#f3e8ff,stroke:#9333ea,color:#111827 + classDef computed fill:#ede9fe,stroke:#7c3aed,color:#111827 +``` + +At this point a proof may be answered entirely from memory even though a tile +copy exists. + +### Snapshot B: 40 leaves, after compaction + +With zero retention, compaction drops old leaf detail while preserving leaf 31 +as the rollback boundary: + +```mermaid +flowchart TB + R["R[0, 40)
current in-memory root"]:::computed + P["R[0, 32)
prefix represented by compacted summaries"]:::summary + C["[0, 31)
not leaf-addressable in memory"]:::summary + B["h31
retained boundary leaf"]:::both + M["R[32, 40)
fully resident frontier"]:::memory + T["tile/0/000
proof detail for [0, 32)"]:::tile + + R --> P + P --> C + P --> B + R --> M + P -.->|subtree and leaf detail| T + + classDef tile fill:#dbeafe,stroke:#2563eb,color:#111827 + classDef memory fill:#dcfce7,stroke:#16a34a,color:#111827 + classDef both fill:#f3e8ff,stroke:#9333ea,color:#111827 + classDef summary fill:#e5e7eb,stroke:#6b7280,color:#111827 + classDef computed fill:#ede9fe,stroke:#7c3aed,color:#111827 +``` + +There are now three logical ownership ranges: + +| Leaf range | Proof detail available from | +|---|---| +| `[0, 31)` | tiles only | +| `[31, 32)` | tiles and memory | +| `[32, 40)` | memory only | + +The compacted in-memory summaries still contribute to `root()`. "Tiles only" +means that a request for a leaf or complete subtree in that range must use the +tile source; it does not mean the in-memory root forgot the prefix hash. + +### Snapshot C: grow from 40 to 72 leaves + +Assume the tree was flushed and compacted at size 40, then 32 more leaves were +appended. + +Before the second flush: + +```mermaid +flowchart TB + N["n = 72
flushed_size() is still 32"]:::computed + T["[0, 31)
tiles only"]:::tile + B["[31, 32)
boundary leaf in both"]:::both + E["[32, 64)
complete and eligible, but still memory only"]:::memory + F["[64, 72)
incomplete memory frontier"]:::memory + + N --> T + T --> B + B --> E + E --> F + + classDef tile fill:#dbeafe,stroke:#2563eb,color:#111827 + classDef memory fill:#dcfce7,stroke:#16a34a,color:#111827 + classDef both fill:#f3e8ff,stroke:#9333ea,color:#111827 + classDef computed fill:#ede9fe,stroke:#7c3aed,color:#111827 +``` + +After the second flush and compaction: + +```mermaid +flowchart TB + F["flush() writes tile/0/001
flushed_size() = 64"]:::computed + T0["tile/0/000 covers [0, 32)"]:::tile + T1["tile/0/001 covers [32, 64)"]:::tile + C["[0, 63)
tiles only after compaction"]:::tile + B["[63, 64)
new boundary leaf in both"]:::both + M["[64, 72)
memory-only frontier"]:::memory + + F --> T0 + T0 --> T1 + T1 --> C + C --> B + B --> M + + classDef tile fill:#dbeafe,stroke:#2563eb,color:#111827 + classDef memory fill:#dcfce7,stroke:#16a34a,color:#111827 + classDef both fill:#f3e8ff,stroke:#9333ea,color:#111827 + classDef computed fill:#ede9fe,stroke:#7c3aed,color:#111827 +``` + +### Snapshot D: 1,030 leaves + +This is the first snapshot with a full illustrative level-1 tile: + +```mermaid +flowchart TB + N["n = 1,030"]:::computed + T0["Level 0
32 files cover [0, 1024)"]:::tile + T1["Level 1
tile/1/000 contains their 32 roots"]:::tile + C["After compaction
[0, 1023) uses tiles for proof detail"]:::tile + B["h1023
boundary leaf in both"]:::both + M["[1024, 1030)
memory-only frontier"]:::memory + + N --> T0 + T0 --> T1 + T1 --> C + C --> B + B --> M + + classDef tile fill:#dbeafe,stroke:#2563eb,color:#111827 + classDef memory fill:#dcfce7,stroke:#16a34a,color:#111827 + classDef both fill:#f3e8ff,stroke:#9333ea,color:#111827 + classDef computed fill:#ede9fe,stroke:#7c3aed,color:#111827 +``` + +### Snapshot summary + +This table assumes each snapshot has just completed a successful flush and +compaction with zero retention: + +| Tree size | `flushed_size()` | Files written | Tiles only | Tiles + memory | Memory only | +|---:|---:|---|---|---|---| +| 20 | 0 | none | none | none | `[0, 20)` | +| 40 | 32 | `tile/0/000` | `[0, 31)` | `[31, 32)` | `[32, 40)` | +| 72 | 64 | `tile/0/000..001` | `[0, 63)` | `[63, 64)` | `[64, 72)` | +| 1,030 | 1,024 | 32 level-0 tiles + `tile/1/000` | `[0, 1023)` | `[1023, 1024)` | `[1024, 1030)` | + +Again, multiply the tile geometry back to 256 for production. In particular, +the production level-1 example starts at 65,536 leaves, not 1,024. + +## How a proof finds a subtree root + +`TiledTree` gives `ProofEngine` a combined source. It tries the resident tree +first because that avoids I/O, then falls back to tiles: + +```mermaid +flowchart TB + Q["ProofEngine requests R[a, b)"]:::computed + L{"Is the range one leaf?"}:::decision + A{"Otherwise, is its width a power of two
and is the range aligned to that width?"}:::decision + P{"Is this complete subtree
fully resident in memory?"}:::decision + M["Return the in-memory hash"]:::memory + T{"Can the tile source resolve it
inside flushed_size()?"}:::decision + D["Read the appropriate tile entries
and roll them up"]:::tile + S["Split the range into smaller subtrees
and resolve each one"]:::computed + E["Fail if no source can resolve a required leaf"]:::error + + Q --> L + L -->|yes| P + L -->|no| A + A -->|yes| P + A -->|no| S + P -->|yes| M + P -->|no| T + T -->|yes| D + T -->|no, and range has multiple leaves| S + T -->|no, and range is one leaf| E + S -->|smaller range| Q + + classDef tile fill:#dbeafe,stroke:#2563eb,color:#111827 + classDef memory fill:#dcfce7,stroke:#16a34a,color:#111827 + classDef computed fill:#ede9fe,stroke:#7c3aed,color:#111827 + classDef decision fill:#fff7ed,stroke:#c2410c,color:#111827 + classDef error fill:#fee2e2,stroke:#dc2626,color:#111827 +``` + +For example, after compacting the 40-leaf tree: + +- `R[0, 32)` is not fully resident, so memory declines it and tiles return it. +- `R[32, 36)` is resident, so memory returns it without touching disk. +- `R[24, 40)` crosses the boundary and is not one complete aligned subtree. + The proof engine splits it into resolvable pieces. + +## Inclusion proof 1: entirely from one tile + +Consider a proof against tree size 32 after `tile/0/000` has been written and +the old leaves have been compacted. This may be the current size or a historical +prefix of a larger tree. We want to prove leaf 5. + +Every required hash is reconstructed from `tile/0/000`: + +```mermaid +flowchart TB + R032["R[0, 32)"]:::tile + R016["R[0, 16)"]:::tile + P1632["R[16, 32)
proof"]:::proofTile + R08["R[0, 8)"]:::tile + P816["R[8, 16)
proof"]:::proofTile + P04["R[0, 4)
proof"]:::proofTile + R48["R[4, 8)"]:::tile + R46["R[4, 6)"]:::tile + P68["R[6, 8)
proof"]:::proofTile + P4["h4
proof"]:::proofTile + X5["h5
target leaf"]:::target + + R032 --> R016 + R032 --> P1632 + R016 --> R08 + R016 --> P816 + R08 --> P04 + R08 --> R48 + R48 --> R46 + R48 --> P68 + R46 --> P4 + R46 --> X5 + + classDef tile fill:#dbeafe,stroke:#2563eb,color:#111827 + classDef target fill:#fed7aa,stroke:#ea580c,stroke-width:3px,color:#111827 + classDef proofTile fill:#fef3c7,stroke:#2563eb,stroke-width:3px,color:#111827 +``` + +The proof payload is ordered from the leaf toward the root: + +| Order | Proof hash | Position relative to the running hash | Source | +|---:|---|---|---| +| 1 | `h4` | left | `tile/0/000` | +| 2 | `R[6, 8)` | right | `tile/0/000` | +| 3 | `R[0, 4)` | left | `tile/0/000` | +| 4 | `R[8, 16)` | right | `tile/0/000` | +| 5 | `R[16, 32)` | right | `tile/0/000` | + +The internal roots in this table are computed on demand from the tile's leaf +hashes. They are not additional files. + +Verification starts with `h5`, combines the five proof hashes in order, and +arrives at `R[0, 32)`. + +## Inclusion proof 2: tiles and memory together + +Return to the compacted 40-leaf tree and prove leaf 36 against the current root +`R[0, 40)`. + +The target and its nearby siblings are in the resident frontier. The old +32-leaf prefix is supplied as one tile-backed subtree root: + +```mermaid +flowchart TB + R040["R[0, 40)"]:::computed + P032["R[0, 32)
proof from tile"]:::proofTile + R3240["R[32, 40)
resident frontier"]:::memory + P3236["R[32, 36)
proof from memory"]:::proofMemory + R3640["R[36, 40)"]:::memory + R3638["R[36, 38)"]:::memory + P3840["R[38, 40)
proof from memory"]:::proofMemory + X36["h36
target leaf"]:::target + P37["h37
proof from memory"]:::proofMemory + + R040 --> P032 + R040 --> R3240 + R3240 --> P3236 + R3240 --> R3640 + R3640 --> R3638 + R3640 --> P3840 + R3638 --> X36 + R3638 --> P37 + + classDef memory fill:#dcfce7,stroke:#16a34a,color:#111827 + classDef computed fill:#ede9fe,stroke:#7c3aed,color:#111827 + classDef target fill:#fed7aa,stroke:#ea580c,stroke-width:3px,color:#111827 + classDef proofTile fill:#fef3c7,stroke:#2563eb,stroke-width:3px,color:#111827 + classDef proofMemory fill:#fef3c7,stroke:#16a34a,stroke-width:3px,color:#111827 +``` + +The mixed proof payload is: + +| Order | Proof hash | Position | Source | +|---:|---|---|---| +| 1 | `h37` | right | memory | +| 2 | `R[38, 40)` | right | memory | +| 3 | `R[32, 36)` | left | memory | +| 4 | `R[0, 32)` | left | `tile/0/000` | + +The caller sees one ordinary `merkle::Path`. Source selection is internal; the +proof format does not mark some hashes as "tile" and others as "memory". + +Proving an old leaf in the current tree is mixed in the opposite direction. +For example, a proof for leaf 5 at size 40 gets its target and lower siblings +from `tile/0/000`, then gets the final sibling `R[32, 40)` from memory. + +## Consistency proofs: the idea + +An inclusion proof answers: + +> Is this leaf part of this tree root? + +A consistency proof answers: + +> Can the tree with `m` leaves be extended, without changing its first `m` +> leaves, to produce the tree with `n` leaves? + +The verifier already knows: + +- `m` and the old root `R[0, m)`; +- `n` and the new root `R[0, n)`. + +The proof supplies enough complete subtree roots to reconstruct both roots +through a shared history. + +The producer recursively follows the part of the new tree that contains the +old boundary and emits the sibling subtree at each split: + +```mermaid +flowchart TB + A["Start with [0, n) and old size m"]:::computed + B["Split at the largest power of two
smaller than the current range"]:::computed + C{"Which side contains
the old boundary?"}:::decision + D["Recurse into that side"]:::computed + E["Emit the other side's root
as a proof hash"]:::proof + F{"Reached exactly
the old boundary?"}:::decision + G["Return proof hashes
from deepest to highest"]:::proof + + A --> B + B --> C + C --> D + D --> E + E --> F + F -->|no| B + F -->|yes| G + + classDef computed fill:#ede9fe,stroke:#7c3aed,color:#111827 + classDef decision fill:#fff7ed,stroke:#c2410c,color:#111827 + classDef proof fill:#fef3c7,stroke:#b45309,color:#111827 +``` + +Each emitted range is resolved through the same memory-first, tile-second +source used by inclusion proofs. + +## Consistency proof 1: a perfect old tree + +First prove that the 32-leaf tree is a prefix of the 40-leaf tree: + +```cpp +auto proof = log.consistency_proof(32, 40); +``` + +Because 32 is a power of two, the old root is already one complete left +subtree. The proof needs only the new right-hand range: + +```mermaid +flowchart TB + OLD["Known old root
R[0, 32)"]:::tile + EXT["proof[0]
R[32, 40) from memory"]:::proofMemory + JOIN["H(R[0, 32), R[32, 40))"]:::computed + NEW["Expected new root
R[0, 40)"]:::result + + OLD --> JOIN + EXT --> JOIN + JOIN --> NEW + + classDef tile fill:#dbeafe,stroke:#2563eb,color:#111827 + classDef proofMemory fill:#fef3c7,stroke:#16a34a,stroke-width:3px,color:#111827 + classDef computed fill:#ede9fe,stroke:#7c3aed,color:#111827 + classDef result fill:#dcfce7,stroke:#15803d,stroke-width:3px,color:#111827 +``` + +The old root may have been calculated from `tile/0/000`; the extension is +resident in memory. Verification combines the known old root with the single +proof hash and compares the result with the known new root. + +## Consistency proof 2: a non-perfect old tree + +Now prove that the 20-leaf tree is a prefix of the 40-leaf tree: + +```cpp +auto proof = log.consistency_proof(20, 40); +``` + +Size 20 is not a power of two, so the old root does not line up with a single +node in the 40-leaf tree. The proof decomposes the relevant ranges: + +```mermaid +flowchart TB + R040["R[0, 40)"]:::computed + R032["R[0, 32)"]:::tile + P3240["P4 = R[32, 40)
memory"]:::proofMemory + P016["P3 = R[0, 16)
tile"]:::proofTile + R1632["R[16, 32)"]:::tile + R1624["R[16, 24)"]:::tile + P2432["P2 = R[24, 32)
tile"]:::proofTile + P1620["P0 = R[16, 20)
tile seed"]:::proofTile + P2024["P1 = R[20, 24)
tile"]:::proofTile + + R040 --> R032 + R040 --> P3240 + R032 --> P016 + R032 --> R1632 + R1632 --> R1624 + R1632 --> P2432 + R1624 --> P1620 + R1624 --> P2024 + + classDef tile fill:#dbeafe,stroke:#2563eb,color:#111827 + classDef computed fill:#ede9fe,stroke:#7c3aed,color:#111827 + classDef proofTile fill:#fef3c7,stroke:#2563eb,stroke-width:3px,color:#111827 + classDef proofMemory fill:#fef3c7,stroke:#16a34a,stroke-width:3px,color:#111827 +``` + +The proof vector contains hashes only; the range labels are shown here to make +the algorithm visible. Given `m = 20` and `n = 40`, the verifier derives where +each hash belongs. + +| Order | Illustrative range | Source | Why it is needed | +|---:|---|---|---| +| `P0` | `R[16, 20)` | tile | Seed shared by old and new reconstructions | +| `P1` | `R[20, 24)` | tile | Extend only the new reconstruction | +| `P2` | `R[24, 32)` | tile | Extend only the new reconstruction | +| `P3` | `R[0, 16)` | tile | Complete both the old and new left sides | +| `P4` | `R[32, 40)` | memory | Extend the new reconstruction to size 40 | + +Verification evolves two accumulators: + +The verifier uses the bit structure of `m` and `n` to decide which accumulator +each proof hash updates. Intuitively, `P0` seeds a subtree shared by both +histories and `P3` completes that shared old-tree boundary. `P1`, `P2`, and +`P4` cover leaves at or beyond the old size, so they extend only the new +accumulator. + +```mermaid +flowchart TB + S["Seed both accumulators with P0
old = new = R[16, 20)"]:::proofTile + A["Combine P1 on the right
new = R[16, 24)"]:::tile + B["Combine P2 on the right
new = R[16, 32)"]:::tile + C["Combine P3 on the left
old = R[0, 20)
new = R[0, 32)"]:::tile + D["Combine P4 on the right
new = R[0, 40)"]:::memory + V["Compare both reconstructed roots
with the caller's old and new roots"]:::result + + S --> A + A --> B + B --> C + C --> D + D --> V + + classDef tile fill:#dbeafe,stroke:#2563eb,color:#111827 + classDef memory fill:#dcfce7,stroke:#16a34a,color:#111827 + classDef proofTile fill:#fef3c7,stroke:#2563eb,stroke-width:3px,color:#111827 + classDef result fill:#dcfce7,stroke:#15803d,stroke-width:3px,color:#111827 +``` + +This example is mixed in a useful way: + +- The old 20-leaf state can be reconstructed from the first tile even though + the live in-memory tree has compacted those leaves. +- The newly appended range `[32, 40)` comes from memory. +- The proof is still an ordinary vector of hashes, independent of where each + hash was found. + +## The complete mental model + +```mermaid +flowchart TB + A["Append leaf hashes"]:::memory + B["In-memory left-balanced Merkle tree"]:::memory + C["A 32-entry range becomes complete
(256 entries in production)"]:::both + D["flush() publishes immutable full tiles"]:::tile + E["compact() optionally drops old resident detail"]:::summary + F["MemoryHashSource
serves whatever remains resident"]:::memory + G["TileHashSource
serves the flushed prefix"]:::tile + H["CombinedHashSource
tries memory, then tiles"]:::both + I["ProofEngine"]:::computed + J["Current or historical root"]:::result + K["Inclusion proof"]:::result + L["Consistency proof"]:::result + + A --> B + B --> C + C --> D + D -.->|optional| E + D --> F + D --> G + E --> F + F --> H + G --> H + H --> I + I --> J + I --> K + I --> L + + classDef tile fill:#dbeafe,stroke:#2563eb,color:#111827 + classDef memory fill:#dcfce7,stroke:#16a34a,color:#111827 + classDef both fill:#f3e8ff,stroke:#9333ea,color:#111827 + classDef summary fill:#e5e7eb,stroke:#6b7280,color:#111827 + classDef computed fill:#ede9fe,stroke:#7c3aed,color:#111827 + classDef result fill:#dcfce7,stroke:#15803d,stroke-width:3px,color:#111827 +``` + +The important boundary is always the last successfully flushed full tile: + +- Below it, immutable tiles can preserve proof detail after compaction. +- Above it, the incomplete frontier must remain resident in memory. +- A proof may resolve several component subtrees from either side of the + boundary, but the caller receives one normal proof. +- None of these rules depends on the illustrative width of 32. Production uses + the same model with 256-entry tiles.