feat(syncing): maintained RBSR fingerprint index for fast blob propagation

[juligasa]

What

Replaces the per-round RBSR store rebuild with a maintained, persistent fingerprint index so an interested, already-connected peer gets a new blob in ~1s without melting CPU.

Today every reconcile round (and every peer) rebuilds the entire sorted blob set from SQLite via collectBlobs. This PR persists each scope's resolved set, serves range fingerprints from an O(log N) monoid tree, keeps the set current incrementally as blobs are indexed, and drops the hot discovery cooldown to 1s.

How

• Monoid tree (rbsr.NewTreeStore): an augmented treap with deterministic hash-derived balancing. RangeFingerprint is O(log N) and byte-identical to the existing linear fold (the fold never sets the accumulator length, so the tree doesn't either). Supports post-seal insert. Wired via an optional RangeFingerprinter seam in Session.Fingerprint (falls back to the fold otherwise).
• Persistence (rbsr_scope / rbsr_item + migration): a scope = a canonical discovery key + protocol version. We persist the resolved blob set (not literal tree nodes — a treap with rotations doesn't map to stable rows) and rebuild the in-memory tree from those rows. Materialization runs collectBlobs once; reindex drops the tables (lazy re-materialize).
• loadStore serves from the persistent index, with a reindex-gated fallback to the legacy rebuild.
• Oracle (affectedScopes): the incremental membership decision — which materialized scopes a freshly indexed blob joins. Seed-and-forward: only the seeding decision is new logic; the closure reuses collectBlobs' own forward walk, so it can't silently diverge. Covers the dominant document-edit path (a new Ref + the Changes it heads via the media closure).
• Index hook (Index.SetIndexedHook): patches materialized scopes inside the indexing transaction, so the maintenance commits atomically with the blob.
• Shadow-verify: a periodic sweep recomputes each scope's set the authoritative way and re-materializes any that drifted — the safety net for the edges the oracle defers (inbound Contact-by-subject, capability delegation, late link targets).
• /0.9.3 canonicalization primitives: canonicalCodecFor (dag-pb → raw), scope keyed by protocol version, build-from-rows applies it. Unit tests prove a dag-pb blob advertises identically to a raw-storing peer.
• Hot cooldown hardcoded to 1s.

Deferred (NOT in this PR): live /0.9.3 protocol negotiation

The canonicalization primitives are in and tested, but the live wire negotiation is deliberately left out and must be a separate, e2e-tested change. I implemented it, found a concrete breakage, and reverted it:

• connect.go gates connections with CheckHyperMediaProtocolVersion(..., n.protocol.Version), requiring the peer's FindHypermediaProtocol result to exactly equal 0.9.2. Peerstore GetProtocols returns a map (nondeterministic order), so advertising both 0.9.2+0.9.3 would intermittently surface 0.9.3 and reject the peer.
• Activating it needs: rewriting FindHypermediaProtocol/CheckHyperMediaProtocolVersion for highest-mutual negotiation, a second listener/server tagging the canonical context, and client-side dial selection + a canonicalized client store. That touches connection establishment and must be validated with the daemon e2e harness.

Until then the index serves the legacy-codec set (identical fingerprints to the old path), so this PR is backward-compatible.

[burdiyan]

the fold never sets the accumulator length, so the tree doesn't either

@juligasa this is actually an oversight I've been meaning to improve for a long time, and bumping the protocol version. Maybe we should take advantage of this change and do it?

It should also help us with measurements a little bit, because we'll avoid new peers talking to old peers as it could show quite different performance characteristics.

[burdiyan]

Does it mean that every document will produce multiple scopes by default? E.g. the regular one, the recursive one, the depth-one one, and maybe some else, e.g. combining different blob types?

And for each of these scopes there's going to be tons of rows in rbsr_items table?

Maybe we should quit having filters per scope, and for things that we need filters for we simply define different scopes. This should simplify the whole thing by flattening it into one dimensions, instead of having multiple.

[burdiyan]

@juligasa

Looks like the indexing hook misses blobs indexed by unstash cascades:

seed/backend/blob/index.go

Line 1947 in 10d9ec3

return indexBlob(trackUnreads, false, conn, id, c, data, bs, log, wc)

calls indexBlob for previously stashed blobs during reindexStashedBlobs, but those blob IDs are never passed to idx.runIndexedHook. The only hook calls are in Put / PutMany for the top-level fresh IDs it seems:

seed/backend/blob/index_blockstore.go

Line 44 in 10d9ec3

return idx.runIndexedHook(conn, []int64{id})

and

seed/backend/blob/index_blockstore.go

Line 104 in 10d9ec3

return idx.runIndexedHook(conn, indexed)

. So when a missing dependency/capability arrives and unstashes older refs/changes, the legacy tables get updated but rbsr_item does not. Reconcile can then serve a stale-short maintained index.

[burdiyan]

@juligasa

1. Don't we want to bump p2p protocol version?

   seed/backend/hmnet/hmnet.go

   Line 59 in 1b52e54

   protocolVersion = "0.9.2"

2. I'm really concerned with this every 5 minute full sweep:

   seed/backend/hmnet/syncing/syncing.go

   Lines 529 to 556 in 1b52e54

   	func (s *Service) runShadowVerify(ctx context.Context) {
   	// Timer, not Ticker: the sweep recomputes every scope via collectBlobs and
   	// can run long; a Ticker would keep firing on a fixed cadence regardless,
   	// stacking runs. Fire once at 0, then reset to the interval after each cycle.
   	timer := time.NewTimer(0)
   	defer timer.Stop()
   	for {
   	select {
   	case <-ctx.Done():
   	return
   	case <-timer.C:
   	}
   	checked, drifted, err := shadowVerifySweep(ctx, s.db)
   	switch {
   	case err != nil:
   	s.log.Warn("ShadowVerifyFailed", zap.Error(err))
   	case drifted > 0:
   	s.log.Warn("ShadowVerifyDrift", zap.Int("checked", checked), zap.Int("drifted", drifted))
   	default:
   	s.log.Debug("ShadowVerifyClean", zap.Int("checked", checked))
   	}
   	timer.Reset(shadowVerifyInterval)
   	}
   	}

Isn't it super wasteful to re-evaluate all the scopes every 5 minutes? I mean, it really doesn't make sense doing it if there's no new blobs for example. It seems like we are not really confident in that we are able to accurately detect which blobs affect which scopes, so we have to do this safety net thing every 5 minutes to make sure we don't mess it up.

I don't mind if we run it for a while, but having it long-term doesn't seem right. And it doesn't seem right to run it when we can know for sure there's nothing that could have changed the world (e.g. no new blobs since the last sweep).

[burdiyan]

I just realized something that made me think that we definitely should bump the protocol version, and change the signature of the ReconcileBlobs RPC in syncing.proto. We must adapt it to this new notion of scopes, and we must avoid combining multiple scopes into a single range, like we currently do by allowing multiple filters. This will break all the performance improvements from having well-defined scopes, I think.

[burdiyan]

Here's a review finding from my agent (GPT 5.6 Sol). It looks valid, especially if we do something about my previously shared concerns about the need for the background shadow sweep I shared above.

Preserve missing Raw targets across materialization

Filtering out size < 0 placeholders here loses late Raw media targets permanently from this scope: Raw blocks bypass indexBlob and therefore never reach MaintainRBSRIndex when their data later arrives. The authoritative set will contain the Raw blob after download, but the maintained set stays short until shadow verification runs and another reconciliation re-materializes it. Either retain placeholder membership (filter only while serving) or invoke maintenance for newly downloaded non-indexable blocks.