feat(bus): catch-up/resync — high-water pos + bounded replayable outbox

[hartsock]

Motivation

The mesh is fire-and-forget. If a peer is offline at the moment a message is published to it, that message is simply gone — there is no "what did I miss?" path when the peer comes back. docs/decisions/bus_vs_nats.md states this plainly twice: messages "live only as long as the connection" (the Durability row of the comparison table, line 112) and "If a worker is offline when the foreman publishes, the message is gone" (Honest downsides #2, lines 126-130).

For the one-shot dispatch-foreman shape this is acceptable, because workers pull present jobs. But the moment we want agents that survive restarts, crash-loop, or roam off-LAN and rejoin — exactly the long-lived crew/mesh agents this project is heading toward — silent loss across a reconnect is a correctness hole, not a convenience gap. An agent that was down for thirty seconds should be able to deterministically resync the handful of messages it missed, and an agent that was down for an hour should at minimum get an honest gap signal instead of a silently-truncated view of the world.

We already carry exactly the bookkeeping a catch-up protocol needs — we just don't expose it as a request verb. This issue proposes the smallest broker-less, LAN-first, opt-in mechanism that closes the hole.

Current state

Per-peer high-water marks already exist — as a replay gate, not a resume cursor. agent-mesh-bus/src/replay.rs defines SequenceTracker, a Mutex<HashMap<Fingerprint, u64>> that remembers the highest sequence accepted from each sender. Its only method that mutates is check_and_advance(peer, seq) (replay.rs:111-124), which accepts iff seq > last_seen and otherwise returns Err((last_seen + 1, actual)). last_seen(peer) (replay.rs:128-135) exposes the mark for diagnostics. This is consumed in inbox.rs::on_envelope (inbox.rs:188-209): nonce check first, then check_and_advance, and a stale/duplicate sequence is rejected as BusError::BadSequence { expected, actual }. So the receiver knows precisely the next sequence it expects from each peer — but that knowledge is only ever used to slam the door on replays. Nothing reads it to ask for the gap.

There is no history to backfill from. bus.rs dials per outbound message and deliberately keeps no per-peer state: the module doc says "this version dials per outbound message ... the bus has no per-peer state to clean up when a peer disappears" (bus.rs:16-21). publish_to (bus.rs:246-257) is fire-and-forget; send_to → send_one (bus.rs:286-405) signs one SignedEnvelope, ships it, and drops it. Once send_one returns, the message exists nowhere. There is no outbox, no ring buffer, no on-disk log. A Publish whose recipient is unreachable returns BusError::Unreachable to the sender and is never retried or stored.

The outbound sequence is per-bus, not per-recipient. Worth flagging for the design: the sender's sequence is a single Arc<AtomicU64> started at 1 (bus.rs:124) and fetch_add-ed once per outbound envelope regardless of destination (bus.rs:388). So the receiver-side per-peer high-water mark in SequenceTracker is a high-water mark over that sender's global outbound counter, and a given receiver sees a subsequence of it (only the envelopes addressed to or fanned to it). Any "deliver everything after N" verb must define N against a sequence space the responder can actually reconstruct — see the design note below.

Confidentiality model (so resync proposals stay honest). Envelopes are signed, not payload-encrypted. SignedEnvelope (agent-mesh-protocol/src/envelope.rs:43-52) carries payload: ByteBuf in cleartext plus payload_cid = BLAKE3(payload) and agent_sig over ENVELOPE_TAG || recipient || nonce || sequence || payload_cid (envelope.rs:120-135). verify() checks the cert chain, the CID match, and the signature (envelope.rs:85-105) — it never decrypts. Confidentiality today rides entirely on the iroh QUIC/TLS session established by do_handshake in bus.rs. Any stored outbox therefore stores signed-but-cleartext payloads at rest, which the design must call out explicitly.

Proposed design

All of this is opt-in and backward compatible: a bus that doesn't enable the outbox behaves exactly as today, and a peer that never sends a resync request is unaffected. No broker, no homeserver — the responder is the durable point for its own outbound stream, LAN-first.

1. Resume cursor (pos) per peer. Reuse the receiver-side high-water mark already in SequenceTracker as the resume cursor: when a peer reconnects, it knows the last sequence N it accepted from each counterpart (last_seen(peer)). That N is the catch-up cursor. No new receiver state is required to ask; the new state is all on the responder side.

2. Bounded replayable outbox (depends on the durable-store seam). Add an optional, bounded, per-bus outbox that retains the last K envelopes (or last T of wall-budgeted bytes) this bus emitted, keyed so they can be re-served by destination + sequence. This is built on the durable-store seam (the same store abstraction issue #38 "data agnostic" gestures at — the outbox must not presume text, only opaque signed envelopes; file the store-seam issue if it doesn't exist yet). Bound is a hard ceiling, evicted oldest-first exactly like NonceCache (replay.rs:58-71) — bounded memory, default-safe. Default K = 0 (disabled) so the change is inert until configured.

3. The resync request verb. Add a fourth BusMessage variant alongside Request/Reply/Publish (inbox.rs:45-75), e.g. Resync { after: u64 }, meaning "re-deliver every envelope you sent me after sequence after, in order, up to your retention bound." The responder walks its outbox for that destination, re-ships the retained envelopes (already signed — no re-signing, the original agent_sig/nonce/sequence are intact), and the requester runs them through the normal on_envelope path. Note: replayed envelopes will hit the NonceCache and SequenceTracker — the resync path must bypass the duplicate-nonce rejection for deliberately-requested replays while still preserving monotonic ordering, or the requester must reset its expectation for that peer to after before draining. This interaction is the load-bearing correctness detail and needs its own tests.

4. Gap semantics beyond the bound (no silent loss). If a requester asks for after = N but the responder's oldest retained envelope is M > N + 1, the responder returns an explicit gap signal (e.g. ResyncGap { earliest_available: M }) rather than silently starting at M. A peer offline longer than the retention window learns it has an unrecoverable hole and can react (full re-bootstrap, operator alert) instead of believing it caught up. This mirrors the project's existing "never silent loss" doctrine: an unrepresentable case is an explicit signal, not a quiet truncation.

5. Confidentiality note. Because payloads are signed-not-encrypted (see Current state), the outbox stores cleartext-at-rest. The design must document this and gate it behind the same opt-in: deployments that can't tolerate at-rest cleartext keep K = 0. Encryption-at-rest for the outbox is explicitly out of scope here and a candidate follow-up.

Reference: matrix-rust-sdk

matrix-rust-sdk solves the same "client was away, deterministically re-converge" problem with a homeserver — we borrow the shape (cursor + bounded gap-aware backfill), never the broker.

• A pos resume marker (Sliding Sync / MSC4186, sync v5). crates/matrix-sdk/src/sliding_sync/mod.rs:91-103 documents pos: "The pos marker represents a progression when exchanging requests and responses ... the server acknowledges the request by responding with a new pos. If the client sends two ... requests with the same pos, the server has to reply with the same identical response." The client sends its last pos as from/to on the next request (crates/matrix-sdk/src/sliding_sync/client.rs:243-249) to resume exactly where it left off. This is the analogue of our per-peer high-water N.
• The pos is persisted so it survives a restart. crates/matrix-sdk/src/sliding_sync/cache.rs freezes it (FrozenSlidingSyncPos, cache.rs:156-158) and restores it on reconnect (cache.rs:188-193) — the cursor is durable, not just in-memory. Our equivalent is persisting the SequenceTracker high-water marks via the store seam.
• Deterministic gap backfill via the EventCache LinkedChunk gap model. A room's history is a LinkedChunk whose chunks are either events or a gap: ChunkContent::Items(..) vs ChunkContent::Gap(..) (crates/matrix-sdk-common/src/linked_chunk/mod.rs). A Gap { token } (crates/matrix-sdk-base/src/event_cache/mod.rs:24-32) holds the prev_batch//messages token used as the from parameter to fetch exactly the missing span. Back-pagination resolves a gap and reports whether the span is fully closed: BackPaginationOutcome { reached_start: bool, events: Vec<Event> } (crates/matrix-sdk/src/event_cache/caches/pagination.rs:415-425, driven by run_backwards_until/run_backwards_once at pagination.rs:61-93). This is precisely our "deliver everything after N, and tell me (reached_start / ResyncGap) whether the gap is fully closed or I've fallen off the retention window."

The pattern to lift: fast resume from a durable cursor + a bounded, gap-aware backfill that is honest about what it could not recover. The mechanism stays broker-less — in our mesh the sender of a stream is its own durable origin, not a central server.

Acceptance criteria

• An optional, bounded per-bus outbox exists (hard ceiling on retained envelopes or bytes, oldest-first eviction), built on the durable-store seam; default configuration disables it (K = 0) so behavior is unchanged for existing buses.
• A new BusMessage resync variant (e.g. Resync { after }) is defined and wired through inbox.rs::on_envelope without breaking Request/Reply/Publish serde round-trips.
• A bus exposes a resume cursor per peer derived from the existing SequenceTracker high-water mark (last_seen), persisted across restart via the store seam.
• An agent that was offline within the retention bound reconnects, issues a resync request, and receives every message it missed, in order, with each replayed envelope verifying normally (signature/CID intact, no re-signing).
• The resync replay path correctly coexists with replay defense: deliberately-requested replays are not rejected as duplicate nonces, and post-resync monotonic sequence ordering is preserved for that peer. Tested for both the in-bound and out-of-bound case.
• A peer offline longer than the retention window receives an explicit gap signal (e.g. ResyncGap { earliest_available }) and never a silently-truncated stream; a regression test asserts the gap signal fires when after < earliest_retained.
• Outbox memory is provably bounded under a flood (eviction test analogous to replay.rs::nonce_cache_evicts_oldest_at_capacity).
• Docs note that the outbox stores signed-but-cleartext payloads at rest (confidentiality rides on the QUIC/TLS session, not the payload), and that at-rest encryption is out of scope; docs/decisions/bus_vs_nats.md Durability row / downside feat: Phase 1 — mDNS LAN discovery #2 is updated to point at this mechanism.
• just check and just cov-ci pass; coverage stays at or above the 75% floor.

Relationships

• Depends on feat(bus): sans-IO MeshStore seam for durable replay state + offline outbox #42 — needs durable per-peer high-water marks + a bounded outbox to replay from.

---

Meta · risk: high (per repo CLAUDE.md) · follow-up from the matrix-rust-sdk ↔ agent-mesh architectural comparison (matrix-rust-sdk is a reference implementation, not a dependency).

File/line references were drafted against a recent checkout; line numbers are indicative — symbols are authoritative (grep by name). Substance verified against origin/main @ f63c55f.

🤖 Generated with Claude Code