Edge read replicas — complete client-side replica (ADR-0023, M20–M29)#7
Closed
naveed949 wants to merge 10 commits into
Closed
Edge read replicas — complete client-side replica (ADR-0023, M20–M29)#7naveed949 wants to merge 10 commits into
naveed949 wants to merge 10 commits into
Conversation
Add a read-only, non-voting tap on the committed log: the consensus-core primitive an edge read replica consumes. A consumer bootstraps from the snapshot boundary, replays the committed tail, then live-tails new commits. None of it participates in consensus — a stream consumer never votes, acks, or affects quorum/commit. Core (RaftNode): - onCommitted(listener) — subscribe to committed entries in commit order, emitted from applyCommitted; listener exceptions isolated. - getCommittedEntries(afterIndex) / getCommitIndex() / getSnapshotIndex() — replay the in-memory committed tail and report the live head/boundary. - needsSnapshot(afterIndex) / getStreamSnapshot() — detect a compacted gap and hand off the snapshot at the boundary for bootstrap. HTTP: - GET /raft/stream?fromIndex=N — Server-Sent Events stream (snapshot → entry* → caughtup → live entry*), served from ANY node's local, eventually-consistent state, so read serving fans out across the cluster (and, via a browser EventSource, past it). Keepalive + reconnect hints. Tests: tests/readStream.test.ts — commit-order live tail, late-consumer replay, follower fan-out, and snapshot handoff after compaction. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01NneoGGKxBp5ZExL7D7i1Lu
A read-only, non-voting replica of the application state machine that tails a node's committed-log stream and serves reads from local state — in Node or the browser. Writes still go through the leader; the replica never votes or acks. - EdgeReplica<C,T>: wraps any StateMachine (the SAME deterministic code a server runs), bootstraps from the snapshot boundary, applies committed commands, and serves reads locally. Owns reconnection with backoff (resumes from its applied index, re-bootstrapping cleanly if compaction outran it), a change-feed (onChange), and read-your-writes (waitForIndex). - LogStreamSource seam with two implementations: HttpStreamSource (Node, a stdlib SSE client — no new deps, ADR-0013) and EventSourceStreamSource (browser, native EventSource injected structurally so it builds without DOM lib). - Re-exported from the public surface (src/index.ts). Tests: tests/edgeReplica.test.ts — over real HTTP + SSE: bootstrap + live tail, onChange + read-your-writes, snapshot handoff after compaction, and a 3-node cluster proving a FOLLOWER serves the stream (read fan-out). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01NneoGGKxBp5ZExL7D7i1Lu
A runnable demonstration of the edge read replica, browser + Node: - examples/edge-replica/index.html + app.js — a zero-build browser page that tails GET /raft/stream with the native EventSource, applies committed book commands to an in-page store, and renders a live-updating list. Add a book (a write, forwarded to the leader) and watch it stream back. CORS is already enabled on the node. - examples/edge-replica/node-example.js — the SAME-CODE path: runs the real EdgeReplica + BookStateMachine from dist (no hand-port), printing a live table. - examples/edge-replica/README.md — how it works + how to run both. Docs: README gets a /raft/stream API row and an "Edge read replicas" section; CLAUDE.md architecture map gains src/edge/. The browser reducer is a hand-port flagged with the determinism/code-sharing caveat ADR-0023 raises; the Node example shows the shared-code alternative. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01NneoGGKxBp5ZExL7D7i1Lu
…-0023)
Close the ADR's make-or-break prerequisite: never expose the whole committed
log to an untrusted edge client. A StreamGuard authenticates each /raft/stream
connection and returns a per-connection ScopedFilter that restricts BOTH the
bootstrap snapshot and the live entry feed to the client's authorized scope.
Framework (src/edge/streamGuard.ts):
- StreamGuard.authorize(token) -> ScopedFilter | null (null => 401), and a
ScopedFilter { filterSnapshotState, includes } seam. Domain-agnostic: the
framework authenticates-and-filters; the app defines what a scope means.
- extractStreamToken: Authorization: Bearer <token> (Node) or ?token= (browser,
since EventSource can't set headers).
Endpoint (raftRoutes): authorize on connect; scoped snapshot (state only — no
audit/dedup leak) + only in-scope entries. The cursor still advances past
out-of-scope entries so the client stays current; EdgeReplica's index check is
relaxed from strict-contiguous to monotonic (a scoped stream legitimately skips
indices; SSE rides one ordered TCP connection so nothing is dropped mid-stream).
Wiring: createApp accepts a streamGuard; server.ts builds one from STREAM_TOKENS
(default public `demo=*`, swap for JWT in prod). Token support added to both
stream sources and to the worked example.
Book example (models/bookStreamGuard.ts): scope by publisher ("acme" sees only
"Acme Press"), with a TOKEN=scope registry.
Tests: tests/edgeAuthz.test.ts — 401 on missing/invalid token, scoped client
sees only its publisher (snapshot + live), all-scope sees everything.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01NneoGGKxBp5ZExL7D7i1Lu
Ship the REAL state machine to the client instead of hand-porting it — the answer to ADR-0023's "determinism across client builds" hazard. The browser now runs the exact EdgeReplica + BookStateMachine the server runs. - src/edge/browser.ts: browser entry re-exporting the browser-safe surface (EdgeReplica, EventSourceStreamSource) plus the shared BookStateMachine. Excludes the Node-only HttpStreamSource/StreamGuard so the emitted graph touches no Node builtin. - bookStateMachine.ts: type-only RaftNode import, so it pulls no consensus runtime into the browser bundle. - tsconfig.browser.json + scripts/build-browser.js + `yarn build:browser`: compile to ESM and append the .js import extensions browsers require — zero extra dependency. Output (examples/edge-replica/lib/) is gitignored. - Example rewritten to import the compiled SDK (app.js, index.html); README documents the build + scoped-token demo. Verified: the compiled ESM loads as a browser would and converges against a live node under both all-scope and a scoped token. Tests: tests/eventSourceSource.test.ts — the browser source (EventSourceStream- Source) over a real node via a Node EventSource polyfill: convergence, live tail, and token-in-URL scoping. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01NneoGGKxBp5ZExL7D7i1Lu
…ADR-0023) Capstone for the edge read replica: - Observability: raft_stream_subscribers gauge — active committed-log stream subscribers (edge replicas) per node, for read-fan-out monitoring. - ADR-0023 → Accepted, with an Implementation status section recording what landed (M20–M24) and what is deferred (SSE backpressure/connection caps, JWT/session auth + TLS/wss, client-side audit-chain verification). - docs/OPERATIONS.md: "Edge read replicas in production" — auth (swap the demo token registry for JWT), token transport caveats (?token= vs Bearer), scaling reads via followers, partial replication, compaction-outrun behavior, limits. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01NneoGGKxBp5ZExL7D7i1Lu
naveed949
changed the title
Harden edge-replica auth: replace reliance on a guessable static token
registry with cryptographically-verified, short-lived, scoped tokens.
Dependency-free (Node crypto only, ADR-0013).
- src/edge/signedToken.ts: a compact JWT-shaped HS256 token. mintStreamToken /
verifyStreamToken (base64url, constant-time length-guarded timingSafeEqual,
always computes HS256 so there is no algorithm-confusion, never throws on
adversarial input) and a generic createSignedTokenGuard({ secret, toFilter }).
exp is REQUIRED and numeric — a missing/non-numeric expiry fails closed (no
perpetual tokens).
- bookStreamGuard: extract bookScopeToFilter / claimsToBookScope so the signed
and static guards share one scope->filter source of truth; buildSignedBook-
StreamGuard(secret) maps a `scope` claim ("*" or a publisher) to the filter.
- server.ts: use signed tokens when STREAM_TOKEN_SECRET is set, else fall back
to the demo STREAM_TOKENS registry. scripts/mint-token.js + `yarn mint-token`.
- Public exports + docs (OPERATIONS.md auth/token-transport, ADR-0023 status).
Reviewed (security pass, approve-with-nits): applied the fail-closed exp policy
and added algorithm-confusion + missing-exp regression tests.
Tests: tests/edgeSignedToken.test.ts — 8 unit (valid, tampered payload/sig,
wrong secret, expired, malformed, alg-confusion, missing-exp) + 3 integration
(scoped/all over the real endpoint, 401 on missing/expired/tampered).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01NneoGGKxBp5ZExL7D7i1Lu
Protect a node from slow or abundant edge-stream consumers. - Per-node connection cap: GET /raft/stream tracks active connections and, at capacity, fails closed with 503 + Retry-After (checked AFTER auth, so a rejected token never consumes a slot). Default 10000, env STREAM_MAX_CLIENTS. - Backpressure: after each SSE write, if res.writableLength exceeds a ceiling (default 1 MiB, env STREAM_MAX_BUFFER_BYTES) the consumer can't keep up — the connection is dropped (res.destroy) instead of buffering unboundedly. It reconnects and resumes from its applied index. Cleanup (clear keepalive, unsubscribe, release the slot) is idempotent across the drop path and the req/res close events. - Observability: raft_stream_rejected_total and raft_stream_dropped_total counters; RaftNode.streamSubscriberCount() backs raft_stream_subscribers. - Wiring: AppDeps.streamLimits threaded through createApp; server.ts reads the env knobs. Docs: OPERATIONS.md limits + ADR-0023 status appendix. Reviewed (resource-safety pass, request-changes): fixed B1 — a connection dropped DURING catch-up (before the live-tail subscription was set up) would register an onCommitted listener that nothing removed, leaking it and inflating raft_stream_subscribers. The subscription is now guarded by `if (!closed)`, mirroring the keepalive. Added a deterministic regression test (verified it fails on the pre-fix code). Tests: tests/edgeBackpressure.test.ts — cap (503 + Retry-After, slot freed on close), slow-consumer drop, and the B1 listener-leak regression. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01NneoGGKxBp5ZExL7D7i1Lu
Let an edge replica re-derive and verify the tamper-evident audit hash-chain the server maintains — proving the served history is internally consistent (ADR-0023's end-to-end tamper-evidence). Node + full streams; reuses the existing ReplicatedStateMachine chain logic (Node crypto, sync). - EdgeReplica gains `verifyAudit: true`. An internal Applier seam swaps strategy: DirectApplier (default, behavior unchanged) vs AuditingApplier, which applies through a ReplicatedStateMachine wrapping the same app so it rebuilds the chain. Reads still come from replica.app. New API: verifyAudit() / auditHead() / getAuditLog() — all null when not auditing OR when the bootstrap snapshot lacks audit data (a scoped/partial stream), so a scoped client never gets a false "valid" (the uniform-log-vs-authz tension). - Browser audit verification remains future work (needs async WebCrypto). Reviewed (request-changes): fixed B1 — a SCOPED, from-scratch stream sent the full materialized scoped state as a bootstrap marker but left the cursor at 0, so the catch-up loop replayed every committed entry ON TOP of it, double-applying non-idempotent commands (BORROW/RETURN) and silently diverging the replica (the existing scoped tests used ADD-only fixtures and masked it). The marker now labels lastIncludedIndex = the live head and advances the cursor, so scoped clients materialize at the head and tail from there — no replay. Tests: tests/edgeAudit.test.ts (happy path + server-head match, live tail, tamper detection, scoped⇒unavailable, non-auditing⇒null) and an edgeAuthz regression for the B1 BORROW double-apply (verified it fails pre-fix). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01NneoGGKxBp5ZExL7D7i1Lu
naveed949
changed the title
Complete client-side audit-chain verification in the BROWSER, and fix an M28 regression that pulled Node crypto into the browser bundle. - src/consensus/auditChain.ts (new): the audit-hash preimage as a shared, pure module (GENESIS_HASH, auditEntryPayload, AuditRecord/AuditPayloadInput). ReplicatedStateMachine now uses it, so the server and client hash formats are one source of truth and cannot drift. - src/edge/sha256.ts (new): webcryptoSha256Hex over globalThis.crypto.subtle — one async hasher that runs in Node 20+ AND the browser, producing the identical lowercase-hex digest the server's Node crypto produces. - edgeReplica.ts: AuditingApplier reworked to be browser-safe — applies app commands directly and records audit INPUTS (mirroring the server's NOOP-skip / CONFIG-audited-as-200 / metadata-default rules), with NO ReplicatedStateMachine value-import (import type only), so the browser graph no longer touches Node crypto. verifyAudit()/auditHead() are now async (fold the chain via WebCrypto); EdgeReplicaOptions.auditHasher allows injection. Scoped streams still report unavailable (null); the non-auditing path is unchanged. M28 broke the browser bundle (edgeReplica -> replicatedStateMachine -> 'crypto'); this restores it: the browser runtime graph trace is clean (BAD: NONE, no replicatedStateMachine.js). Verified end to end: the COMPILED browser bundle, loaded as a browser would, verifies the chain (valid) and its audit head equals the live server's head byte-for-byte. Reviewed (approve-with-nits): unified the duplicated audit-record type into auditChain.ts and made auditHead() return GENESIS_HASH for an empty-but-available chain (so available => non-null head, consistent with verifyAudit). Tests: tests/edgeAudit.test.ts — async verify/head over WebCrypto, head equality with the server, async tamper detection, scoped⇒null, non-auditing⇒null. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01NneoGGKxBp5ZExL7D7i1Lu
naveed949
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements ADR-0023 — Edge read replicas end to end, with nothing deferred: a read-only, non-voting replica of the application state machine that runs outside the cluster — browser or Node — by tailing a node's committed-log stream and serving reads from a local copy with no network round-trip, live-updating as writes commit. ADR is Accepted.
Milestones
RaftNode+GET /raft/streamSSE, served by any nodeEdgeReplicaSDK — bootstrap → tail → local reads, reconnect/backoff,onChange,waitForIndex; Node + browser stream sourcesStreamGuard/ScopedFilter)BookStateMachineas the server (yarn build:browser)raft_stream_subscribersmetric, ADR → Accepted, ops guidecrypto), fail-closedexp, no alg-confusion;yarn mint-tokenraft_stream_rejected/dropped_totalEvery hardening milestone went implementer → reviewer → fix
The review gate caught three real bugs that a green suite was hiding:
exp+ an algorithm-confusion regression test.onCommittedlistener (inflatingraft_stream_subscribersforever) → guarded subscription + regression test.BORROW/RETURN, masked by ADD-only fixtures) → materialize-at-head fix + regression test.cryptointo the browser bundle; reviewer approve-with-nits → unified the audit-record type + madeauditHead()consistent.Each fix shipped with a regression test verified to fail on the pre-fix code.
What makes it client-usable & production-shaped
raft_stream_subscribers/raft_stream_rejected_total/raft_stream_dropped_total.Testing
tscclean · full suite 236/236 (32 suites) · new socket/audit suites stable across repeated runs · browser bundle re-verified to import no Node builtins · the compiled browser ESM loaded as a browser and verified the audit chain (valid, head == server head) against a live node.🤖 Generated with Claude Code
https://claude.ai/code/session_01NneoGGKxBp5ZExL7D7i1Lu