Productionize the module runtime: HTTP adapter, effect wiring, hardening, core Raft limitations#4
Merged
Merged
Conversation
Makes the module runtime (ADR-0019) runnable over REST, not just in tests:
- moduleController: runCommand (POST /modules/:module/:command), query
(GET .../query/:name, with opt-in ?consistency=strong via readBarrier), and
getState. Reuses the book controller's NotLeaderError -> forward-or-421 leader
forwarding verbatim; default reads are local. ApplyResult { status, data,
message } is relayed honestly (401/404/413/500 pass through).
- Signing over HTTP: the server never holds a private key. The client signs the
logical payload and sends x-signature; the server relays it onto command.sig,
building the command from x-actor/x-request-id (the same values signed over),
so a genuine client verifies on apply and a forged signature is rejected 401.
- moduleApp/moduleServer: a module analog of createApp/server.ts, reusing the
app-agnostic /raft, /audit, /health, /ready, /metrics routes; demo modules
counter/notes/accounts.
- ModuleNode type added to moduleStateMachine.ts (mirrors BookNode).
Additive: no src/consensus or moduleHost changes; the prior 160 tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01NneoGGKxBp5ZExL7D7i1Lu
…leader Closes the committed-intent effect loop in a running cluster (M2 only ran in tests). Additive — no src/consensus changes, no RaftNode commit hook. - Effect result as an app command: ModuleAppCommand becomes a union of ModuleInvokeCommand (type 'MODULE') and ModuleEffectResultCommand (type 'MODULE_EFFECT_RESULT', carrying the EffectResultEntry). ModuleStateMachine.apply routes the latter to host.applyEffectResult. - EffectDriver: a leadership-aware poller that, on the LEADER only, drains host.pendingEffects(), runs handlers via an EffectExecutor, and submits the result back through the log (buildEffectResultCommand, stable requestId=effect:<key>). Re-entrancy guarded; interval unref'd; NotLeaderError swallowed so a lost-leadership drain leaves the effect pending for the new leader. Exactly-once committed state (outbox done-guard + idempotent applyEffectResult) with at-least-once handlers. - moduleServer wires an EffectDriver with a demo http handler. Review fix: the internal onResult dispatch (e.g. settle) now bypasses signature verification via a shared dispatchAndEnqueue seam — without it, effects on a signed cluster would 401 and stall. verifySignature for actor commands is unchanged. Adds signed-host effect tests proving the loop completes. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01NneoGGKxBp5ZExL7D7i1Lu
Defines Consensus<C,T,A> (src/consensus/consensus.ts) — the commit-ordered-log contract the application/HTTP/runtime layers actually use (submit, readBarrier, changeMembership, leadership/status, lifecycle, app + stateMachine accessors). RaftNode now `implements Consensus` (a 2-line change: import + clause; no algorithm/behavior change). The app-facing consumers — book/module controllers, audit routes, and the effect driver — depend on the interface, not RaftNode; ShardRouter's structural node type is already a subset of it. This is the ordering seam ADR-0021 names as the BFT-swap prerequisite. The Raft RPC surface (RpcHandler) stays protocol-specific (raftRoutes on RaftNode), and RaftNode is still constructed directly by the servers/tests (wiring, not the contract). ADR-0021 updated to record that both the StateMachine seam (below the log) and the Consensus seam (above it) are now extracted. Behavior-preserving: all 176 tests pass, tsc clean, cast-free. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01NneoGGKxBp5ZExL7D7i1Lu
Adds tests/runtime/moduleCrashConsistency.test.ts, the module analog of the book crashConsistency suite: a durable single-node cluster (RaftNode + LocalTransport + real FileStorage over a temp dir + ModuleStateMachine) is restarted by rebuilding a fresh node on the same data dir, proving module state survives: - log-replay restart (no snapshot): whole-state + keyed module state, the leader-resolved note id/createdAt (deterministic MODULE replay), and the Merkle audit root all round-trip; - snapshot + restart: state reconstructed from the snapshot (keyed StateStore dump + __audit leaves round-trip); - snapshot-then-tail: restore from snapshot + replay the post-boundary tail, exact arithmetic, no double-apply or loss. No bugs found — module state, the keyed store, and the audit root all survive a real durable restart; no source changes needed. The reconcileLog "snapshot landed, log didn't" path is application-agnostic (book suite covers it; modules ride the same RSM snapshot). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01NneoGGKxBp5ZExL7D7i1Lu
Exposes module-runtime signals over the existing Prometheus /metrics endpoint
(per-node, like the Raft gauges), additive and host-metrics-free:
- module_commands_total{module,command,status} + module_command_duration_ms,
incremented in the ModuleStateMachine adapter (pure observability on the apply
path — never touches replicated state/snapshot/audit/outbox; optional, no-op
without a registry).
- Scrape-time collector: module_outbox_pending/done, module_audit_size,
module_registered (read from host accessors; the one host addition is a pure
moduleCount() read).
- effect_runs_total{kind,outcome} from the EffectExecutor; shard_has_leader{shard}
+ shard_count from ShardRouter.collectMetrics.
- moduleServer wires metrics into the state machine, driver, and a collector.
Review fix: outcome is attributed from a try/catch around the handler call ALONE
(success XOR failure, counted once); a later submit failure is control flow and
no longer double-counts as a handler failure. Adds a unit test proving both the
failure path and the success-with-failing-submit case.
Label cardinality is bounded (module/command from registered defs, status a small
set, kind a handler key) — no requestId/actor/input becomes a label.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01NneoGGKxBp5ZExL7D7i1Lu
Replaces the single-RPC snapshot transfer (a documented POC limitation) with bounded-size chunked streaming, so a far-behind follower is caught up without a single oversized RPC. - Wire (types.ts): InstallSnapshotArgs.data is now a string slice plus offset/done; identity fields ride every chunk. - Leader (sendSnapshot): serializes the durable snapshot's data once, streams snapshotChunkBytes-sized slices sequentially (awaiting each reply), aborts on lost reply / lost leadership / term change, steps down on a higher-term reply, and sets matchIndex/nextIndex only on the final done ack. A snapshot that fits in one chunk sends exactly one done chunk (single-RPC parity). - Follower (handleInstallSnapshot): a per-node reassembly buffer keyed by snapshot identity; contiguous-extend only, else drop-and-ack-without-install (fail closed → leader retries from 0); JSON.parse failure also fails closed. The install logic (restore, indexing, fig.13 tail, members, persist-before-log) is byte-for-byte unchanged — chunking is pure transport framing. - Concurrency: a snapshotInFlight guard stops a heartbeat launching a second overlapping stream to the same peer (now that a stream spans many awaits); cleared in a finally on every exit + on becomeLeader. - Config: SNAPSHOT_CHUNK_BYTES (default 64 KiB). Review confirmed install semantics/indexing unchanged, reassembly fail-closed (no corrupt-install sequence), and the in-flight guard leak-free. NIT docs applied (code-unit offset wording; CFT buffer-bound note). All 188 tests pass. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01NneoGGKxBp5ZExL7D7i1Lu
…6.4) Linearizable reads can now be served LOCALLY by a follower instead of being forwarded whole to the leader (a documented POC limitation). Additive — the leader readBarrier()/confirmLeadership() paths are byte-for-byte unchanged. - New lightweight ReadIndex RPC (types/transport/raftRoutes): the follower asks the leader to confirm leadership and return a safe read index. - handleReadIndex (leader half exposed as RPC): captures readIndex = commitIndex BEFORE running the existing confirmLeadership heartbeat-quorum round; returns success only if still leader at the same term. Steps down + refuses if the requester's term is higher. - readBarrierLocal (Consensus seam): leader → existing readBarrier; follower → ReadIndex RPC then waitForApplied(readIndex), then serve locally. Fail-closed on every uncertainty (no/unknown leader, null RPC, success:false, higher term, apply timeout, role churn) — never serves stale. - Controllers' strong-read path calls readBarrierLocal and serves locally, keeping the forward/421 fallback. Adds raft_follower_reads_total. Review confirmed: no path serves a non-linearizable read, every uncertainty is fail-closed, the leader barrier is unchanged, and confirmLeadership from within the RPC handler has no re-entrancy/deadlock under LocalTransport. Review NITs applied (requester-term step-down now used; clarifying comments). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01NneoGGKxBp5ZExL7D7i1Lu
Replace the single-server membership changes of ADR-0015 with full joint consensus (Raft §6 / dissertation §4.3), recorded as ADR-0022 (supersedes ADR-0015). A change C-old→C-new now transitions through a joint configuration C-old,new in which EVERY quorum decision — election tally, commit advance, and leadership confirmation — requires a dual majority: a majority of BOTH C-old and C-new separately. Safety is now enforced by the quorum math rather than assumed by a one-node restriction, so even an arbitrary (non-overlapping) change is safe. Core changes (src/consensus/raftNode.ts, types.ts): - One dual-majority predicate `inMajority(ids)` gates election, commit, and leadership confirmation; replication targets the voting UNION C-old ∪ C-new. - Configuration stays derived from the log, adopted on append; `configOld`/ `configNew` carry the two decision sets, `members` the union. - Two-phase `changeMembership`: append joint → await commit → append final → await commit. A leader not in C-new steps down once the final config commits. - A new leader that INHERITS an unfinished joint config completes the transition (`inheritedJoint` → finalize in `applyCommitted`), so a mid-transition crash can't wedge the cluster in joint consensus (Raft §4.3). - Joint configs survive compaction/transfer: `oldMembers` is threaded through `Snapshot` and `InstallSnapshotArgs` so a snapshotting/catching-up node keeps enforcing dual majority instead of collapsing to a simple config. - Guard against starting a second change while a transition is in flight (uncommitted CONFIG or still-joint), and reject an empty target configuration. Tests (tests/membership.test.ts): dual-majority partition test (a C-old-only majority can neither elect nor commit), an arbitrary non-overlapping change, and new-leader finalization of an inherited joint transition. The dual-majority test is made deterministic via asymmetric election timers (n1 the sole initiator), so it no longer flaked under parallel-worker CPU contention. Docs: ADR-0022, ADR-0015 marked superseded, README/CLAUDE membership + limitations updated. Full suite green (196 tests). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01NneoGGKxBp5ZExL7D7i1Lu
naveed949
changed the title
naveed949
marked this pull request as ready for review
June 21, 2026 19:52
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Productionizes the module runtime (ADR-0019) across three tiers — usability → hardening → core POC limitations — milestone by milestone. Each milestone was implemented, then reviewed by a separate pass, with blocking/should-fix findings resolved before commit;
tsc --noEmitand the full Jest suite are green at every step. 196 tests passing.Builds on the merged
StateMachineseam (PR #2) and module runtime (PR #3).Tier 1 — usable end-to-end
45cc49a— Generic HTTP adapter for module commands:POST /modules/:module/:command,GET .../query/:name(opt-in strong reads), leader forwarding reused from the book controller, and signing-over-HTTP (client signs, server relaysx-signature). PlusmoduleApp/moduleServerrunnable entry.f2b44f6— Drive the effect outbox post-commit on the live leader: anEffectDriverpolls the committed outbox, runs each effect handler at-least-once, and folds the edge-resolved result back through aMODULE_EFFECT_RESULTcommand (exactly-once state). Fixes a signed-cluster path where the internal settle was re-verified as a client signature.Tier 2 — hardening
7103ca6— Extract aConsensus<C,T,A>interface above the log (the commit-ordered-log seam, ADR-0021);EffectDriver/ShardRouter/controllers depend on it rather thanRaftNodeconcretely.4b8d3d5— Module crash-consistency overFileStorage: snapshot/restore + log replay round-trips the whole runtime (reducers, outbox, Merkle audit, keyed store).3bf38a4— Runtime observability over the existing Prometheus registry: command throughput/latency, outbox pending/done, audit size, effect-run outcomes, and shard leadership — all pure side-effects off replicated state (determinism-safe). Handler outcome is attributed exactly once.Tier 3 — core POC limitations
c9f5422— Chunked InstallSnapshot (Raft fig.13): snapshots stream in bounded string slices with offset/done reassembly, fail-closed, guarded by an in-flight gate — no more single-RPC whole-snapshot transfer.b535c65— Follower read offloading via a ReadIndex RPC (Raft §6.4): a follower can confirm a safe read index with the leader and serve a linearizable read locally, instead of forwarding the whole read. Fail-closed (requester steps down on a newer term). No time-based leases.6521a87— Joint-consensus membership (Raft §6 / ADR-0022, supersedes ADR-0015): a change C-old→C-new transitions through a joint configuration in which every decision (election, commit, leadership confirmation) needs a dual majority of BOTH sets, replicating to the voting union. Safe even for arbitrary, non-overlapping changes. A new leader completes a transition its crashed predecessor left unfinished (Raft §4.3); joint configs survive compaction/transfer viaoldMembers. Tests cover the dual-majority partition property, an arbitrary non-overlapping change, and inherited-joint finalization.Docs
ADR-0022 added (ADR-0015 superseded); README + CLAUDE.md membership invariant and Limitations updated to reflect joint consensus, snapshot chunking, and follower reads.
🤖 Generated with Claude Code
Generated by Claude Code