Productionize the module runtime: HTTP adapter, effect wiring, hardening, core Raft limitations

CLAUDE.md

@@ -67,10 +67,18 @@ NODE_ID=node1 PORT=3001 PEERS="node2@http://localhost:3002,node3@http://localhos
   eventually consistent by default. `?consistency=strong` reads go through the
   leader's ReadIndex barrier (`node.readBarrier()`) and must never be served from
   a node that can't confirm leadership — fail closed (`NotLeaderError`) instead.
-- **Membership: one change at a time (ADR-0015).** A `CONFIG` entry takes effect
-  on *append*, not commit; never apply two overlapping changes (the code refuses
-  a second while one is uncommitted). Membership must be derived from the log so
-  every replica agrees — don't track it out of band.
+- **Membership: joint consensus, one transition at a time (ADR-0022, supersedes
+  ADR-0015).** A change C-old→C-new goes through a **joint** `CONFIG` (`members`=
+  C-new, `oldMembers`=C-old) then a **final** `CONFIG` (`members`=C-new). A `CONFIG`
+  entry takes effect on *append*, not commit. During the joint phase EVERY quorum
+  decision (election, commit, leadership confirmation) needs a **dual majority** —
+  a majority of BOTH C-old and C-new separately; route every decision through
+  `inMajority()`, never a single count. Replicate to the voting **union**
+  (`members` / `otherMembers()`). Never start a second change while one is in
+  flight (the code refuses while a `CONFIG` is uncommitted *or* the config is still
+  joint). Membership must be derived from the log so every replica agrees — don't
+  track it out of band. A new leader that inherits an uncompleted joint config must
+  finish the transition (`inheritedJoint` → finalize in `applyCommitted`).
 
 ## Conventions
 

README.md

@@ -152,11 +152,15 @@ the consensus core, so any app built on it inherits them for free.
   even if a forward times out and returns `421`. The dedup cache is **bounded**
   (`DEDUP_LIMIT`, default 10,000) with deterministic FIFO eviction, so it never
   grows without limit.
-- **Dynamic membership** — add or remove nodes at runtime (one at a time, Raft
-  §4.1) via `POST`/`DELETE /raft/members`, with no cluster restart. The new
-  configuration replicates as a log entry; a joining node catches up by normal
-  replication or an InstallSnapshot, and a leader removed from the cluster steps
-  down. A removed/partitioned node can't disrupt the cluster (leader stickiness).
+- **Dynamic membership** — add or remove nodes at runtime via **joint consensus**
+  (Raft §6 / ADR-0022) through `POST`/`DELETE /raft/members`, with no cluster
+  restart. A change transitions C-old→C-new through a joint configuration in which
+  every decision needs a majority of **both** sets (so even an arbitrary,
+  non-overlapping change is safe); the configuration replicates as a log entry, a
+  joining node catches up by normal replication or an InstallSnapshot, a leader
+  removed from the cluster steps down, and a new leader finishes a transition its
+  predecessor left unfinished. A removed/partitioned node can't disrupt the cluster
+  (leader stickiness).
 - **Leader forwarding** + `/ready` for load-balancer health checks.
 
 **Audit** — the replicated log *is* the audit trail. Each committed change is
@@ -248,13 +252,15 @@ Tests use `LocalTransport`, so they need no sockets and no database.
 
 ## Limitations (it's a POC)
 
-- Membership changes are one node at a time (Raft §4.1, no joint consensus) and a
-  joining node has no separate non-voting catch-up phase, so adding a far-behind
-  node while another is down can briefly stall commits.
+- Membership changes still apply one transition at a time, and a joining node has
+  no separate non-voting catch-up phase, so adding a far-behind node while another
+  is down can briefly stall commits. (Changes now use joint consensus — Raft §6 /
+  ADR-0022 — so the *single-server* restriction is gone; arbitrary changes are
+  safe.)
 - Reads default to the local replica (eventually consistent); linearizable reads
-  are available opt-in via `?consistency=strong` (leader-routed, no follower
-  read offloading or leases).
+  are available opt-in via `?consistency=strong`. These can be served by the leader
+  or offloaded to a follower via a ReadIndex RPC (Raft §6.4), but there are no
+  time-based leader leases (every barrier confirms a fresh heartbeat quorum).
 - The audit log grows unbounded (kept in full inside snapshots by design). The
   idempotency cache is bounded (`DEDUP_LIMIT`, FIFO), so a retry older than the
   window re-applies rather than being deduped.
-- Snapshots are sent in a single RPC (no chunking) — fine for a POC-sized state.

docs/adr/0015-dynamic-cluster-membership.md

@@ -1,6 +1,6 @@
 # 0015. Dynamic cluster membership via single-server changes
 
-- Status: Accepted
+- Status: Superseded by [ADR-0022](./0022-joint-consensus-membership.md)
 - Date: 2026-06-19
 
 ## Context

docs/adr/0021-pluggable-bft-consensus-seam.md

@@ -41,11 +41,15 @@ pluggability point.
 
 **Do not implement BFT in the prototype.** Instead:
 
-1. **Name the real seam.** Define (in documentation, and as a refactoring target)
-   a `Consensus` interface capturing exactly what the runtime needs: `submit`,
-   `readBarrier`, membership, and a leadership/status view — the subset of
-   `RaftNode` that `ReplicatedStateMachine` and the controllers actually use. A
-   BFT engine would implement *that*, not the Raft `Transport`.
+1. **Name the real seam.** Define a `Consensus` interface
+   (`src/consensus/consensus.ts`, extracted in M13) capturing exactly what the
+   runtime needs: `submit`, `readBarrier`, membership, and a leadership/status
+   view — the subset of `RaftNode` that `ReplicatedStateMachine` and the
+   controllers actually use. `RaftNode implements Consensus`, and the
+   application-facing consumers (the book/module controllers, the audit routes,
+   the effect driver, and the structural `ShardRouter` node) now depend on the
+   interface, not the concrete node. A BFT engine would implement *that*, not the
+   Raft `Transport`.
 2. **Reuse M7 as a building block.** BFT requires signed, non-repudiable messages;
    the ed25519 signing + `KeyRegistry` from M7 is the cryptographic primitive a
    BFT layer would build on (signing *consensus votes*, not just client commands).
@@ -74,11 +78,22 @@ pluggability point.
 
 - The system remains single-trust-domain (CFT) — unsuitable for adversarial,
   multi-organization deployments, exactly as ADR-0018 concluded.
-- The application boundary *below* the log — the `StateMachine` interface — is now
-  extracted (ADR-0017), so applications already plug in cleanly. But the ordering
-  boundary *above* the log (a `Consensus` interface a BFT engine would implement)
-  is still not extracted: `RaftNode` is referenced directly (e.g. `ShardRouter`,
-  controllers), so a real BFT swap would first require that second refactor.
+- The application boundary *below* the log — the `StateMachine` interface
+  (ADR-0017) — and the ordering boundary *above* the log — the `Consensus`
+  interface (extracted in M13, `src/consensus/consensus.ts`) — are now BOTH
+  extracted, so applications and the HTTP/runtime layers plug in cleanly and a BFT
+  engine could implement `Consensus` without those consumers changing. Two gaps
+  remain, both narrow and deliberate:
+  - **Transport/RPC surface.** `Consensus` is the commit-ordered-log contract, not
+    the wire protocol. The Raft RPCs (`RpcHandler` in `transport.ts`:
+    `handleRequestVote`/`handleAppendEntries`/`handleInstallSnapshot`) are
+    Raft-shaped and are NOT part of `Consensus`; a BFT engine needs a different,
+    multi-phase, signed message surface, so `raftRoutes.ts` stays typed to the
+    concrete `RaftNode`. That surface is replaced differently (not 1:1) by BFT.
+  - **Construction/wiring.** `RaftNode` is still instantiated directly by
+    `server.ts`/`moduleServer.ts`/tests. That is wiring (choosing the engine), not
+    the contract — swapping engines means changing those few construction sites,
+    not the controllers or routes.
 
 ## Alternatives considered
 

docs/adr/0022-joint-consensus-membership.md

@@ -0,0 +1,98 @@
+# 0022. Dynamic cluster membership via joint consensus
+
+- Status: Accepted
+- Date: 2026-06-21
+- Supersedes: [ADR-0015](./0015-dynamic-cluster-membership.md)
+
+## Context
+
+ADR-0015 implemented runtime membership changes with the Raft dissertation's
+**single-server change** approach (§4.1): add or remove exactly one voting member
+at a time, relying on the fact that consecutive single-server configurations
+always overlap in a majority, so two configurations can never independently elect
+a leader or commit conflicting entries. That kept the implementation small, but it
+buys safety by *constraint*, not by mechanism:
+
+- It can only ever express **one-node** changes. Replacing a node is two changes
+  (add then remove); re-sharding or swapping several nodes is a slow sequence.
+- The overlap argument is *assumed* by the single-node restriction rather than
+  *enforced* by the vote/commit math — nothing in the quorum code would stop an
+  arbitrary (non-overlapping) change from splitting the log if one were ever
+  appended. The safety property lived in a comment, not in the predicate.
+- Diehl/Ongaro later showed even single-server changes have a subtle corner case
+  around the initial configuration; the robust, fully-general answer Raft actually
+  specifies is joint consensus.
+
+For a project whose whole subject is the consensus layer, the membership change
+should be safe by *construction* and able to express arbitrary reconfigurations.
+
+## Decision
+
+Implement membership changes using Raft's **joint consensus** (§6 / dissertation
+§4.3). A change from configuration `C-old` to `C-new` transitions through a
+**joint configuration `C-old,new`** in which every quorum decision requires a
+majority of **both** `C-old` and `C-new` *separately*. Two phases, each adopted
+on append (not on commit) and each awaited to commit:
+
+1. Append a **joint** `CONFIG` entry (`members: C-new`, `oldMembers: C-old`);
+   replicate and await its commit under dual majority.
+2. Append a **final** simple `CONFIG` entry (`members: C-new`); replicate and
+   await its commit.
+
+`changeMembership` runs both phases and resolves when the final config commits.
+
+The dual-majority rule is the single gate for **every** quorum decision, so the
+safety property is now enforced by the math rather than assumed by a restriction:
+
+- **One predicate, `inMajority(ids)`** — for a simple config it is the ordinary
+  `|ids ∩ C-new| ≥ ⌊|C-new|/2⌋ + 1`; for a joint config it additionally requires
+  `|ids ∩ C-old| ≥ ⌊|C-old|/2⌋ + 1`. Election tally, commit advance, and
+  leadership confirmation all route through it. During the joint phase no decision
+  can be carried by a majority of only one configuration — making even an
+  **arbitrary** change (one whose `C-old` and `C-new` do not overlap in a
+  majority) safe.
+- **Replication is driven by the voting UNION** `C-old ∪ C-new`: the leader
+  replicates to, and confirms leadership against, every node in either
+  configuration; the dual-majority predicate then decides. (`members` is the
+  union; `configOld`/`configNew` are the two decision sets.)
+- **Configuration remains derived from the log** (latest `CONFIG` over the
+  snapshot base) and is adopted the instant an entry is **appended**, exactly as
+  before — so every replica agrees on the shape without out-of-band tracking.
+- **Joint configs survive compaction.** Snapshots and `InstallSnapshot` carry an
+  optional `oldMembers` alongside `members`, so a node that snapshots, restarts,
+  or catches up while a transition is in flight reconstructs the joint config and
+  keeps enforcing dual majority rather than silently collapsing to a simple one.
+- **One transition at a time** is still enforced (`hasUncommittedConfig` refuses a
+  second change while one is uncommitted), and a leader **not in `C-new`** steps
+  down once the final config commits (leader self-removal).
+- If leadership is lost between the two phases, the leader does not append the
+  final config from a stale term; a crash leaves the joint `CONFIG` in the log,
+  where the next leader observes it and can drive the transition forward.
+
+The HTTP surface and `raft_cluster_size` metric are unchanged from ADR-0015.
+
+## Consequences
+
+- Membership safety is enforced by the quorum math, not by a one-node restriction:
+  the dual-majority predicate makes it impossible for `C-old` and `C-new` to elect
+  leaders or commit entries independently during a transition, for *any* change.
+- The API still exposes single add/remove operations (the common case), but the
+  underlying mechanism now generalizes to arbitrary changes — a follow-up can
+  expose a batch reconfiguration with no further consensus work.
+- More moving parts than single-server changes: two voting sets, a union for
+  replication, and `oldMembers` threaded through `CONFIG`, `Snapshot`, and
+  `InstallSnapshot`. This is the price of being correct by construction and is
+  covered by tests (including a partition test proving a `C-old`-only majority can
+  neither elect nor commit during the joint phase).
+- Only one transition may be in flight at a time, as before. A new voting member
+  still joins immediately (no non-voting learner phase) — the same documented
+  catch-up caveat as ADR-0015 applies and remains a future refinement.
+
+## Alternatives considered
+
+- **Keep single-server changes (ADR-0015)** — simpler, but limited to one-node
+  changes and safe only by constraint; the corner cases and the inability to
+  express arbitrary reconfigurations motivated this supersession.
+- **Non-voting learner phase before promotion** — strictly better availability
+  during catch-up; orthogonal to joint vs. single-server and still deferred as an
+  optimisation on top of this decision.

docs/adr/README.md

@@ -27,13 +27,14 @@ supersedes the old one rather than editing history.
 | [0012](./0012-platform-layer.md) | A reusable platform layer for cross-cutting concerns | Accepted |
 | [0013](./0013-minimal-dependencies.md) | Minimal external dependencies (implement primitives in-house) | Accepted |
 | [0014](./0014-opt-in-linearizable-reads.md) | Opt-in linearizable reads via ReadIndex | Accepted |
-| [0015](./0015-dynamic-cluster-membership.md) | Dynamic cluster membership via single-server changes | Accepted |
+| [0015](./0015-dynamic-cluster-membership.md) | Dynamic cluster membership via single-server changes | Superseded by ADR-0022 |
 | [0016](./0016-crash-consistent-snapshot-persistence.md) | Crash-consistent snapshot/log persistence and snapshot transfer | Accepted |
 | [0017](./0017-generic-pluggable-state-machine.md) | Generic pluggable state machine (framework core) | Accepted |
 | [0018](./0018-application-logic-as-smart-contracts.md) | Deploying application logic as smart contracts on this framework | Proposed |
 | [0019](./0019-native-backend-blockchain-benefits.md) | Bringing blockchain benefits to backend development natively | Proposed |
 | [0020](./0020-multi-raft-sharding.md) | Multi-Raft sharding for write scaling | Proposed |
 | [0021](./0021-pluggable-bft-consensus-seam.md) | Pluggable BFT consensus — the seam, and why it is not built | Proposed |
+| [0022](./0022-joint-consensus-membership.md) | Dynamic cluster membership via joint consensus | Accepted |
 
 ## Template
 

src/consensus/config.ts

@@ -14,6 +14,7 @@ export interface NodeEnvOptions {
     electionMaxMs?: number;
     heartbeatMs?: number;
     snapshotThreshold?: number;
+    snapshotChunkBytes?: number;
     dedupLimit?: number;
 }
 
@@ -50,6 +51,7 @@ export function loadRaftConfig(env: NodeJS.ProcessEnv = process.env): NodeEnvOpt
         electionMaxMs: env.ELECTION_MAX_MS ? Number(env.ELECTION_MAX_MS) : undefined,
         heartbeatMs: env.HEARTBEAT_MS ? Number(env.HEARTBEAT_MS) : undefined,
         snapshotThreshold: env.SNAPSHOT_THRESHOLD ? Number(env.SNAPSHOT_THRESHOLD) : undefined,
+        snapshotChunkBytes: env.SNAPSHOT_CHUNK_BYTES ? Number(env.SNAPSHOT_CHUNK_BYTES) : undefined,
         dedupLimit: env.DEDUP_LIMIT ? Number(env.DEDUP_LIMIT) : undefined,
     };
 }