Conclave

A cluster of equal peers that agree on every change — no central database, no single point of authority.

Conclave is a decentralized, database-free backend built on a from-scratch Raft consensus implementation — leader election, log replication, snapshotting + InstallSnapshot, joint-consensus dynamic membership, and opt-in linearizable reads. State is a replicated log applied to a per-node in-memory state machine, so every node converges to identical state through agreement rather than a shared store. The replicated log doubles as a tamper-evident, hash-chained audit trail, and the consensus layer ships with built-in observability, idempotency, and fault tolerance.

The demo application is a small library book service (add / list / update / delete / borrow / return), but the books are deliberately incidental — a deterministic state machine riding on top of the consensus layer. Consensus is the real subject.

Why it exists. This is a study of distributed-systems correctness: how a backend stays consistent and available when no single machine is in charge. Implemented with the Node standard library only (no consensus/distributed-systems dependencies), backed by ~8.6k lines of tests against ~9.2k lines of source, and documented as a series of Architecture Decision Records. See docs/PHILOSOPHY.md for the why.

Highlights

• 🗳️ Raft consensus from scratch — leader election, log replication, and commit (Raft fig. 2), implemented on the Node standard library.
• 🗜️ Log compaction — snapshotting + chunked InstallSnapshot keep the log bounded and catch up lagging followers.
• 🔀 Dynamic membership — add/remove nodes at runtime via joint consensus (Raft §6) with dual-majority safety, no restart.
• 🎯 Linearizable reads — opt-in ReadIndex barrier (?consistency=strong); fast, eventually-consistent local reads by default.
• 🧾 Tamper-evident audit trail — the replicated log is hash-chained; altering any past entry breaks the chain.
• 🛰️ Edge read replicas — a read-only Node/browser SDK tails the committed log over SSE and serves reads locally (ADR-0023).
• 🔌 Pluggable application — bring your own deterministic StateMachine; the library book service is just the worked demo.
• 📈 Built-in platform — structured logs, Prometheus metrics (incl. consensus signals), tracing, idempotency, and durable crash-consistent persistence.

Contents

• Why no database?
• Architecture
• Bring your own state machine
• API
• Edge read replicas
• Built-in platform concerns
• Running a cluster / Operations
• Tests
• Limitations
• Design docs: Philosophy · ADRs · Context · Operations

Why no database?

A shared MongoDB would re-centralize the system — every node would depend on one store. Instead, each node keeps its own in-memory copy of the state. Raft guarantees every node applies the same committed commands in the same order, so all copies converge to identical state. Lose a node and the cluster keeps serving; elect a new leader and writes continue.

Architecture

                 client
                   │  (writes go to the leader; reads to any node)
        ┌──────────┼───────────┐
        ▼          ▼           ▼
     ┌──────┐  ┌──────┐    ┌──────┐
     │node1 │  │node2 │    │node3 │      each node = Express HTTP server
     │leader│◄►│follwr│◄──►│follwr│      peers talk Raft RPCs over HTTP
     └──┬───┘  └──┬───┘    └──┬───┘
        │         │           │
   replicated  replicated  replicated    BookStateMachine (in-memory, per node)
      log         log         log

Layer	File(s)	Responsibility
Raft node	src/consensus/raftNode.ts	Leader election + log replication (Raft fig. 2); generic over the application
State machine contract	src/consensus/stateMachine.ts	The StateMachine<C, T> interface an application implements
Replicated wrapper	src/consensus/replicatedStateMachine.ts	Adds audit hash-chain + idempotency over any application state machine
Transport	src/consensus/transport.ts	HttpTransport (real network) / LocalTransport (in-process tests)
Persistence	src/consensus/storage.ts	Durable term/vote/log + snapshots (file or in-memory)
Platform	src/platform/	Logging, metrics, request-context/tracing, leader-forwarding
Example app	src/models/book*.ts	The book domain: a sample StateMachine + its command builders
HTTP API (example)	src/controllers, src/routes, src/app.ts	Adapts REST ↔ consensus for the book app
Entry point	src/server.ts	Wires a node (with the book state machine) from env and starts it
Library surface	src/index.ts	Public exports for embedded-library use

The application is pluggable (ADR-0017). The consensus core knows nothing about books: an application supplies a StateMachine<C, T> (its domain command union C and result payload T) and plugs it in with new RaftNode({ stateMachine: myStateMachine, … }). The book service is just the worked example — swap it for payments, inventory, a feature-flag store, etc., and inherit replication, audit, idempotency, and observability for free.

Key design point: all non-deterministic values (e.g. a book id, borrow timestamps) are generated by the leader before a command enters the log, so every node applies an identical command and stays consistent. Your apply must be deterministic.

Bring your own state machine

import { RaftNode, HttpTransport, StateMachine, ApplyResult } from './src';

type CounterCommand = { type: 'INCR'; by: number };

class CounterStateMachine implements StateMachine<CounterCommand, number> {
    private n = 0;
    apply(cmd: CounterCommand): ApplyResult<number> {
        this.n += cmd.by;
        return { status: 200, data: this.n };
    }
    snapshot() { return this.n; }
    restore(data: unknown) { this.n = (data as number) ?? 0; }
    size() { return 1; }
}

const node = new RaftNode(
    { id: 'node1', peers: [], stateMachine: new CounterStateMachine() },
    new HttpTransport(),
);
node.start();
// later, on the leader: await node.submit({ type: 'INCR', by: 5 });

📚 Background reading: the project's core philosophy and the architecture decision records explain why the system is built this way.

API

Method	Route	Notes
GET	/books	List all books (local replica; ?consistency=strong for a linearizable read)
GET	/books/:id	Get one book (?consistency=strong for a linearizable read)
POST	/books	Add a book (write — forwarded to leader)
PUT	/books/:id	Update provided fields (write)
DELETE	/books/:id	Delete a book (write)
PUT	/books/borrow/:id	Borrow (decrement copies) (write)
PUT	/books/return/:id	Return (increment copies) (write)
GET	/audit	Replicated, hash-chained audit trail (?actor=, ?type=)
GET	/audit/verify	Verify the audit hash chain is intact
GET	/metrics	Prometheus metrics (raft + HTTP)
GET	/health	Liveness + node status
GET	/ready	Readiness (200 once a leader is known, else 503)
GET	/raft/stream	Server-Sent Events stream of the committed log (?fromIndex=N) — for edge read replicas (ADR-0023)
GET	/raft/status	Node role, term, leader, log/commit indices, members
GET	/raft/members	Current cluster configuration (voting members)
POST	/raft/members	Add a voting node { id, url } (leader-routed)
DELETE	/raft/members/:id	Remove a voting node (leader-routed)

A write can be sent to any node: a follower transparently forwards it to the leader. If no leader is currently known it returns 421 with { "leader": "<id>" } so the client can retry.

Reads are served from the local replica by default (fast, available on any node, eventually consistent). Add ?consistency=strong (or header X-Consistency: strong) for a linearizable read: it goes through the leader's ReadIndex barrier (leadership confirmed via a heartbeat quorum) so the response reflects every write committed before the read — at the cost of one round-trip, and it fails with 421 if the leader can't confirm a quorum. See ADR-0014.

Edge read replicas (ADR-0023)

Read serving can fan out past the cluster: GET /raft/stream?fromIndex=N is a Server-Sent Events feed of the committed log (snapshot handoff → committed tail → live tail), served from any node. The EdgeReplica SDK (src/edge/) tails it in Node or the browser, applies committed commands to a local copy of the same StateMachine, and answers reads from memory with no round-trip — updating live as writes commit. It is a read-only, non-voting learner (it never votes, acks, or writes). Read-your-writes is available via EdgeReplica.waitForIndex(i). See the worked example (browser + Node) under examples/edge-replica/. Scoped/partial replication and per-client authorization (the prerequisite for multi-tenant use) are future work — see ADR-0023.

Built-in platform concerns

These cross-cutting backend concerns live in src/platform/ and are wired into the consensus core, so any app built on it inherits them for free.

Observability

• Structured logs (logger.ts) — one JSON object per line, auto-tagged with node id, role/term, and the request's requestId/actor. LOG_FORMAT=pretty for human-readable local output.
• Metrics (metrics.ts) — /metrics in Prometheus format, exposing both HTTP signals (http_requests_total, http_request_duration_ms) and consensus signals you don't normally get: raft_is_leader, raft_term, raft_commit_index, elections count, raft_replication_lag per follower, raft_cluster_size, state_machine_entries (application entity count), and raft_read_barriers_total (linearizable reads served).
• Tracing (requestContext.ts) — an inbound/generated X-Request-Id is propagated through HTTP → log → committed command via AsyncLocalStorage, so a single write is correlatable across every node.

Fault tolerance

• Persistence (storage.ts) — currentTerm, votedFor, the log, and snapshots are written to disk (atomic rename) and reloaded on restart, so a crash can't violate Raft safety. Tests use an in-memory implementation.
• Log compaction (snapshotting) — once the in-memory log passes SNAPSHOT_THRESHOLD, a node snapshots its state machine and discards the covered log entries, so the log stays bounded. A follower that has fallen behind the leader's snapshot is caught up with an InstallSnapshot RPC.
• Idempotency — every write carries a requestId; a replayed id returns the original result without re-applying, turning at-least-once client retries into exactly-once effects. The receiving node assigns the id (honouring an inbound X-Request-Id) and echoes it on the response before forwarding, so a client that reuses the returned X-Request-Id on retry gets exactly-once semantics even if a forward times out and returns 421. The dedup cache is bounded (DEDUP_LIMIT, default 10,000) with deterministic FIFO eviction, so it never grows without limit.
• Dynamic membership — add or remove nodes at runtime via joint consensus (Raft §6 / ADR-0022) through POST/DELETE /raft/members, with no cluster restart. A change transitions C-old→C-new through a joint configuration in which every decision needs a majority of both sets (so even an arbitrary, non-overlapping change is safe); the configuration replicates as a log entry, a joining node catches up by normal replication or an InstallSnapshot, a leader removed from the cluster steps down, and a new leader finishes a transition its predecessor left unfinished. A removed/partitioned node can't disrupt the cluster (leader stickiness).
• Leader forwarding + /ready for load-balancer health checks.

Audit — the replicated log is the audit trail. Each committed change is recorded as a hash-chained entry ({ actor, requestId, timestamp, prevHash, hash }); altering any past entry breaks the chain (GET /audit/verify), and because it's replicated across a majority it's tamper-evident cluster-wide.

Running a cluster / Operations

For a one-command 3-node Docker cluster with a Prometheus + Grafana metrics stack, membership-change recipes, strong-read examples, and how the fault-injection (chaos) test maps to real failure modes, see docs/OPERATIONS.md:

docker compose up --build   # nodes on :3001–:3003, Prometheus :9090, Grafana :3000

Running a cluster locally

Each node is configured entirely by environment variables (see .env.example):

yarn install
yarn build   # compiles to dist/

# terminal 1
NODE_ID=node1 PORT=3001 PEERS="node2@http://localhost:3002,node3@http://localhost:3003" node dist/server.js
# terminal 2
NODE_ID=node2 PORT=3002 PEERS="node1@http://localhost:3001,node3@http://localhost:3003" node dist/server.js
# terminal 3
NODE_ID=node3 PORT=3003 PEERS="node1@http://localhost:3001,node2@http://localhost:3002" node dist/server.js

Find the leader, add a book, then read it back from a different node:

curl -s localhost:3001/raft/status            # see who is leader
curl -s -X POST localhost:<leader>/books \
  -H 'Content-Type: application/json' \
  -d '{"title":"Raft","author":"Ongaro","publisher":"Stanford","isbn":"R-1","copies":2}'
curl -s localhost:<other-node>/books          # the book is already replicated here

Kill the leader process and watch a follower get elected (/raft/status) while the data survives.

Add or remove a node at runtime (sent to any node; forwarded to the leader):

# start node4 first (PORT=3004, PEERS pointing at the existing nodes), then:
curl -s -X POST localhost:<leader>/raft/members \
  -H 'Content-Type: application/json' \
  -d '{"id":"node4","url":"http://localhost:3004"}'
curl -s localhost:3001/raft/members           # node4 is now a voting member
curl -s -X DELETE localhost:<leader>/raft/members/node4

Environment variables

Var	Default	Meaning
NODE_ID	node1	Unique id for this node
PORT	3000	HTTP port
PEERS	""	Initial other nodes as id@url CSV (membership is dynamic after start)
ADVERTISE_URL	http://localhost:$PORT	Address peers use to reach this node (in membership configs)
ELECTION_MIN_MS / ELECTION_MAX_MS	150 / 300	Election timeout window
HEARTBEAT_MS	50	Leader heartbeat interval
SNAPSHOT_THRESHOLD	1000	Compact the log after this many in-memory entries
DEDUP_LIMIT	10000	Max remembered requestIds for idempotency (FIFO eviction)
DATA_DIR	./data	Where durable state/snapshots are written
LOG_LEVEL / LOG_FORMAT	info / json	Logging verbosity / format (pretty)

Tests

yarn test

• tests/consensus.test.ts — 3-node in-process cluster: single-leader election, write replication to every node, log convergence, and leader failover.
• tests/bookApi.test.ts — the REST API against a single-node cluster.
• tests/platform.test.ts — audit hash-chain + tamper detection, idempotency (incl. bounded dedup cache), persistence across restart, a linearizable read, and the /audit /metrics /ready endpoints.
• tests/snapshot.test.ts — log compaction, InstallSnapshot catch-up of a lagging follower, and snapshot restore on restart.
• tests/readBarrier.test.ts — the ReadIndex linearizable-read barrier: serves on a healthy leader, refuses when a quorum can't be confirmed, rejects on a follower.
• tests/membership.test.ts — dynamic membership: add a node and watch it catch up and join the quorum, remove a follower, a self-removing leader steps down, and invalid changes are rejected.
• tests/crashConsistency.test.ts — recovery when a crash lands the snapshot but not the compacted log, InstallSnapshot no-rollback, and durable-boundary snapshot transfer.
• tests/logBacktracking.test.ts — accelerated conflict-hint backtracking and reconciliation of a divergent follower log.

Tests use LocalTransport, so they need no sockets and no database.

Limitations (it's a POC)

• Membership changes still apply one transition at a time, and a joining node has no separate non-voting catch-up phase, so adding a far-behind node while another is down can briefly stall commits. (Changes now use joint consensus — Raft §6 / ADR-0022 — so the single-server restriction is gone; arbitrary changes are safe.)
• Reads default to the local replica (eventually consistent); linearizable reads are available opt-in via ?consistency=strong. These can be served by the leader or offloaded to a follower via a ReadIndex RPC (Raft §6.4), but there are no time-based leader leases (every barrier confirms a fresh heartbeat quorum).
• The audit log grows unbounded (kept in full inside snapshots by design). The idempotency cache is bounded (DEDUP_LIMIT, FIFO), so a retry older than the window re-applies rather than being deduped.