M19 — operational hardening: Docker cluster, observability stack, chaos tests

[naveed949]

Summary

Makes the POC runnable and demoable as a real multi-node cluster, and exercises the HTTP transport that nothing tested before. Operational only — no consensus-core changes. Builds on the merged M11–M18 work (#4).

Docker / cluster

• Multi-stage Dockerfile (build on node:20 → prod-pruned run on node:20-slim) + .dockerignore.
• docker-compose.yml: a 3-node book-service cluster wired via NODE_ID/PORT/PEERS/ADVERTISE_URL/DATA_DIR (PEERS excludes self, ADVERTISE_URL uses compose DNS names), named per-node volumes, and a /ready healthcheck.

Observability stack

• Prometheus (deploy/prometheus.yml) scraping all three nodes' /metrics — avoids a target node label since the app already emits node="<id>".
• Grafana provisioning (datasource + dashboard provider) and a raft-cluster dashboard wired to the real metric names: leadership/cluster size, commit/apply progress, elections, replication lag, HTTP throughput + p95 latency, read barriers and follower reads.

Fault-injection over the real HttpTransport

• tests/chaos.test.ts spins up a 3-node cluster on actual HTTP sockets (two-pass ephemeral-port allocation, stdlib http client, no new deps) and covers:
  1. single leader election,
  2. a replicated write readable on a different node,
  3. leader-failure recovery + a new commit through the surviving majority,
  4. partition tolerance (majority commits, isolated minority cannot, then heals and converges),
  5. a strong (ReadIndex) linearizable read.
  • Teardown stops every node and closes every server (no leaked handles under --detectOpenHandles --forceExit).

Docs

• docs/OPERATIONS.md (run the cluster, hit the API, change membership, scrape metrics, open Grafana) + a README pointer.

Verification

• npx tsc --noEmit clean.
• Chaos suite stable across repeated runs (3× here + 6× by the implementer).
• Full suite green: 25 suites / 201 tests.
• docker compose config validates and the Grafana dashboard JSON parses.
• ⚠️ An end-to-end docker compose up --build (image build + cluster boot) was not run — no Docker daemon in the sandbox. Worth a local smoke test.

🤖 Generated with Claude Code

---

Generated by Claude Code