Split-Brain Lab

An interactive observability experiment about a cluster where every node passes its health check while two incompatible authorities act at once.

A five-node cluster splits into a two-node island containing the old leader and a three-node majority that elects a new term. Process health remains 100% on both sides. Local liveness therefore cannot tell an operator which history is valid, and a quorum-safe replicated log still cannot stop stale work against an external system unless that system checks a fencing token.

The deterministic model and HTTP server are written in Scala 3, adding another JVM language to the telemetry.sh labs. The interactive client is dependency-free HTML, CSS, and JavaScript.

The four authority models

• Local health only lets each reachable side treat a live process as authority. The minority and majority both accept writes after the new election.
• Lease timeout bounds the old leader's authority using local time, but creates an overlap between the new election and the old lease expiry.
• Quorum, no fence keeps the replicated log consistent while an old external worker continues sending charges, emails, or object writes.
• Quorum + fencing carries the new epoch to every side effect, allowing downstream systems to reject stale term-41 work after term 42 wins.

Every strategy sees the same asymmetric partition and client routing. The experiment reports accepted and rejected writes, divergent history, stale external effects, simultaneous leaders, token rejections, and recovery actions.

Run it

Requirements:

• Scala CLI 1.15 or newer
• JDK 21
• jq, curl, and Node.js for checks

make run

Open http://localhost:8080. Moving a control recomputes all four authority models through the Scala API.

Run the native suite:

make check

Or run the production container:

docker build -t split-brain-lab .
docker run --rm -p 8080:8080 split-brain-lab

The multi-stage image packages a self-contained Scala assembly, copies it into a JRE 21 Alpine runtime, and runs as the unprivileged lab user.

Query the model

curl --silent \
  'http://localhost:8080/api/simulate?nodes=5&minority_nodes=2&write_rps=120' |
  jq '.strategies[] | {
    label,
    availability_percent,
    divergent_writes,
    stale_side_effects,
    stale_token_rejections,
    recovery_actions
  }'

For a machine-readable default run without starting the server:

scala-cli run project.scala src/main/scala --server=false \
  --main-class lab.Main -- --json | jq

What to look for

The default incident begins 30 seconds into a 210-second run. A two-node island keeps 40% of client traffic while a three-node majority elects term 42 after 10 seconds. The link heals after 120 seconds.

1. Every process remains healthy. process.up = 1 does not prove that a node can see a quorum or that its term is current.
2. Local-health authority creates two histories. The losing island accepts thousands of writes that must be rolled back or reconciled after heal.
3. A lease reduces, but does not eliminate, ambiguity. If the majority election finishes before the old lease expires, both leaders act during the overlap.
4. Quorum protects only the system that participates in quorum. An old worker can still perform an external effect when the downstream service cannot compare terms.
5. Fencing exports the consensus decision. The downstream system rejects every stale epoch even though the old process is alive and still trying.

Increase minority traffic or partition length to expand the losing history. Increase lease time to widen the two-leader overlap. Raise external effects to show how a perfectly safe consensus log can coexist with unsafe work outside that log.

Why telemetry.sh

No node-local graph proves or disproves split brain:

• process health establishes liveness, not authority;
• visible peer count shows which island can reach the commit boundary;
• cluster term exposes simultaneous leaders from different epochs;
• accepted writes by node and term reveal divergent histories;
• external actions by fencing epoch show whether consensus reaches downstream;
• reconciliation and compensation events quantify the recovery bill.

telemetry.sh makes node health, network reachability, consensus state, application writes, and downstream side effects queryable in one investigation. Instead of trusting five green dots, an operator can align the partition, the new term, stale actions, token rejections, and the exact work that will need repair.

Project structure

src/main/scala/lab/Model.scala     deterministic partition and authority model
src/main/scala/lab/Json.scala      dependency-free JSON encoder
src/main/scala/lab/Main.scala      JDK HTTP server, API, static assets, and CLI
src/test/scala/lab/ModelTest.scala model and sanitization invariants
public/                            responsive interactive investigation
tests/http-test.sh                 API, assets, methods, and security headers
tests/container-test.sh            production image, CLI, user, and health checks
.github/workflows/ci.yml

Safety

The workload is synthetic. It does not connect to a real cluster, isolate nodes, perform writes, need credentials, or ingest telemetry.

References

• In Search of an Understandable Consensus Algorithm (Raft)
• Scala and JDK compatibility
• Scala CLI assembly packaging

License

MIT