Skip to content

feat(deploy_tee): genesis-ceremony polls for cohort readiness #55

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
samlaf merged 1 commit into from
Jul 7, 2026
Merged

feat(deploy_tee): genesis-ceremony polls for cohort readiness #55

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 32 additions & 19 deletions deploy_tee/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,8 +30,8 @@ for its [InitConfig](https://github.com/SeismicSystems/enclave/blob/seismic/crat
VHD upload to the Azure image registry is a seismic-images concern
(`make push-azure-*`), not this tool.
- **For `seismic-tee-bootstrap genesis-ceremony`:** the `genesis` binary (built from
summit) on PATH, plus a summit genesis template passed via
`--summit-template`.
summit) on PATH. (The summit genesis template defaults to the
artifact-set copy beside `--manifest`.)
- **For `seismic-tee-bootstrap up`** (cohort provisioning): the `pulumi`
CLI on PATH. The passphrase secrets provider needs
`PULUMI_CONFIG_PASSPHRASE` — set it, or let `up`/`down` prompt you on
Expand Down Expand Up @@ -77,6 +77,13 @@ small JSON (`public_ip`/`fqdn`) describing one provisioned node — see
(just `{public_ip, fqdn}`); a BYO-infra operator can hand-write it (or use
`pulumi stack output --json` — only those two keys are read).

Cohort provisioning is one Pulumi stack **per node** (not one cohort
stack), driven as a *local* Automation-API program: any node can be
recycled or torn down without touching the others, and every stack stays
manageable with the plain `pulumi` CLI (`pulumi stack output` / `up` /
`destroy --stack …`) — the orchestrator and the CLI are interchangeable
front-ends over the same state.

### Where this is heading: a public operator repo

Longer-term these two CLIs are intended to live in **two repos**, split on
Expand Down Expand Up @@ -162,19 +169,21 @@ uv run seismic-tee-bootstrap manifest assemble deploy_tee/networks/devnet-3
uv run seismic-tee-bootstrap up --count 2 --network deploy_tee/networks/devnet-3
# → deploy_tee/networks/devnet-3/nodes/dev-bootstrap-node-{1,2}.json

# 3. Configure each node against the shared manifest and POST. Node 1 founds
# the network (genesis, via the bootstrap CLI); node 2 joins (operator CLI),
# fetching root_key from node 1.
# 3. Configure the whole cohort in one command: exactly one genesis node
# (mints root_key) plus N joiners pointed at it (each fetches root_key
# from node 1 over the attested handshake); every first-boot disk wipe
# is watched in parallel. (A node joining the live network later uses
# `seismic-tee configure` instead — see "Joining an existing network".)
uv run seismic-tee-bootstrap configure \
--genesis deploy_tee/networks/devnet-3/nodes/dev-bootstrap-node-1.json \
--manifest deploy_tee/networks/devnet-3/network-manifest.json
uv run seismic-tee configure \
--node deploy_tee/networks/devnet-3/nodes/dev-bootstrap-node-2.json \
--peer deploy_tee/networks/devnet-3/nodes/dev-bootstrap-node-1.json \
--join deploy_tee/networks/devnet-3/nodes/dev-bootstrap-node-2.json \
--manifest deploy_tee/networks/devnet-3/network-manifest.json

# 4. Run the genesis ceremony once: builds genesis.toml from the cohort
# (needs the `genesis` binary on PATH) and fans it out to every summit.
# Waits for cohort readiness itself — polls until each node's reth
# serves the manifest's genesis hash as block 0 and its summit pubkeys
# are readable — so it can run straight after step 3.
# One-shot: send_genesis only exists in the pre-genesis node API, so this
# can never be re-run against a live network. --summit-template defaults
# to the artifact-set copy beside --manifest (verified against the
Expand Down Expand Up @@ -227,7 +236,10 @@ back as 4xx and the deploy script raises.
`configure` POSTs this body verbatim to the VM's
`http://<ip>:8080/`, retrying for 180s while the listener comes
up (LUKS format + mkfs + TPM enroll takes ~20–40s on first
boot). On success tdx-init writes:
boot). tdx-init is a one-shot listener: once a boot's POST is
consumed, `:8080` stays down until the next reboot — "listener
never came up" against an already-configured node means *reboot it
first*, not that the node is broken. On success tdx-init writes:

| File | Content | Consumer |
| ----------------------------------------- | --------------------------------------------------------------- | --------------------------- |
Expand All @@ -242,13 +254,15 @@ only gates against a second POST within the same boot.

### Operator footguns to be aware of

- **Running `seismic-tee-bootstrap configure` against more than one VM
silently splits the network.** Each genesis enclave generates a fresh
`OsRng` `root_key` locally; downstream nodes that fetch from one cohort
can't decrypt state from the other. There is no in-band check yet — the
planned parallel cohort founder assigns exactly one genesis, making this
unrepresentable. (The operator `seismic-tee configure` can't trigger it: it
only joins.)
- **Two root_key-genesis nodes silently split the network.** Each genesis
enclave generates a fresh `OsRng` `root_key` locally; downstream nodes
that fetch from one cohort can't decrypt state from the other.
`seismic-tee-bootstrap configure` makes this unrepresentable within one
invocation (exactly one `--genesis`), but nothing yet stops a second
*invocation* against a fresh VM — a durable guard (genesis mode refusing
to start against an already-formatted volume or a live peer) is planned
upstream in tdx-init. (The operator `seismic-tee configure` can't
trigger it: it only joins.)
- **The `:8080` listener is unauthenticated, first-POST-wins.**
An attacker who reaches port 8080 ahead of the operator can
write a malicious `peers` list, redirecting `root_key` fetch to
Expand All @@ -268,8 +282,7 @@ serves that RA-TLS endpoint, and cvm-reverse-proxy's pre-handshake aTLS
is a different protocol than the `flashbots/attested-tls` the enclave
standardizes on. Verification will be reimplemented against attested-tls
/ `seismic-attestation` (the enclave already exposes
`getAttestationEvidence` plus a deploy-preflight binding). Tracked in
`plans/tee`.
`getAttestationEvidence` plus a deploy-preflight binding).

When it returns, the design is **bring-your-own measurements**: the
operator supplies the image's expected measurements (published by
Expand Down
187 changes: 136 additions & 51 deletions deploy_tee/genesis.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,11 @@
booted from a stale image or wrong genesis fails the ceremony loudly
instead of parking summit in SYNCING forever.

The ceremony doubles as the cohort barrier: its two probes (reth block 0,
summit `getPublicKeys`) answer only after a node finishes root_key → LUKS
open → summit keygen, and both are polled until every node responds — so
it can run straight after `configure` without hand-timing the boot tail.

Each node is located by a descriptor file (see deploy_tee/descriptor.py),
so this never calls Pulumi.
"""
Expand All @@ -23,18 +28,27 @@
import shutil
import subprocess
import tempfile
import time
from pathlib import Path

import requests

from deploy_tee import manifest as manifest_mod
from deploy_tee.descriptor import load_descriptor, require
from deploy_tee.utils.summit_client import SummitClient
from deploy_tee.utils.summit_client import PublicKeys, SummitClient

# Summit's consensus (BLS) port. Each validator entry in the generated
# genesis.toml pins "<ip>:<CONSENSUS_PORT>".
CONSENSUS_PORT = 18551

# Cohort-readiness polling (the barrier). `configure` already watches the
# 1h+ first-boot disk wipes, so by ceremony time the residual boot tail is
# minutes; the timeout turns a wedged node into a loud per-node report
# instead of hanging the ceremony indefinitely.
POLL_INTERVAL_SECONDS = 5
READY_TIMEOUT_SECONDS = 15 * 60
WAIT_LOG_INTERVAL_SECONDS = 30

# anvil/sanvil dev accounts: the first 10 addresses from the well-known
# "test test test ... junk" mnemonic. Their private keys are public, so
# these are devnet-only — never use them as withdrawal_credentials on a
Expand Down Expand Up @@ -150,6 +164,9 @@ def _parse_args(argv: list[str] | None = None) -> argparse.Namespace:

def _get_pubkeys(
descriptors: list[Path],
*,
timeout: float = READY_TIMEOUT_SECONDS,
interval: float = POLL_INTERVAL_SECONDS,
) -> tuple[list[dict[str, str]], list[tuple[Path, SummitClient]]]:
"""Gather each cohort node's summit pubkeys from its descriptor file.

Expand All @@ -158,30 +175,62 @@ def _get_pubkeys(
infra tool produced the descriptor — this script never calls Pulumi.
Returns the validator entries and the per-node summit clients (reused
for the genesis.toml fanout).

Each node is polled until its pubkeys are readable: getPublicKeys
answers only once summit is up with generated keys (after root_key →
LUKS open → keygen), so an unreadable node is normally just still
booting. A node still unreadable at `timeout` aborts the ceremony with
a per-node report.
"""
validators = []
node_clients: list[tuple[Path, SummitClient]] = []
for i, path in enumerate(descriptors):
ip_addresses: dict[Path, str] = {}
for path in descriptors:
descriptor = load_descriptor(path)
fqdn = require(descriptor, "fqdn", path)
ip_address = require(descriptor, "public_ip", path)
client = SummitClient(f"https://{fqdn}/summit")
node_clients.append((path, client))
try:
pubkeys = client.get_public_keys()
validators.append(
{
"node_public_key": pubkeys.node,
"consensus_public_key": pubkeys.consensus,
"ip_address": f"{ip_address}:{CONSENSUS_PORT}",
"withdrawal_credentials": _ANVIL_ADDRESSES[
i % len(_ANVIL_ADDRESSES)
],
}
ip_addresses[path] = require(descriptor, "public_ip", path)
node_clients.append((path, SummitClient(f"https://{fqdn}/summit")))

pubkeys: dict[Path, PublicKeys] = {}
last_error: dict[Path, str] = {}
deadline = time.monotonic() + timeout
next_log = 0.0
while True:
for path, client in node_clients:
if path in pubkeys:
continue
try:
pubkeys[path] = client.get_public_keys()
print(f" ✓ {path.stem}: summit pubkeys readable")
except Exception as e:
last_error[path] = str(e)
pending = [path for path in descriptors if path not in pubkeys]
if not pending:
break
now = time.monotonic()
if now >= deadline:
listing = "\n".join(f" ✗ {path}: {last_error[path]}" for path in pending)
raise SystemExit(
f"{len(pending)} node(s) still without readable summit pubkeys "
f"after {int(timeout)}s (stuck before keygen?); aborting the "
f"ceremony:\n{listing}"
)
except Exception as e:
print(f"Error: {e}")
raise e
if now >= next_log:
print(
"waiting for summit pubkeys: "
+ ", ".join(path.stem for path in pending)
)
next_log = now + WAIT_LOG_INTERVAL_SECONDS
time.sleep(interval)

validators = [
{
"node_public_key": pubkeys[path].node,
"consensus_public_key": pubkeys[path].consensus,
"ip_address": f"{ip_addresses[path]}:{CONSENSUS_PORT}",
"withdrawal_credentials": _ANVIL_ADDRESSES[i % len(_ANVIL_ADDRESSES)],
}
for i, path in enumerate(descriptors)
]
return validators, node_clients


Expand All @@ -208,7 +257,13 @@ def _verify_template_commitment(template_path: Path, manifest: dict) -> None:
)


def _assert_cohort_genesis_hash(descriptors: list[Path], expected: str) -> None:
def _assert_cohort_genesis_hash(
descriptors: list[Path],
expected: str,
*,
timeout: float = READY_TIMEOUT_SECONDS,
interval: float = POLL_INTERVAL_SECONDS,
) -> None:
"""Assert every cohort node's reth serves `expected` as block 0.

summit's genesis `eth_genesis_hash` must equal reth's real genesis hash:
Expand All @@ -219,45 +274,75 @@ def _assert_cohort_genesis_hash(descriptors: list[Path], expected: str) -> None:
node booted from a different image/genesis before the ceremony pins the
validator set — and doubles as a reth-readiness probe before
send_genesis.

A node that doesn't answer is polled until `timeout` — reth comes up
only after root_key → LUKS open, so early unreachability is the normal
boot tail. A *wrong* answer fails immediately: waiting can't fix a node
booted from a stale image or different genesis.
"""
# Query every node before judging, and report the full cohort on failure —
# a partial listing is ambiguous with "stopped at the first bad node", and
# which nodes match is exactly the diagnostic (one stale node vs. a
# manifest that matches nobody).
observed: dict[Path, str] = {} # block-0 hash, or an error description
urls: dict[Path, str] = {}
for path in descriptors:
descriptor = load_descriptor(path)
fqdn = require(descriptor, "fqdn", path)
url = f"https://{fqdn}/rpc" # nginx proxies /rpc -> reth :8545
try:
response = requests.post(
url,
json={
"jsonrpc": "2.0",
"id": 1,
"method": "eth_getBlockByNumber",
"params": ["0x0", False],
},
timeout=30,
)
response.raise_for_status()
data = response.json()
if data.get("result") is None:
raise RuntimeError(
f"eth_getBlockByNumber returned {data.get('error') or data}"
urls[path] = f"https://{fqdn}/rpc" # nginx proxies /rpc -> reth :8545

observed: dict[Path, str] = {} # block-0 hash, once a node has answered
last_error: dict[Path, str] = {}
deadline = time.monotonic() + timeout
next_log = 0.0
while True:
for path in descriptors:
if path in observed:
continue
try:
response = requests.post(
urls[path],
json={
"jsonrpc": "2.0",
"id": 1,
"method": "eth_getBlockByNumber",
"params": ["0x0", False],
},
timeout=30,
)
observed[path] = data["result"]["hash"]
except Exception as e:
observed[path] = f"unreachable via {url}: {e}"
if any(h.lower() != expected.lower() for h in observed.values()):
response.raise_for_status()
data = response.json()
if data.get("result") is None:
raise RuntimeError(
f"eth_getBlockByNumber returned {data.get('error') or data}"
)
observed[path] = data["result"]["hash"]
except Exception as e:
last_error[path] = f"unreachable via {urls[path]}: {e}"
pending = [path for path in descriptors if path not in observed]
mismatch = any(h.lower() != expected.lower() for h in observed.values())
if mismatch or not pending:
break
now = time.monotonic()
if now >= deadline:
break
if now >= next_log:
print(
"waiting for reth block 0: " + ", ".join(path.stem for path in pending)
)
next_log = now + WAIT_LOG_INTERVAL_SECONDS
time.sleep(interval)

if mismatch or pending:
# Report the full cohort, not just the first bad node — a partial
# listing is ambiguous with "stopped at the first bad node", and
# which nodes match is exactly the diagnostic (one stale node vs. a
# manifest that matches nobody).
listing = "\n".join(
f" {'✓' if h.lower() == expected.lower() else '✗'} {path}: {h}"
for path, h in observed.items()
for path, h in (
(path, observed.get(path) or last_error[path]) for path in descriptors
)
)
raise SystemExit(
"Cohort disagrees with the declared eth_genesis_hash (stale image "
"or wrong reth genesis?); refusing to build genesis.toml:\n"
f" declared: {expected}\n{listing}"
"Cohort disagrees with the declared eth_genesis_hash (stale image, "
"wrong reth genesis, or a node that never became ceremony-ready); "
f"refusing to build genesis.toml:\n declared: {expected}\n{listing}"
)


Expand Down
Loading
Loading