Skip to content

feat(hub): install the control box from the npm package, keep --ref for source#258

Merged
madarco merged 6 commits into
nightlyfrom
feat/hub-deploy-from-package
Jul 26, 2026
Merged

feat(hub): install the control box from the npm package, keep --ref for source#258
madarco merged 6 commits into
nightlyfrom
feat/hub-deploy-from-package

Conversation

@madarco

@madarco madarco commented Jul 26, 2026

Copy link
Copy Markdown
Owner

Why

agentbox hub deploy hetzner cloned the monorepo onto the VPS and built 14 workspace packages plus the CLI there — ~2m40s of a ~4m45s deploy — to produce an artifact the published package already carries: runtime/hub/apps/hub/server.js, the same standalone bundle agentbox hub spawns locally (resolveHubServer, staged by stage-runtime.mjs).

Building from a ref also couples the VPS artifact to a branch name rather than a version. That coupling is what produced the 502 fixed last week: DEFAULT_DEPLOY_REF was main while the CLI was a nightly, so main's hub listened on :3000 behind Postgres while the CLI wrote reverse_proxy app:8787.

What changed

The VPS now runs npm i -g @madarco/agentbox@<the deploying CLI's exact version> in a small node:22-slim image, and the host ships the compose stack: apps/hub/docker-compose.yml verbatim plus docker-compose.package.yml, an override that replaces only the build: block. There is no repo on the VPS at all — cloud-init no longer clones.

Sharing the base compose verbatim is deliberate: the service's environment:/volumes:/ports: block keeps exactly one definition, so the two modes cannot drift.

Consequences worth calling out:

  • The version-skew bug class becomes structurally impossible in the default mode — the version is the contract.
  • Base-image fingerprints now match by construction. The control box's runtime/ tree is literally the same npm tarball the PC installed, rather than a ref that happens to correspond. (AGENTBOX_RUNTIME_ROOT still has to be set, and now AGENTBOX_CLI_RUNTIME_DIR too — a container has no parent CLI process to inherit it from.)
  • NODE_ENV=production is load-bearing in the image: without it Next takes the dev path and dies with Couldn't find any `pages` or `app` directory.

--ref/--repo keep the clone-and-build path (a dev CLI, whose version was never published, falls back to it automatically); --package <spec> pins a different npm spec. deploy.json now records which build a control box runs — nothing else on the host or the VPS did — and hub status shows it.

Also in here, both prompted by the live test:

  • diagnoseUnhealthy no longer reports a rate-limited certificate as "Caddy can't reach it, so the upstream port is wrong". It greps Caddy's log and names the real cause.
  • --domain <host> on hub deploy hetzner (the deploy already accepted a domain; the CLI never exposed it) — the way out of a Let's Encrypt rate limit, since an sslip.io name is derived from a recyclable IP.

Verification

Unit: resolveHubDeploySource, the asset resolver, cloud-init with/without the clone, and a drift guard that the shipped compose still yields hubContainerPort === 8787 / isFullHubCompose. Full pnpm build, typecheck, lint, and the web build all green. npm pack --dry-run confirms runtime/hub-deploy/** ships.

Live on Hetzner:

  • Package mode — image built in ~60s (vs ~160s of pnpm builds), containers up 2m56s after start. /opt/agentbox holds no repo clone, only the six host-shipped files. In-container: version matches the deploying CLI exactly, gh 2.96.0 present, all four AGENTBOX_* pointers resolve under /opt/agentbox-cli, every provider's staged-runtime marker resolves (hetzner/scripts/install-box.sh, e2b/scripts/build-template.sh, vercel/scripts/provision.sh, digitalocean/…, _shared/ctl.cjs, docker/Dockerfile.box), and /healthz returns {"ok":true,"ui":true,"cliEntry":true}.
  • --ref nightly regression — fully green end to end: healthy over public HTTPS, relay.controlPlaneUrl wired, and the e2b bake shared with the control box.

One caveat, external to this change: two package-mode runs landed on a recycled Hetzner IP that had already hit Let's Encrypt's 5-certs-per-hostname-per-week cap, so their public HTTPS never came up (the hub itself was healthy on 127.0.0.1:8787 throughout). That is what prompted the diagnostic fix and --domain. The --ref run on a fresh IP proves the shared Caddy path, which package mode does not modify.


Note

Medium Risk
Changes the default production deploy path and VPS provisioning (cloud-init, compose files, env wiring); mistakes could break new control-box installs, though source mode and extensive tests preserve the escape hatch.

Overview
Hetzner control-box deploys now install @madarco/agentbox from npm (pinned to the deploying CLI’s version) instead of cloning and building the monorepo on the VPS. Package mode scp’s a small compose stack (docker-compose.yml + docker-compose.package.yml + Dockerfile.package) from runtime/hub-deploy/ (staged by stage-runtime.mjs); cloud-init skips the git clone. --ref / --repo still use the old clone-and-build path; dev builds (0.0.0) auto-fall back to source. resolveHubDeploySource centralizes that choice; CLI flags add --package and expose --domain on hub deploy hetzner.

Observability and ops: deploy.json records HubDeploySource; hub status shows the remote build line. Failed HTTPS health checks use an on-VPS curl probe (describeCaddyHop) so cert/rate-limit issues aren’t misreported as wrong Caddy upstream ports; recovery hints use the correct compose -f list per mode.

Images: New Dockerfile.package global-installs the CLI and runs the staged hub bundle; source Dockerfile gains AGENTBOX_DOCKER_CONTEXT. Docs updated for default npm deploy, flags, and troubleshooting.

Reviewed by Cursor Bugbot for commit 230d94e. Configure here.

madarco added 5 commits July 26, 2026 09:41
…or source

`agentbox hub deploy hetzner` cloned the monorepo onto the VPS and built 14
workspace packages plus the CLI there — ~2m40s of a ~4m45s deploy — to produce
an artifact the published package already carries: `runtime/hub/apps/hub/server.js`,
the same standalone bundle `agentbox hub` spawns locally.

It now installs `@madarco/agentbox` pinned to the deploying CLI's exact version.
The host ships the compose stack (the base docker-compose.yml verbatim, plus an
override that swaps only the build block), so there is no repo on the VPS at all
and no version-skew surface — that coupling is what made a nightly CLI's deploy
502 against a `main` hub listening on a different port.

It also fixes base-image fingerprint parity by construction: the control box's
runtime/ tree is now the same npm tarball the PC installed, rather than a ref
that happens to match.

`--ref`/`--repo` keep the clone-and-build path for deploying unreleased code, and
a dev build (whose version was never published) falls back to it automatically.
`--package <spec>` pins a different npm spec. deploy.json now records which build
a control box is running, and `hub status` shows it.
… port

A hub answering on the VPS but not over HTTPS was reported as one guess —
"Caddy cannot reach it, so its upstream port does not match" — which is wrong
whenever the certificate is the problem. Hit live: sslip.io derives the hostname
from the IP, so a recycled Hetzner address arrived already at Let's Encrypt's
5-certs-per-identifier-per-week cap, and the deploy blamed a port that was
perfectly correct.

diagnoseUnhealthy now greps Caddy's own log when the hub is healthy locally, and
names the actual cause — including the fix for the rate-limited case (redeploy
for a different IP, or pass --domain).
…ntrol

The sslip.io default derives the hostname from the IP, so it isn't really yours
and Hetzner reuses released addresses — landing on one that already had five
certificates issued this week leaves a healthy hub with no usable HTTPS until
the 7-day window rolls. --domain is the way out (and what the new rate-limit
diagnostic points at); the underlying deploy already accepted a domain, only the
CLI never exposed it.
MDX parses the attribute value as JSX, so \" is an invalid attribute name
character, not an escaped quote — `next build` failed to evaluate the page.
@vercel

vercel Bot commented Jul 26, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated (UTC)
agentbox-web Skipped Skipped Jul 26, 2026 9:26am

Request Review

Comment thread packages/sandbox-hetzner/src/control-plane-deploy.ts Outdated
@madarco

madarco commented Jul 26, 2026

Copy link
Copy Markdown
Owner Author

bugbot run

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

1 issue from previous review remains unresolved.

Fix All in Cursor

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 0563cca. Configure here.

… text

Bugbot caught that the certificate check matched Caddy's SUCCESS lines:
`tls.obtain` appears in "obtaining certificate", "lock acquired", and
"certificate obtained successfully", so a healthy cert plus a genuine upstream
mismatch would have been reported as a certificate problem.

The verdict now comes from an HTTPS request made ON the VPS (--resolve pinned to
loopback), which exercises the real certificate and the real reverse_proxy
upstream: 200 means the break is outside the box (DNS / :443 rule), 000 means the
TLS handshake never completed, any other status means Caddy answered and the
upstream is wrong. The log is read only for the rate-limit detail, which has an
unambiguous error signature.

Split out as the pure `describeCaddyHop` so the three-way verdict is unit-tested,
including the exact log that fooled the old check.
@madarco

madarco commented Jul 26, 2026

Copy link
Copy Markdown
Owner Author

bugbot run

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 230d94e. Configure here.

@madarco
madarco merged commit 230d94e into nightly Jul 26, 2026
5 checks passed
@github-actions github-actions Bot locked and limited conversation to collaborators Jul 26, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant