Skip to content

feat(orch): disk-pressure guard so a full disk can't wedge the orchestrator#73

Merged
nathanwhit merged 1 commit into
mainfrom
disk-guard
Jul 1, 2026
Merged

feat(orch): disk-pressure guard so a full disk can't wedge the orchestrator#73
nathanwhit merged 1 commit into
mainfrom
disk-guard

Conversation

@nathanwhit

Copy link
Copy Markdown
Owner

Why

The orch host is itself a scheduler placement target (placed on local), so worker/PR checkouts pile up under the work root on the same volume as orcha.db. When that volume hit 100%, SQLite writes failed (SQLITE_FULL) and the whole orchestrator went silent — a total outage (an assigned issue sat un-triggered for hours because the poller couldn't write its row), not one degraded worker.

Two leaks fed the fill:

  • A closed PR's multi-GB pr_branch checkout is only reclaimed once the whole objective goes terminal — it stayed active for hours after the PR closed.
  • The "shared mirror stays small" assumption in reclaim.go was false: the bare mirror stood at 21G.

What

Four fixes, all mirroring the existing load-gate pattern:

  • Disk guard — the per-target health probe also reads df free-space on the work root (SetTargetDisk stores free_disk_mb/disk_probed_at); SelectTarget skips a target below MinFreeDiskMB (-min-free-disk-gb, default 10) so work parks waiting_capacity instead of filling the volume. Fails open on stale/missing samples, like load.
  • Watchdog — a below-floor probe kicks an immediate ReclaimWorkspaces and emits a cooldown-deduped, push-notified disk_pressure alert.
  • Closed-PR reclaim — reclaim pr_branch checkouts whose PR is merged/closed even mid-active-objective, path-aware so a dir a live session shares is never yanked. Deliberately not general mid-objective pr_branch reclaim — a follow-up's un-pushed commit lives only on that checkout (the reviewer-commit loss class); the terminal-PR gate is the safe signal.
  • Mirror gc — throttled (6h/target) git gc on each target's .orcha-cache so the bare mirror stops growing unbounded.

Default-on at 10GB, so protection activates on the next deploy without changing the prod launch command.

Deferred

The structural decoupling — don't place workers on the orch host, or split the DB and work roots onto separate volumes — is the real long-term fix, left as a follow-up rather than bundled into an incident fix.

Test plan

  • go build ./... && go vet ./... && go test ./... — all pass.
  • New unit tests: df parse (TestParseFreeDiskMB), placement gate + all-pressured backpressure + staleness fail-open + disabled (TestSelectTarget_*Disk*), store round-trip + gate (TestSetTargetDisk_RoundTripAndGate), and path-aware closed-PR reclaim including a finished row sharing a dir with a running follow-up (TestReclaimWorkspaces_ClosedPRBranchCheckout).

…trator

The orch host is itself a scheduler placement target, so worker/PR checkouts
accumulate under the work root on the same volume as orcha.db. When that volume
hit 100%, SQLite writes failed (SQLITE_FULL) and the whole orchestrator went
silent — a total outage rather than one degraded worker (an assigned issue sat
un-triggered for hours because the poller couldn't write its row). Two leaks fed
it: a closed PR's multi-GB pr_branch checkout is only reclaimed once the whole
objective goes terminal (it stayed active for hours), and the "shared mirror
stays small" assumption was false — the bare mirror stood at 21G.

Four fixes, all mirroring the existing load-gate pattern:

- Disk guard: the per-target health probe also reads df free-space on the work
  root (SetTargetDisk stores free_disk_mb/disk_probed_at); SelectTarget skips a
  target below MinFreeDiskMB (-min-free-disk-gb, default 10) so work parks
  waiting_capacity instead of filling the volume. Fails open on stale/missing
  samples, like load.
- Watchdog: a below-floor probe kicks an immediate ReclaimWorkspaces and emits a
  cooldown-deduped, push-notified disk_pressure alert.
- Closed-PR reclaim: reclaim pr_branch checkouts whose PR is merged/closed even
  mid-active-objective, path-aware so a dir a live session shares is never
  yanked. Deliberately not general mid-objective pr_branch reclaim — a
  follow-up's un-pushed commit lives only on that checkout (the reviewer-commit
  loss class); the terminal-PR gate is the safe signal.
- Mirror gc: throttled (6h/target) git gc on each target's .orcha-cache so the
  bare mirror stops growing unbounded.

Deferred: the structural decoupling (don't place workers on the orch host, or
split the DB and work roots onto separate volumes) — the real long-term fix.

Tests: disk parse/gate/staleness + closed-PR path-aware reclaim.
@nathanwhit nathanwhit merged commit e37eb66 into main Jul 1, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant