feat(orch): disk-pressure guard so a full disk can't wedge the orchestrator#73
Merged
Conversation
…trator The orch host is itself a scheduler placement target, so worker/PR checkouts accumulate under the work root on the same volume as orcha.db. When that volume hit 100%, SQLite writes failed (SQLITE_FULL) and the whole orchestrator went silent — a total outage rather than one degraded worker (an assigned issue sat un-triggered for hours because the poller couldn't write its row). Two leaks fed it: a closed PR's multi-GB pr_branch checkout is only reclaimed once the whole objective goes terminal (it stayed active for hours), and the "shared mirror stays small" assumption was false — the bare mirror stood at 21G. Four fixes, all mirroring the existing load-gate pattern: - Disk guard: the per-target health probe also reads df free-space on the work root (SetTargetDisk stores free_disk_mb/disk_probed_at); SelectTarget skips a target below MinFreeDiskMB (-min-free-disk-gb, default 10) so work parks waiting_capacity instead of filling the volume. Fails open on stale/missing samples, like load. - Watchdog: a below-floor probe kicks an immediate ReclaimWorkspaces and emits a cooldown-deduped, push-notified disk_pressure alert. - Closed-PR reclaim: reclaim pr_branch checkouts whose PR is merged/closed even mid-active-objective, path-aware so a dir a live session shares is never yanked. Deliberately not general mid-objective pr_branch reclaim — a follow-up's un-pushed commit lives only on that checkout (the reviewer-commit loss class); the terminal-PR gate is the safe signal. - Mirror gc: throttled (6h/target) git gc on each target's .orcha-cache so the bare mirror stops growing unbounded. Deferred: the structural decoupling (don't place workers on the orch host, or split the DB and work roots onto separate volumes) — the real long-term fix. Tests: disk parse/gate/staleness + closed-PR path-aware reclaim.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The orch host is itself a scheduler placement target (
placed on local), so worker/PR checkouts pile up under the work root on the same volume asorcha.db. When that volume hit 100%, SQLite writes failed (SQLITE_FULL) and the whole orchestrator went silent — a total outage (an assigned issue sat un-triggered for hours because the poller couldn't write its row), not one degraded worker.Two leaks fed the fill:
pr_branchcheckout is only reclaimed once the whole objective goes terminal — it stayed active for hours after the PR closed.reclaim.gowas false: the bare mirror stood at 21G.What
Four fixes, all mirroring the existing load-gate pattern:
dffree-space on the work root (SetTargetDiskstoresfree_disk_mb/disk_probed_at);SelectTargetskips a target belowMinFreeDiskMB(-min-free-disk-gb, default 10) so work parkswaiting_capacityinstead of filling the volume. Fails open on stale/missing samples, like load.ReclaimWorkspacesand emits a cooldown-deduped, push-notifieddisk_pressurealert.pr_branchcheckouts whose PR is merged/closed even mid-active-objective, path-aware so a dir a live session shares is never yanked. Deliberately not general mid-objectivepr_branchreclaim — a follow-up's un-pushed commit lives only on that checkout (the reviewer-commit loss class); the terminal-PR gate is the safe signal.git gcon each target's.orcha-cacheso the bare mirror stops growing unbounded.Default-on at 10GB, so protection activates on the next deploy without changing the prod launch command.
Deferred
The structural decoupling — don't place workers on the orch host, or split the DB and work roots onto separate volumes — is the real long-term fix, left as a follow-up rather than bundled into an incident fix.
Test plan
go build ./... && go vet ./... && go test ./...— all pass.dfparse (TestParseFreeDiskMB), placement gate + all-pressured backpressure + staleness fail-open + disabled (TestSelectTarget_*Disk*), store round-trip + gate (TestSetTargetDisk_RoundTripAndGate), and path-aware closed-PR reclaim including a finished row sharing a dir with a running follow-up (TestReclaimWorkspaces_ClosedPRBranchCheckout).