Skip to content

feat: automatically reap terminal crews and orphaned lifecycle state - #41

Open
ruby-dlee wants to merge 28 commits into
mainfrom
fm/fm-auto-reap-gc
Open

feat: automatically reap terminal crews and orphaned lifecycle state#41
ruby-dlee wants to merge 28 commits into
mainfrom
fm/fm-auto-reap-gc

Conversation

@ruby-dlee

Copy link
Copy Markdown
Owner

Intent

Verify the backlog premise against current code and history, then make Firstmate continuously garbage-collect done and merged crews and recover orphaned no-mistakes or agent lifecycle state so terminal zombies cannot accumulate memory. Reaping must happen automatically from the supervision and terminal-state paths, use the ordinary guarded teardown rather than broad process-name killing, cancel only the exact matching active no-mistakes run, and fail closed on ambiguous identity, live owners, dirty or unlanded work, X-linked tasks, or uncertain Treehouse leases. Recover acquisitions stranded by an owner crash only with exact PID plus process-start proof and exact unique lease authority. Fix any teardown false-positive that prevents safe native Herdr auto-reap while retaining exact workspace, tab, pane, label, ancestry, and process identity guards. Exercise Herdr only in a named nondefault lab through the guarded helper with before and after fleet-state tripwires, never touch unrelated sessions, and explicitly state that this Firstmate tooling change requires no web-app, API, or realtime deployment.

What Changed

  • Automatically reap merged ship tasks and completed scouts through the existing guarded teardown path, cancelling only exactly attributed active no-mistakes runs.
  • Recover owner-dead, pre-metadata Treehouse acquisitions using generation-scoped ownership and fail-closed identity, lease, worktree, landing, and X-link checks.
  • Fix native Herdr teardown validation and add isolated coverage for watcher handoffs, acquisition recovery, lease ambiguity, and Herdr lifecycle cleanup.

Risk Assessment

✅ Low: The prior Treehouse findings are resolved with fail-closed authoritative-state validation, distinct corruption diagnostics, retained recovery authority, focused regression coverage, and the required deployment declaration; no additional material source risks were found.

Testing

Completed 1 recorded test check.

  • Outcome: ⚠️ 1 error across 5 runs (15h28m4s)

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

🔧 **Review** - 2 issues found → auto-fixed (2) ✅
  • 🚨 bin/fm-auto-reap.sh:343 - The intent requires recovery to “fail closed on … uncertain Treehouse leases,” but a zero result here only means no matching lease was found among git worktree list entries. A leased slot absent from that list (for example, partially registered or prunable after the crash) is not checked against authoritative Treehouse pool state, yet the acquisition record is deleted as proof of no lease. Retain the record unless absence is proven directly across the relevant authoritative Treehouse state.
  • 🚨 docs/architecture.md:38 - The authoritative intent requires the change to “explicitly state that this Firstmate tooling change requires no web-app, API, or realtime deployment.” The new architecture documentation describes auto-reaping but contains no such deployment statement. Add the required explicit declaration to the change documentation.

🔧 Fix: Captain, fail closed on uncertain Treehouse lease absence
1 error still open:

  • 🚨 bin/fm-treehouse-lib.sh:129 - The intent requires recovery to “fail closed on … uncertain Treehouse leases,” but this absence proof accepts malformed authoritative entries: a leased entry with a missing/non-string lease_holder (or missing/invalid path) does not raise and therefore counts as proof that the expected holder is absent. That lets recover_acquisition delete its authority record despite corrupt/ambiguous Treehouse state. Validate every leased entry’s holder and path schema before treating a non-match as proven absence.

🔧 Fix: Captain, reject malformed Treehouse lease absence proofs
✅ Re-checked - no issues remain.

⚠️ **Test** - 1 error
  • 🚨 tests failed with exit code 1
  • command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; for t in tests/*.test.sh; do echo "== $t =="; bash "$t" || rc=1; done; uv run --directory tools/agent-fleet --locked pytest || rc=1; uv run --directory tools/agent-fleet --locked python -m compileall -q src || rc=1; exit "$rc"

🔧 Fix: Stabilize watcher handoff and lock takeover tests
1 error still open:

  • 🚨 tests failed with exit code 1
  • command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; for t in tests/*.test.sh; do echo "== $t =="; bash "$t" || rc=1; done; uv run --directory tools/agent-fleet --locked pytest || rc=1; uv run --directory tools/agent-fleet --locked python -m compileall -q src || rc=1; exit "$rc"

🔧 Fix: Isolate account recovery fixtures from concurrent runs
1 error still open:

  • 🚨 tests failed with exit code 1
  • command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; for t in tests/*.test.sh; do echo "== $t =="; bash "$t" || rc=1; done; uv run --directory tools/agent-fleet --locked pytest || rc=1; uv run --directory tools/agent-fleet --locked python -m compileall -q src || rc=1; exit "$rc"

🔧 Fix: Fix generation-scoped task temp ownership
1 error still open:

  • 🚨 tests failed with exit code 1
  • command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; for t in tests/*.test.sh; do echo "== $t =="; bash "$t" || rc=1; done; uv run --directory tools/agent-fleet --locked pytest || rc=1; uv run --directory tools/agent-fleet --locked python -m compileall -q src || rc=1; exit "$rc"

🔧 Fix: Waiting for overlapping full suite before clean verification
1 error still open:

  • 🚨 tests failed with exit code 1
  • command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; for t in tests/*.test.sh; do echo "== $t =="; bash "$t" || rc=1; done; uv run --directory tools/agent-fleet --locked pytest || rc=1; uv run --directory tools/agent-fleet --locked python -m compileall -q src || rc=1; exit "$rc"
✅ **Document** - passed

✅ No issues found.

🔧 **Lint** - 1 issue found → auto-fixed ✅
  • ⚠️ linter found issues (exit code 1)

🔧 Fix: Fix ShellCheck ownership and test-script warnings, captain
✅ Re-checked - no issues remain.

✅ **Push** - passed

✅ No issues found.

@ruby-dlee
ruby-dlee force-pushed the fm/fm-auto-reap-gc branch from 97a61ed to 3f0e3c5 Compare July 27, 2026 18:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant