Skip to content

fix: prevent false spawn failures for live agents - #47

Open
ruby-dlee wants to merge 8 commits into
mainfrom
fm/fm-spawn-readiness-p3
Open

fix: prevent false spawn failures for live agents#47
ruby-dlee wants to merge 8 commits into
mainfrom
fm/fm-spawn-readiness-p3

Conversation

@ruby-dlee

@ruby-dlee ruby-dlee commented Jul 29, 2026

Copy link
Copy Markdown
Owner

Summary

Fixes six observed false spawn failures on 2026-07-29 across the Firstmate and Relvino pools, including this task's own Codex spawn. In every incident the worktree-path readiness probe timed out even though the harness agent was provably alive and working.

The root cause was shared with fm-herdr-send-fix-h7: Herdr path reads were gated by a legacy reachability/certificate path that could reject a healthy native server. This branch builds on that fix rather than duplicating it.

What changed

  • After the 60-second path-readiness loop, fm-spawn.sh now independently asks the backend's conservative harness-agent liveness classifier before condemning an unreadable endpoint.
  • An unreadable path plus confidently alive commits the spawn through the normal success path, with no direct_spawn_cleanup, direct_spawn_artifacts, or rollback_pending poison markers.
  • A dead or unknown agent still enters the existing rollback path.
  • A concrete wrong-worktree observation remains a hard failure even when an agent is alive.
  • Regression coverage exercises all three outcomes.
  • No-mistakes also hardened secondmate test fixtures so default-branch resolution works without assuming a local main branch in CI-created clones.

Verification

Implementation-specific automated coverage is green:

  • The new readiness regression failed before the fix with the reproduced 60-poll timeout.
  • FM_TEST_FOCUSED=spawn-readiness bash tests/fm-account-directory.test.sh passes the live-agent success, genuinely dead rollback, and wrong-worktree rollback cases.
  • The full tests/fm-account-directory.test.sh affected suite passes.
  • FM_TEST_FOCUSED=readsteer-native-cutover bash tests/fm-backend-herdr.test.sh passes the coordinated Herdr read/steer regressions.
  • Repository-pinned ShellCheck lint and Agent Fleet tests pass.
  • GitHub reports 4 passed, 0 failed, and 1 skipped check. Behavior Tests was cancelled and recorded as skipped while Firstmate reduced machine saturation; no-mistakes declared the PR CI-ready.

Live validation limitation

No live end-to-end Herdr spawn was run because this task was not scaffolded with --herdr-lab, and its safety contract forbids touching Herdr lifecycle without that guard.

That leaves real backend timing, native process registration, and the post-land behavior of an ordinary production spawn unproven by this branch. The real-world reproduction is unusually strong—six independent incidents across two pools, all with active Codex agents—but it is reproduction evidence, not post-fix live validation.

Firstmate will verify the next ordinary spawn after landing and report whether it records clean success metadata. An isolated Herdr-lab lifecycle regression is being filed separately.

Risk

The fallback is deliberately narrow: only confident harness-agent liveness can rescue an unreadable readiness probe. Concrete isolation violations and dead or unknown endpoints still fail closed and roll back.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant