feat: rescue exhausted direct-account crews

.github/workflows/ci.yml

@@ -27,9 +27,10 @@ jobs:
   tests:
     name: Behavior tests
     runs-on: ubuntu-latest
-    # The pre-cutover suite reached 25m35s, and the exhaustive Bridge crash
-    # matrix adds bounded work. Keep enough margin without masking a real hang.
-    timeout-minutes: 45
+    # The serial suite now includes the bounded direct-account recovery matrix
+    # as well as the exhaustive Bridge crash matrix. Keep enough margin for both
+    # without masking a real hang.
+    timeout-minutes: 60
     steps:
       - uses: actions/checkout@v6
         with:

AGENTS.md

@@ -108,6 +108,7 @@ state/               volatile runtime signals; gitignored
   <id>.meta          written by fm-spawn: window=, worktree=, project=, harness=, model=, effort=, kind=, mode=, yolo=, tasktmp=, generation_id=, report_required=
   Direct ship and scout launches also own account_home=, worktree_git_dir=, worktree_git_dir_identity=, and exactly one authoritative final-state field: worktree_git_ref= for an attached branch or worktree_git_head= for an intentional detached HEAD. Their metadata may temporarily carry worktree_git_setup_ref= and worktree_git_setup_head= while the brief's required `fm/<id>` branch transition is pending; recovery accepts only that exact setup state or the authoritative `fm/<id>` ref and removes the setup fields after adoption.
   Direct recovery validates the canonical worktree path, exact physical Git-dir identity, and authoritative final state before account preparation and again immediately before endpoint creation; any drift fails closed without launching.
+  Automatic account-scoped capacity rescue appends capacity_rescue_attempts=, repeated capacity_rescue_exhausted_account= entries, per-attempt audit/result fields, and an optional capacity_rescue_stopped= terminal reason; Codex model capacity writes none of them, and docs/account-capacity-rescue.md owns the full routing, transaction, and retry boundary.
   If endpoint removal after a failed new direct spawn cannot be confirmed, direct_spawn_cleanup=pending and rollback_pending=1 retain the endpoint and worktree identity for explicit teardown.
   A failed new direct spawn that never created an endpoint records direct_spawn_endpoint=not-created and an empty window= so teardown skips endpoint quiescence without skipping worktree safety.
   Secondmate Agent Fleet routing and legacy managed recovery own account_pool=, account_profile=, account_task=, account_attempt=, and provider_session_id= (docs/configuration.md "Agent Fleet account routing").
@@ -585,6 +586,8 @@ The emitted block is the only per-harness operating recipe in the session contex
 Do not substitute another harness's command shape for it.
 **Always-on wake triage (absorb only when provably working).**
 `bin/fm-watch.sh` classifies every wake in bash and absorbs the benign majority without waking you: crewmates with positive working evidence (an actively-running no-mistakes step for their branch, or a busy pane read via `bin/fm-crew-state.sh`) unless the separate permission-stall no-progress threshold has expired, a declared `paused:` external wait until its bounded recheck cadence, and no-change heartbeats.
+Before ordinary stale classification, it recognizes empirically verified provider-capacity chrome only on an idle endpoint: account-scoped Claude exhaustion enters guarded direct-account recovery, while Codex model capacity surfaces once without endpoint removal, account rotation, or rescue metadata.
+Account rescue is lifecycle-serialized, attempt-capped, and durably stopped with one blocked wake when no unused eligible account remains; `docs/account-capacity-rescue.md` owns the signatures, routing, audit fields, and failure contract.
 It never absorbs a crewmate that stopped without that evidence - whatever its stale status log claims - and only an actionable wake is queued durably and ends the supervision wait, so you resume the emitted protocol exactly once per actionable event.
 A `paused:` status is a deliberate external wait, not `blocked:`; its initial signal still surfaces once, and a forgotten pause re-surfaces for a recheck once per window.
 Repeated unchanged wedge or permission-stall escalations eventually add `demand-deep-inspection` to the wake reason so they are not mistaken for another routine validation wait.

bin/backends/herdr.sh

@@ -2654,20 +2654,20 @@ fm_backend_herdr_expected_label_matches() {  # <target> [expected-label]
 #
 # Reading a pane or sending keys to it does not care how the server was
 # launched - the pane already exists and the server is up. It only needs the
-# server to be running and adapter-owned (this HOME's own certificate names the
-# live pid), which fm_backend_herdr_server_adapter_owned proves without the
-# closed-shell launch certification. This is what keeps peek/steer working when
-# FirstMate itself runs INSIDE the herdr session it manages (HERDR_ENV=1): that
-# server was not launched through the crewmate adapter's own closed-shell path, so
-# closed_shell_environment_ready is false and the full ensure would try to
-# restart+recertify - impossible while the session is occupied by live crewmates and
-# FirstMate itself. The SPAWN path deliberately keeps the strict ensure so a new
-# crewmate still launches only in a certified closed-shell server.
+# server to be running. During the legacy certificate lifecycle it must also be
+# adapter-owned (this HOME's certificate names the live pid), which
+# fm_backend_herdr_server_adapter_owned proves without requiring the current
+# release's closed-shell launch certification. Native-agent production no longer
+# requires that legacy server certificate; pane isolation is applied directly by
+# `agent start --env`, so read/steer follows server ensure's native cutover and
+# accepts a reachable server. Certificate-enforcing tests retain the stricter
+# ownership proof.
 fm_backend_herdr_server_reachable_for_readsteer() {  # <session>
   local session=$1 running
   running=$(fm_backend_herdr_cli "$session" status --json 2>/dev/null \
     | fm_backend_herdr_control_jq -r '.server.running // false' 2>/dev/null)
   [ "$running" = true ] || return 1
+  fm_backend_herdr_server_certificate_required || return 0
   fm_backend_herdr_server_adapter_owned "$session"
 }
 
@@ -3013,9 +3013,10 @@ fm_backend_herdr_send_text_submit() {  # <target> <text> <retries> <enter-sleep>
 # tmux-kill-window's `|| true` contract). Verified: closing a tab's only pane
 # closes the tab too, so a separate tab close is unnecessary.
 fm_backend_herdr_kill() {  # <target> [backend-id] [expected-label]
-  if ! fm_backend_herdr_target_ready "$1" "${3:-}"; then
-    [ -z "${3:-}" ] && return 0
-    return 1
+  if [ -n "${3:-}" ]; then
+    fm_backend_herdr_target_ready "$1" "$3" || return 1
+  else
+    fm_backend_herdr_parse_target "$1" || return 0
   fi
   fm_backend_herdr_cli "$FM_BACKEND_HERDR_SESSION" pane close "$FM_BACKEND_HERDR_PANE" >/dev/null 2>&1 || true
 }

bin/backends/orca.sh

@@ -58,7 +58,7 @@ fm_backend_orca_authority_capabilities_check() {
   return 1
 }
 
-fm_backend_orca_json_get() {  # <field> ; fields: worktree-id worktree-path terminal-handle terminal-title worktree-terminal-handle repo-id
+fm_backend_orca_json_get() {  # <field> ; fields: worktree-id worktree-path worktree-name terminal-handle terminal-title worktree-terminal-handle repo-id
   # Terminal handles are accepted only from verified terminal result shapes:
   # result.terminal or a root terminal object with .handle. Undocumented
   # result.id and result.worktree.terminal shapes are ignored until a real Orca
@@ -88,6 +88,7 @@ function handle(obj) {
 let v = "";
 if (field === "worktree-id") v = wt.id || wt.worktreeId || r.worktreeId || "";
 if (field === "worktree-path") v = wt.path || (wt.git && wt.git.path) || r.path || "";
+if (field === "worktree-name") v = scalar(wt.name) || scalar(wt.title) || scalar(r.worktreeName) || "";
 if (field === "terminal-handle") v = handle(explicitTerm || r) || "";
 if (field === "terminal-title") v = scalar((explicitTerm || r).title) || scalar((explicitTerm || r).name) || "";
 if (field === "worktree-terminal-handle") v = handle(explicitTerm) || "";
@@ -141,7 +142,7 @@ fm_backend_orca_repo_ensure() {  # <project-path>
 }
 
 fm_backend_orca_worktree_create() {  # <project-path> <name>
-  local project=$1 name=$2 repo_id out wt_id wt_path terminal status proof
+  local project=$1 name=$2 repo_id out wt_id wt_path wt_name terminal status proof
   repo_id=$(fm_backend_orca_repo_ensure "$project") || return 1
   if out=$(orca worktree create --repo "id:$repo_id" --name "$name" --no-parent --setup skip --json); then
     status=0
@@ -151,10 +152,11 @@ fm_backend_orca_worktree_create() {  # <project-path> <name>
   wt_id=$(printf '%s' "$out" | fm_backend_orca_json_get worktree-id 2>/dev/null || true)
   terminal=$(printf '%s' "$out" | fm_backend_orca_json_get worktree-terminal-handle 2>/dev/null || true)
   wt_path=$(printf '%s' "$out" | fm_backend_orca_json_get worktree-path 2>/dev/null || true)
+  wt_name=$(printf '%s' "$out" | fm_backend_orca_json_get worktree-name 2>/dev/null || true)
   proof=unproven
   [ -z "$terminal" ] || proof=recorded
-  printf '%s\t%s\t%s\t%s\t%s' "$wt_id" "$wt_path" "$terminal" "$proof" "$repo_id"
-  if [ "$status" -ne 0 ] || [ -z "$wt_id" ] || [ -z "$wt_path" ]; then
+  printf '%s\t%s\t%s\t%s\t%s\t%s' "$wt_id" "$wt_path" "$terminal" "$proof" "$repo_id" "$wt_name"
+  if [ "$status" -ne 0 ] || [ -z "$wt_id" ] || [ -z "$wt_path" ] || [ -z "$wt_name" ]; then
     echo "error: orca worktree create returned incomplete or unsuccessful authority for $name" >&2
     return 2
   fi

bin/fm-account-directory.sh

@@ -1,9 +1,9 @@
 #!/usr/bin/env bash
 # Select and prepare direct Claude or Codex account-directory launches.
 # Usage:
-#   fm-account-directory.sh select <claude|codex>
+#   fm-account-directory.sh select <claude|codex> [excluded-account-home...]
 #   fm-account-directory.sh install-herdr-hook <claude|codex> <account-home>
-#   fm-account-directory.sh prepare <claude|codex>
+#   fm-account-directory.sh prepare <claude|codex> [excluded-account-home...]
 #
 # This header is the single owner of the direct account-directory contract.
 # FM_ACCOUNT_DIRECTORY_CUTOVER: direct-observe-passwd-home-v2
@@ -12,13 +12,15 @@
 # Codex selection removes that account's quota-axi window cache immediately
 # before every read, sets CODEX_HOME plus the account-isolated XDG_CACHE_HOME,
 # accepts only a fresh result with at least one numeric five_hour or weekly
-# window, and picks the account with the highest minimum remaining percentage.
+# window, skips excluded and zero-capacity accounts, and picks the remaining
+# account with the highest minimum remaining percentage.
 # A Codex account with no such freshly readable window is skipped as unhealthy.
 # Claude quota is not currently distinguishable per config directory because
 # quota-axi cannot non-interactively resolve Claude's config-dir-specific macOS
 # Keychain credential.
 # Claude therefore never treats a missing usage window as account failure and
-# selects the first real account directory in stable bytewise sort order.
+# selects the first non-excluded real account directory in stable bytewise sort
+# order.
 # Selection prints only the chosen absolute account home on stdout and logs
 # health, fallback, and choice diagnostics on stderr.
 # prepare selects the account and idempotently runs Herdr's own integration
@@ -203,8 +205,18 @@ valid_account_home() { # <vendor-dir> <candidate>
   esac
 }
 
-first_account_home() { # <vendor>
+account_home_is_excluded() { # <candidate> [excluded-account-home...]
+  local candidate=$1 excluded
+  shift
+  for excluded in "$@"; do
+    [ "$candidate" != "$excluded" ] || return 0
+  done
+  return 1
+}
+
+first_account_home() { # <vendor> [excluded-account-home...]
   local vendor=$1 root vendor_dir candidate
+  shift
   root=$(account_root) || return 1
   vendor_dir=$root/$vendor
   [ -d "$vendor_dir" ] && [ ! -L "$vendor_dir" ] || {
@@ -215,10 +227,14 @@ first_account_home() { # <vendor>
   export LC_ALL
   for candidate in "$vendor_dir"/*; do
     valid_account_home "$vendor_dir" "$candidate" || continue
+    if account_home_is_excluded "$candidate" "$@"; then
+      log "$vendor account $candidate skipped: exhausted by this task"
+      continue
+    fi
     printf '%s\n' "$candidate"
     return 0
   done
-  echo "error: no account directories found for $vendor under $vendor_dir" >&2
+  echo "CAPACITY_UNAVAILABLE: no unused $vendor account directories remain under $vendor_dir" >&2
   return 1
 }
 
@@ -279,7 +295,7 @@ $1
 EOF
 }
 
-select_codex() {
+select_codex() { # [excluded-account-home...]
   local root vendor_dir quota_bin candidate usage score
   local best_home='' best_score=''
   root=$(account_root) || return 1
@@ -297,12 +313,20 @@ select_codex() {
   export LC_ALL
   for candidate in "$vendor_dir"/*; do
     valid_account_home "$vendor_dir" "$candidate" || continue
+    if account_home_is_excluded "$candidate" "$@"; then
+      log "codex account $candidate skipped: exhausted by this task"
+      continue
+    fi
     usage=$(fresh_codex_usage_json "$candidate" "$quota_bin") || usage=
     score=$(codex_score "$usage") || score=
     if [ -z "$score" ]; then
       log "codex account $candidate skipped: no freshly readable usage window"
       continue
     fi
+    if ! awk -v candidate_score="$score" 'BEGIN { exit !(candidate_score > 0) }'; then
+      log "codex account $candidate skipped: fresh remaining score=$score has no capacity"
+      continue
+    fi
     log "codex account $candidate fresh remaining score=$score"
     if [ -z "$best_home" ] || awk -v candidate_score="$score" -v current_score="$best_score" \
       'BEGIN { exit !(candidate_score > current_score) }'; then
@@ -311,26 +335,28 @@ select_codex() {
     fi
   done
   [ -n "$best_home" ] || {
-    echo "error: no healthy Codex account has a freshly readable usage window" >&2
+    echo "CAPACITY_UNAVAILABLE: no unused Codex account has a freshly readable positive usage window" >&2
     return 1
   }
   log "selected codex account $best_home with fresh remaining score=$best_score"
   printf '%s\n' "$best_home"
 }
 
-select_claude() {
+select_claude() { # [excluded-account-home...]
   local selected
-  selected=$(first_account_home claude) || return 1
+  selected=$(first_account_home claude "$@") || return 1
   log "CLAUDE USAGE UNREADABLE: quota-axi cannot non-interactively resolve Claude's config-dir-specific macOS Keychain credential today; selecting the first account directory by stable sort: $selected"
   printf '%s\n' "$selected"
 }
 
-select_account() { # <vendor>
-  case "$1" in
-    codex) select_codex ;;
-    claude) select_claude ;;
+select_account() { # <vendor> [excluded-account-home...]
+  local vendor=$1
+  shift
+  case "$vendor" in
+    codex) select_codex "$@" ;;
+    claude) select_claude "$@" ;;
     *)
-      echo "error: direct account-directory selection supports only claude or codex, not '$1'" >&2
+      echo "error: direct account-directory selection supports only claude or codex, not '$vendor'" >&2
       return 1
       ;;
   esac
@@ -378,17 +404,21 @@ case "${1:-}" in
     exit 0
     ;;
   select)
-    [ "$#" -eq 2 ] || { usage; exit 2; }
-    select_account "$2"
+    [ "$#" -ge 2 ] || { usage; exit 2; }
+    vendor=$2
+    shift 2
+    select_account "$vendor" "$@"
     ;;
   install-herdr-hook)
     [ "$#" -eq 3 ] || { usage; exit 2; }
     install_herdr_hook "$2" "$3"
     ;;
   prepare)
-    [ "$#" -eq 2 ] || { usage; exit 2; }
-    selected_home=$(select_account "$2") || exit 1
-    install_herdr_hook "$2" "$selected_home" || exit 1
+    [ "$#" -ge 2 ] || { usage; exit 2; }
+    vendor=$2
+    shift 2
+    selected_home=$(select_account "$vendor" "$@") || exit 1
+    install_herdr_hook "$vendor" "$selected_home" || exit 1
     printf '%s\n' "$selected_home"
     ;;
   *)

bin/fm-checkout-refresh.sh

@@ -1869,7 +1869,7 @@ record_alert() {
 
 record_reinspection_failure() {
   local checkout=$1 key alert output
-  key=$(checkout_key "$checkout") || {
+  key=$(fm_checkout_hash_value "$checkout" 24) || {
     printf '%s: skipped: covered checkout lock identity cannot be resolved\n' "$checkout"
     return 1
   }

bin/fm-classify-lib.sh

@@ -64,6 +64,59 @@ FM_PAUSE_RESURFACE_SECS_DEFAULT=3600
 # this is the one owner of the verb literal, overridable via FM_CLASSIFY_RESOLVE_VERB.
 FM_CLASSIFY_RESOLVE_VERB_DEFAULT='resolved'
 
+# Classify a provider-owned capacity failure from the bounded pane capture of a
+# known harness. Prints a stable failure token and returns 0 on a verified shape;
+# prints nothing and returns 1 otherwise.
+#
+# These match complete TUI chrome, not capacity-like prose. Codex requires its
+# standalone warning glyph and exact line. Claude requires either its tool-result
+# glyph plus exact session-limit sentence or the complete usage-limit choice
+# dialog. This prevents a task discussing "usage limits", quoting one sentence,
+# or printing ordinary API rate-limit output from being mistaken for an exhausted
+# harness account. Empirical captures and versions are recorded in
+# docs/account-capacity-rescue.md.
+provider_capacity_failure_kind() {  # <harness> <tail40>
+  local harness=$1 tail40=$2 prompt_tail
+  prompt_tail=$(printf '%s\n' "$tail40" | tail -18)
+  case "$harness" in
+    codex)
+      if printf '%s\n' "$prompt_tail" \
+        | grep -Eq '^[[:space:]]*⚠[[:space:]]+Selected model is at capacity\. Please try a different model\.[[:space:]]*$'; then
+        printf 'codex-model-capacity'
+        return 0
+      fi
+      ;;
+    claude)
+      if printf '%s\n' "$prompt_tail" \
+        | grep -Eq "^[[:space:]]*⎿.*You've hit your session limit · resets .+[[:space:]]*$"; then
+        printf 'claude-session-limit'
+        return 0
+      fi
+      if printf '%s\n' "$prompt_tail" | grep -Fq 'What do you want to do?' \
+        && printf '%s\n' "$prompt_tail" | grep -Eq '1\. Stop and wait for limit to reset[[:space:]]*$' \
+        && printf '%s\n' "$prompt_tail" | grep -Eq '2\. Switch to usage credits[[:space:]]*$' \
+        && printf '%s\n' "$prompt_tail" | grep -Eq '3\. Switch to Team plan[[:space:]]*$' \
+        && printf '%s\n' "$prompt_tail" | grep -Eq 'Enter to confirm.*Esc to cancel'; then
+        printf 'claude-usage-limit-dialog'
+        return 0
+      fi
+      ;;
+  esac
+  return 1
+}
+
+provider_credential_failure_kind() {  # <harness> <tail40>
+  local harness=$1 tail40=$2 prompt_tail
+  [ "$harness" = claude ] || return 1
+  prompt_tail=$(printf '%s\n' "$tail40" | tail -18)
+  if printf '%s\n' "$prompt_tail" \
+    | grep -Eq '^[[:space:]]*Not logged in - Please run /login[[:space:]]*$'; then
+    printf 'claude-not-logged-in'
+    return 0
+  fi
+  return 1
+}
+
 # Return the last non-blank line of a status file (empty if missing/blank).
 last_status_line() {
   local f=$1