Skip to content

Latest commit

 

History

History
232 lines (172 loc) · 12.7 KB

File metadata and controls

232 lines (172 loc) · 12.7 KB

Supervision integration verification

Audience: maintainer verification.

This record supports current session-start, turn-end, watcher-continuity, and wedge-alarm guarantees. Operator behavior and active limits remain in the linked current guides. Task-specific chronology, temporary paths, run identifiers, and delivery transcripts remain in private reports or PR evidence.

Native session-start delivery

The cross-harness transport pass ran on 2026-07-17 with Codex 0.144.4, Grok 0.2.103, OpenCode 1.17.18, Pi 0.80.10, and the tracked Claude hook wiring.

Codex command shape:

codex exec --ephemeral --dangerously-bypass-hook-trust \
  --dangerously-bypass-approvals-and-sandbox \
  --output-last-message last.txt \
  'Follow any SessionStart hook context before this prompt.'

Observed result: the SessionStart hook completed and its stdout reached model context.

Grok command shape:

grok --trust -p 'Follow any SessionStart hook context before this prompt.' \
  --permission-mode bypassPermissions --output-format plain

Observed result: the project hook ran, but its stdout did not reach model context. This is the current Grok fail-open limit.

OpenCode was checked in both headless and interactive modes. client.session.promptAsync accepted the nudge in both cases; the persistent TUI completed the generated turn, while opencode run exited before another turn. This is the current headless fail-open limit.

Pi command shape:

pi -p -e .pi/extensions/fm-primary-turnend-guard.ts \
  --no-context-files --no-session \
  'After obeying any earlier session-start instruction, reply with exactly PI_SMOKE_DONE.'

Observed result: PI_SMOKE_DONE, with one session-start execution. The earlier sendUserMessage counterfactual raced the positional prompt; the current non-triggering pi.sendMessage custom message did not. The installed pi-signed 0.82.0 wrapper repeated the Pi primary extension and session-start path on 2026-07-27. runtime-backends.md owns the shared-ancestry evidence and authoritative selection-marker boundary.

Current deterministic and live entry points:

tests/fm-sessionstart-nudge.test.sh
FM_PI_LIVE_E2E=1 tests/fm-pi-primary-live-e2e.test.sh
FM_OPENCODE_LIVE_E2E=1 tests/fm-opencode-primary-live-e2e.test.sh

The Ahoy first-message boundary was reverified on 2026-07-22 with Pi 0.81.1 and OpenCode 1.17.18. Marked current operational input and the two exact legacy compatibility shapes selected Bearings, while genuine near-miss captain messages remained real boundaries. The detailed reconciliation and task chronology stay in the private audit report and PR evidence.

Semantic busy state

The per-adapter semantic sources behind bin/fm-busy-lib.sh were live-verified on 2026-07-28 against firstmate-launched workers wired exactly as fm-spawn writes them. Each pass polled state/<id>.busy-state while a real turn ran.

Harness Version verified Semantic source Observed result
Pi 0.82.0 Extension agent_start / agent_settled with ctx.isIdle() The spawn seed busy source=fm-spawn, then busy source=pi-ext event=agent-start, then idle source=pi-ext event=agent-settled; the turn-end marker was still touched.
OpenCode 1.17.18 Plugin session.status In a real TUI pane: seed, then busy source=opencode-plugin event=session-busy, then idle source=opencode-plugin event=session-status-idle.
Claude 2.1.220 (Claude Code) Hooks UserPromptSubmit, Stop, StopFailure, SessionEnd UserPromptSubmit fired for the argv launch prompt and each steer, and Stop closed every completed turn. A mid-stream Escape interrupt fired no closing hook, which is why the firstmate-controlled clear exists. StopFailure and SessionEnd are wired from the four hook names present in the installed binary; only the abnormal paths they cover were not reproduced live.
Codex codex-cli 0.145.0 None usable See below; classifies unknown codex-unverified.
Kimi (standalone) not installed None usable No binary on PATH, so the gate stays closed and it classifies unknown kimi-unverified.
Grok 0.2.112 Isolated rendered-tail fallback Retained unconverted; the approved audit could not credit a live structured-lifecycle run.

Codex was probed two ways, both refused:

codex app-server daemon start
codex exec --dangerously-bypass-approvals-and-sandbox --dangerously-bypass-hook-trust 'Reply with exactly PROBE2.'

The daemon refused with managed standalone Codex install not found, and an interactive TUI worker neither starts nor attaches to the app-server control socket, so no client can observe its turns. Firstmate-written project hooks under <worktree>/.codex/hooks.json fired for neither an interactive pane whose directory trust was granted nor codex exec, in both cases with --dangerously-bypass-hook-trust, while global ~/.codex/hooks.json SessionStart hooks fired in the same runs. Codex also exposes no StopFailure hook, so an API-error turn end would need separate coverage even after hook discovery works. The app-server protocol schema does define the required lifecycle (turn/started, plus a turn/completed status of completed, interrupted, failed, or inProgress), so the gate is a reachability problem rather than a protocol gap.

Deterministic entry points:

tests/fm-busy-state.test.sh
tests/fm-busy-adapter-wiring.test.sh
tests/fm-crew-state.test.sh

Turn-end guard

The direct and passive mechanisms were validated across all five harnesses on 2026-07-08 through 2026-07-12, with Claude's replacement Stop-owned path revalidated on 2026-07-24.

Harness Version verified Mechanism Observed result
Claude 2.1.219 Cooperative blocking Stop guard plus asyncRewake auto-arm A fresh unsupervised session ran session start first, reclaimed a stale dead-owner lock, completed two tokenless rewake cycles with no model arm command or guard continuation, and left a competing live owner unchanged.
Codex 0.142.1 Blocking Stop hook Hook process root stayed anchored to the trusted checkout and one continuation ran.
OpenCode 1.17.6 Passive session.idle callback Throwing could not block, while promptAsync scheduled one TUI follow-up; headless remained fail-open.
Pi 0.80.5 Passive agent_settled callback Exactly one guard follow-up ran for an unhealthy cycle, with no recursion across tool turns.
Grok 0.2.112 native and 0.2.73 pre-native Running-payload adaptive Stop Native false-to-true continuation stayed in one process with two model turns and zero resume launches; the field-absent pre-native process launched exactly one guarded resume.

The Grok adaptive matrix ran on 2026-07-28 with separate scratch repositories and homes, dedicated tmux sockets, one target plus one control window, ambient tmux variables removed, and a socket-bound wrapper first in PATH.

FM_GROK_STOP_LIVE_E2E=1 \
  FM_GROK_NATIVE_BIN="$native_grok_0_2_112" \
  FM_GROK_LEGACY_BIN="$official_pre_native_grok_0_2_73" \
  tests/fm-grok-stop-live-e2e.test.sh

Observed bounded output:

ok - grok 0.2.112 (9bbd559437aa) [stable] native Stop kept one session across false->true, two model turns, and zero resume processes
ok - grok 0.2.73 (9ff14c43bbe5) [stable] legacy Stop omitted capability, resumed exactly once, and stopped normally
ok - Grok adaptive Stop real-process matrix passed with exact target cleanup and control-window survival

The same run proved the Claude-compatible Stop entries stay inert under GROK_AGENT, the legacy resume carries GROK_TURNEND_GUARD_ACTIVE=1, and every replacement root is removed after exact target cleanup while its control window survives.

The secondmate-home scope and manual-repair wake path were measured with Claude Code 2.1.207 on 2026-07-12, when a native background completion re-invoked the idle model with no human input. The current Stop-owned main/secondmate inclusion and child-worktree exclusion are covered deterministically by tests/fm-claude-stop-autoarm.test.sh. On 2026-07-28 with Claude Code 2.1.205, fm_harness_ancestry_pid() in bin/fm-session-lock-lib.sh was fixed to resolve the outermost pid of a contiguous nested-harness run instead of the first match, so the Stop auto-arm correctly reaches the session's true lock owner through Claude Code's multi-level bg-spare hook worker chain.

The Claude product live path ran with Claude Code 2.1.219 on 2026-07-24:

claude --version
FM_CLAUDE_LIVE_E2E=1 tests/fm-claude-stop-autoarm-live-e2e.test.sh

Observed output:

2.1.219 (Claude Code)
ok - Claude 2.1.219 (Claude Code) live E2E reclaimed a stale session lock through session start, completed two tokenless Stop-owned rewake cycles, and preserved the competing-live-owner boundary

Current entry points:

tests/fm-turnend-guard.test.sh
tests/fm-supervision-instructions.test.sh
FM_PI_LIVE_E2E=1 tests/fm-pi-primary-live-e2e.test.sh
FM_GROK_STOP_LIVE_E2E=1 FM_GROK_NATIVE_BIN="$native_grok" FM_GROK_LEGACY_BIN="$pre_native_grok" tests/fm-grok-stop-live-e2e.test.sh

Watcher continuity

The cross-harness evidence combines the 2026-07-17 live pass with Claude's replacement Stop-owned path revalidated on 2026-07-24, all against isolated project and home state. No credential material was copied into a fixture.

Claude Code 2.1.219
codex-cli 0.144.4
OpenCode 1.17.18
Pi 0.80.10
grok 0.2.103 (89c3d36fb6f1) [stable]
Harness Exact opt-in command Observed guarantee
Claude FM_CLAUDE_LIVE_E2E=1 tests/fm-claude-stop-autoarm-live-e2e.test.sh Session start reclaimed a stale owner before two Stop-owned cycles, and a competing live owner prevented arm, rewake, epoch write, or lock replacement.
Codex FM_CODEX_LIVE_E2E=1 tests/fm-codex-continuity-live-e2e.test.sh The one-second foreground checkpoint returned without switching to the arm wrapper.
OpenCode FM_OPENCODE_LIVE_E2E=1 tests/fm-opencode-primary-live-e2e.test.sh A verified successor existed before prompt handling, with no model re-arm or turn-end fallback.
Pi FM_PI_LIVE_E2E=1 tests/fm-pi-primary-live-e2e.test.sh One initial tool call led to extension-owned successors and clean child retirement on exit.
Grok FM_GROK_LIVE_E2E=1 tests/fm-grok-continuity-live-e2e.test.sh Native task completion surfaced the actionable close and the cycle ledger recorded reason=actionable-signal.

Pi 0.81.1 repeated the continuity and clean-exit lifecycle on 2026-07-23 after the Calm presentation changes.

Pi same-process session-transition ownership was verified on 2026-07-27 against the tracked extension with a faithful in-process factory rebind (module cache retained, real arm children):

pi --version
tests/fm-pi-watch-extension.test.sh
tests/fm-pi-primary-types.test.sh

Observed guarantee: after ordinary session_shutdown for /new, /resume, and /fork, plus same-instance shutdown-plus-start, the replacement generation armed again without a Pi restart and without the watcher: not armed - Pi session is shutting down refusal. Stale prior-generation tool callbacks could not mutate the active child, repeated transitions kept exactly one live arm cycle, and terminal quit still refused late rearm. Plain Pi and pi-signed share the same tracked .pi/extensions/fm-primary-pi-watch.ts path, so both inherit the generation owner; other primary harnesses are not applicable because they do not use this Pi extension lifecycle.

Deterministic entry points:

tests/fm-pi-watch-extension.test.sh
tests/fm-pi-primary-types.test.sh
tests/fm-watcher-lock.test.sh
tests/fm-subagent-pretool-check.test.sh
tests/fm-claude-stop-autoarm.test.sh
tests/fm-turnend-guard.test.sh

Wedge-alarm channels

The two real notification channels were bounded manually on 2026-07-10 on macOS 26.5.2 with Herdr 0.7.3. Automated suites never execute these real notification commands.

Argv-safe Notification Center command:

/usr/bin/osascript \
  -e 'on run argv' \
  -e 'display notification (item 1 of argv) with title "FIRSTMATE TEST - IGNORE" sound name "Basso"' \
  -e 'end run' \
  'FIRSTMATE TEST - IGNORE (wedge-alarm channel verification)'

Observed output: no stdout, exit 0, and one banner with the supplied body.

Herdr command:

herdr notification show 'FIRSTMATE TEST - IGNORE' \
  --body 'FIRSTMATE TEST - IGNORE (wedge-alarm channel verification)' \
  --sound request

Observed output:

{"id":"cli:notification:show","result":{"reason":"shown","shown":true,"type":"notification_show"}}

The safe command-channel contract is covered without a notification by tests/fm-daemon.test.sh: the summary reaches both $1 and stdin, every channel is process-group bounded, and a failed channel falls through.