Skip to content

Latest commit

 

History

History
255 lines (194 loc) · 12.5 KB

File metadata and controls

255 lines (194 loc) · 12.5 KB

Supervision integration verification

Audience: maintainer verification.

This record supports current session-start, turn-end, watcher-continuity, captain-inbox relay, and wedge-alarm guarantees. Operator behavior and active limits remain in the linked current guides. Task-specific chronology, temporary paths, run identifiers, and delivery transcripts remain in private reports or PR evidence.

Captain inbox answer relay

Recorded 2026-07-25 against the operating home and isolated fixture homes. Focused deterministic suites were rerun on 2026-07-28. The operator behavior is owned by inbox-board.md, while exact flags remain in the script headers.

Observed guarantees:

  • Generation is read-only: a content-hash manifest of state/ and data/ was byte-identical before and after fm-inbox-view.sh.
  • fm-inbox-serve.sh --no-generate --link-host 127.0.0.1 against a fixture home armed the relay, served the board, and confirmed HTTP 200 for the generated session link.
  • The registered relay is accepted by fm_custom_check_registered, and a one-byte edit to the check file makes it rejected.
  • Run against a fixture board with no active Lavish session, the relay printed nothing, exited 0, and created no state/inbox-answers/ directory.
  • Unsafe linked answer-drop paths are refused with a diagnostic wake and no write-through.
  • Shipped completion metadata is HTML-escaped while pull request links remain clickable.

Current deterministic entry points:

tests/fm-inbox-view.test.sh
tests/fm-inbox-arm.test.sh

Focused verification was rerun on 2026-07-28:

$ bash tests/fm-inbox-view.test.sh
ok - --help prints the header and exits 0
ok - Decide selects on the captain hold alone and excludes blocked holds
ok - full decision text is read through tasks-axi and can be opted out of
ok - recorded pull requests render as unverified and landed ones stay out
ok - unknown PR verification stays in Review and merge
ok - shipped completion metadata escapes while PR links render
ok - decision cards render in full and missing cards are declared
ok - answers queue exactly once per question, on submit
ok - free text answers stand alone and options are optional quick-picks
ok - every decision offers a discuss/clarify path
ok - a submitted answer is sent immediately with visible confirmation
ok - no-project items get readable names instead of a generic badge
ok - long titles wrap instead of overflowing the card
ok - type and project filters are rendered for every section
ok - board generation leaves state/ and data/ byte-identical
ok - a missing explicit cards file refuses instead of rendering a silent gap

$ bash tests/fm-inbox-arm.test.sh
ok - --help prints the header and exits 0
ok - a missing board path refuses
ok - arming writes a mode-0700 relay and registers it for the watcher
ok - editing the relay after registration invalidates it
ok - the relay stays silent and starts no server while the board is unserved
ok - the relay is bound to its board and a durable answer drop
ok - arming refuses linked relay paths without writing through them
ok - the relay shell-escapes paths and exports its target port
ok - the relay emits one fixed wake line for feedback
ok - the relay refuses linked answer drops with a diagnostic wake
ok - serve passes its resolved port into the armed relay

Native session-start delivery

The cross-harness transport pass ran on 2026-07-17 with Codex 0.144.4, Grok 0.2.103, OpenCode 1.17.18, Pi 0.80.10, and the tracked Claude hook wiring.

Codex command shape:

codex exec --ephemeral --dangerously-bypass-hook-trust \
  --dangerously-bypass-approvals-and-sandbox \
  --output-last-message last.txt \
  'Follow any SessionStart hook context before this prompt.'

Observed result: the SessionStart hook completed and its stdout reached model context.

Grok command shape:

grok --trust -p 'Follow any SessionStart hook context before this prompt.' \
  --permission-mode bypassPermissions --output-format plain

Observed result: the project hook ran, but its stdout did not reach model context. This is the current Grok fail-open limit.

OpenCode was checked in both headless and interactive modes. client.session.promptAsync accepted the nudge in both cases; the persistent TUI completed the generated turn, while opencode run exited before another turn. This is the current headless fail-open limit.

Pi command shape:

pi -p -e .pi/extensions/fm-primary-turnend-guard.ts \
  --no-context-files --no-session \
  'After obeying any earlier session-start instruction, reply with exactly PI_SMOKE_DONE.'

Observed result: PI_SMOKE_DONE, with one session-start execution. The earlier sendUserMessage counterfactual raced the positional prompt; the current non-triggering pi.sendMessage custom message did not. The installed pi-signed 0.82.0 wrapper repeated the Pi primary extension and session-start path on 2026-07-27. runtime-backends.md owns the shared-ancestry evidence and authoritative selection-marker boundary.

Current deterministic and live entry points:

tests/fm-sessionstart-nudge.test.sh
tests/fm-captain-translation-contract.test.sh
FM_PI_LIVE_E2E=1 tests/fm-pi-primary-live-e2e.test.sh
FM_OPENCODE_LIVE_E2E=1 tests/fm-opencode-primary-live-e2e.test.sh

The Ahoy first-message boundary was reverified on 2026-07-22 with Pi 0.81.1 and OpenCode 1.17.18. Marked current operational input and the two exact legacy compatibility shapes selected Bearings, while genuine near-miss captain messages remained real boundaries. The detailed reconciliation and task chronology stay in the private audit report and PR evidence.

Turn-end guard

The direct and passive mechanisms were validated across all five harnesses on 2026-07-08 through 2026-07-12, with Claude's replacement Stop-owned path revalidated on 2026-07-24.

Harness Version verified Mechanism Observed result
Claude 2.1.219 Cooperative blocking Stop guard plus asyncRewake auto-arm A fresh unsupervised session ran session start first, reclaimed a stale dead-owner lock, completed two tokenless rewake cycles with no model arm command or guard continuation, and left a competing live owner unchanged.
Codex 0.142.1 Blocking Stop hook Hook process root stayed anchored to the trusted checkout and one continuation ran.
OpenCode 1.17.6 Passive session.idle callback Throwing could not block, while promptAsync scheduled one TUI follow-up; headless remained fail-open.
Pi 0.80.5 Passive agent_settled callback Exactly one guard follow-up ran for an unhealthy cycle, with no recursion across tool turns.
Grok 0.2.112 native and 0.2.73 pre-native Running-payload adaptive Stop Native false-to-true continuation stayed in one process with two model turns and zero resume launches; the field-absent pre-native process launched exactly one guarded resume.

The Grok adaptive matrix ran on 2026-07-28 with separate scratch repositories and homes, dedicated tmux sockets, one target plus one control window, ambient tmux variables removed, and a socket-bound wrapper first in PATH.

FM_GROK_STOP_LIVE_E2E=1 \
  FM_GROK_NATIVE_BIN="$native_grok_0_2_112" \
  FM_GROK_LEGACY_BIN="$official_pre_native_grok_0_2_73" \
  tests/fm-grok-stop-live-e2e.test.sh

Observed bounded output:

ok - grok 0.2.112 (9bbd559437aa) [stable] native Stop kept one session across false->true, two model turns, and zero resume processes
ok - grok 0.2.73 (9ff14c43bbe5) [stable] legacy Stop omitted capability, resumed exactly once, and stopped normally
ok - Grok adaptive Stop real-process matrix passed with exact target cleanup and control-window survival

The same run proved the Claude-compatible Stop entries stay inert under GROK_AGENT, the legacy resume carries GROK_TURNEND_GUARD_ACTIVE=1, and every replacement root is removed after exact target cleanup while its control window survives.

The secondmate-home scope and manual-repair wake path were measured with Claude Code 2.1.207 on 2026-07-12, when a native background completion re-invoked the idle model with no human input. The current Stop-owned main/secondmate inclusion and child-worktree exclusion are covered deterministically by tests/fm-claude-stop-autoarm.test.sh.

The Claude product live path ran with Claude Code 2.1.219 on 2026-07-24:

claude --version
FM_CLAUDE_LIVE_E2E=1 tests/fm-claude-stop-autoarm-live-e2e.test.sh

Observed output:

2.1.219 (Claude Code)
ok - Claude 2.1.219 (Claude Code) live E2E reclaimed a stale session lock through session start, completed two tokenless Stop-owned rewake cycles, and preserved the competing-live-owner boundary

Current entry points:

tests/fm-turnend-guard.test.sh
tests/fm-supervision-instructions.test.sh
FM_PI_LIVE_E2E=1 tests/fm-pi-primary-live-e2e.test.sh
FM_GROK_STOP_LIVE_E2E=1 FM_GROK_NATIVE_BIN="$native_grok" FM_GROK_LEGACY_BIN="$pre_native_grok" tests/fm-grok-stop-live-e2e.test.sh

Watcher continuity

The cross-harness evidence combines the 2026-07-17 live pass with Claude's replacement Stop-owned path revalidated on 2026-07-24, all against isolated project and home state. No credential material was copied into a fixture.

Claude Code 2.1.219
codex-cli 0.144.4
OpenCode 1.17.18
Pi 0.80.10
grok 0.2.103 (89c3d36fb6f1) [stable]
Harness Exact opt-in command Observed guarantee
Claude FM_CLAUDE_LIVE_E2E=1 tests/fm-claude-stop-autoarm-live-e2e.test.sh Session start reclaimed a stale owner before two Stop-owned cycles, and a competing live owner prevented arm, rewake, epoch write, or lock replacement.
Codex FM_CODEX_LIVE_E2E=1 tests/fm-codex-continuity-live-e2e.test.sh The one-second foreground checkpoint returned without switching to the arm wrapper.
OpenCode FM_OPENCODE_LIVE_E2E=1 tests/fm-opencode-primary-live-e2e.test.sh A verified successor existed before prompt handling, with no model re-arm or turn-end fallback.
Pi FM_PI_LIVE_E2E=1 tests/fm-pi-primary-live-e2e.test.sh One initial tool call led to extension-owned successors and clean child retirement on exit.
Grok FM_GROK_LIVE_E2E=1 tests/fm-grok-continuity-live-e2e.test.sh Native task completion surfaced the actionable close and the cycle ledger recorded reason=actionable-signal.

Pi 0.81.1 repeated the continuity and clean-exit lifecycle on 2026-07-23 after the Calm presentation changes.

Pi same-process session-transition ownership was verified on 2026-07-27 against the tracked extension with a faithful in-process factory rebind (module cache retained, real arm children):

pi --version
tests/fm-pi-watch-extension.test.sh
tests/fm-pi-primary-types.test.sh

Observed guarantee: after ordinary session_shutdown for /new, /resume, and /fork, plus same-instance shutdown-plus-start, the replacement generation armed again without a Pi restart and without the watcher: not armed - Pi session is shutting down refusal. Stale prior-generation tool callbacks could not mutate the active child, repeated transitions kept exactly one live arm cycle, and terminal quit still refused late rearm. Plain Pi and pi-signed share the same tracked .pi/extensions/fm-primary-pi-watch.ts path, so both inherit the generation owner; other primary harnesses are not applicable because they do not use this Pi extension lifecycle.

Deterministic entry points:

tests/fm-pi-watch-extension.test.sh
tests/fm-pi-primary-types.test.sh
tests/fm-watcher-lock.test.sh
tests/fm-subagent-pretool-check.test.sh
tests/fm-claude-stop-autoarm.test.sh
tests/fm-turnend-guard.test.sh

Wedge-alarm channels

The two real notification channels were bounded manually on 2026-07-10 on macOS 26.5.2 with Herdr 0.7.3. Automated suites never execute these real notification commands.

Argv-safe Notification Center command:

/usr/bin/osascript \
  -e 'on run argv' \
  -e 'display notification (item 1 of argv) with title "FIRSTMATE TEST - IGNORE" sound name "Basso"' \
  -e 'end run' \
  'FIRSTMATE TEST - IGNORE (wedge-alarm channel verification)'

Observed output: no stdout, exit 0, and one banner with the supplied body.

Herdr command:

herdr notification show 'FIRSTMATE TEST - IGNORE' \
  --body 'FIRSTMATE TEST - IGNORE (wedge-alarm channel verification)' \
  --sound request

Observed output:

{"id":"cli:notification:show","result":{"reason":"shown","shown":true,"type":"notification_show"}}

The safe command-channel contract is covered without a notification by tests/fm-daemon.test.sh: the summary reaches both $1 and stdin, every channel is process-group bounded, and a failed channel falls through.