From 793de9eb10b0f1ea92be66b6131f4235f9de14fb Mon Sep 17 00:00:00 2001 From: Dongkeun Lee Date: Tue, 28 Jul 2026 10:55:10 -0400 Subject: [PATCH 1/5] docs: add safe Lavish repair skill --- .agents/skills/lavish-repair/SKILL.md | 166 ++++++++++++++++++++++++++ AGENTS.md | 1 + tests/lavish-repair.test.sh | 91 ++++++++++++++ 3 files changed, 258 insertions(+) create mode 100644 .agents/skills/lavish-repair/SKILL.md create mode 100755 tests/lavish-repair.test.sh diff --git a/.agents/skills/lavish-repair/SKILL.md b/.agents/skills/lavish-repair/SKILL.md new file mode 100644 index 0000000000..0746ff59e0 --- /dev/null +++ b/.agents/skills/lavish-repair/SKILL.md @@ -0,0 +1,166 @@ +--- +name: lavish-repair +description: >- + Agent-only recovery playbook for diagnosing and repairing served Lavish Editor boards that remain loading, lose live feedback, report that the agent is not listening, or prompt consideration of a Lavish or browser process restart. + Use before touching Lavish, Chrome, or chrome-devtools-axi processes during a surface incident. + Do not use it to design or revise a working decision board; use `lavish-decision-boards` for that. +user-invocable: false +metadata: + internal: true +--- + +# lavish-repair + +Prove the failing layer before changing any process. +Use `lavish-axi --help` and its command help as the authority for current serving and polling mechanics. +Consult the `data/learnings.md` content already included in the session-start digest for relevant prior incidents without violating the digest's read-once rule. + +## Fleet-wide process safety gate + +From a firstmate session, `pkill -f ` is a FLEET-WIDE DESTRUCTIVE COMMAND. +The `-f` matcher searches each process's entire command line, and a crewmate launcher carries its full brief in that command line. +An ordinary English phrase or tool name in a pattern can therefore select unrelated crew processes. + +Before sending any process signal: + +1. List candidates with `pgrep -fl ''` and read every row. +2. Inspect each candidate with `ps -p -o pid=,ppid=,etime=,command=`. +3. Reject every shell, harness, crewmate launcher, or process whose role is not proven. +4. Signal only one verified explicit PID with `kill `. +5. Re-list and verify the intended result before considering another signal. + +Never pipe unfiltered `pgrep` output into `kill`. +If an external control path leaves no explicit-PID option, constrain its matcher to a runtime-specific shape that ordinary prose cannot contain, such as the full resolved binary path plus exact operational arguments or a freshly generated `user-data-dir=/tmp/` path. +List and inspect candidates again immediately before acting, and abort if any crew or shell appears. +A full path copied into a brief is no longer a safe matcher. + +The concrete counter-example is the 2026-07-28 incident. +Firstmate ran `pkill -f 'chrome-devtools-axi'` after several crewmate briefs had mentioned that tool in ordinary instructions. +Fifteen of sixteen live crewmate panes died, and a Codex account was pushed to a re-authentication prompt. +The browser count was not proof of the failing layer, and the pattern matched the briefs rather than only browser bridges. + +## Diagnose in cheapest-first order + +Gather the source file and session URL for every affected board before changing anything. +Run every stage below and keep its evidence. + +### 1. HTTP serving + +Run this against each affected `/session/` URL: + +```sh +curl -s -o /dev/null -w '%{http_code} %{time_total}\n' '' +``` + +A fast local `200` proves that the HTTP serving path responds, not that the live channel works. +It moves the fault boundary to the live channel, listener, or browser. +Do not touch Chrome merely because the visible page looks slow. +A refusal, timeout, or non-`200` keeps the server layer in scope. + +### 2. Live-channel listener leak + +Check the default detached-server log: + +```sh +grep -c MaxListenersExceededWarning ~/.lavish-axi/server.log +``` + +When `LAVISH_AXI_STATE_DIR` is set, use its `server.log` as documented by `lavish-axi server --help`. +A nonzero count while multiple boards are simultaneously wedged is the known highest-probability live-channel failure. +The signature is warnings for 11 `reload`, `agent-reply`, and `agent-presence` listeners against the default EventEmitter limit of 10. +The log appends across restarts, so distinguish historical warnings from the current server before claiming an active leak. +Inspect recent warning lines and compare their `(node:)` value with the server PID verified under the repair procedure below. + +### 3. Listener presence + +List poll candidates: + +```sh +pgrep -fl 'lavish-axi.*poll' +``` + +Apply the process safety gate to the output. +For each affected board, verify an actual Node process whose command is `lavish-axi poll ` for that board's source path. +A shell or crewmate command that merely quotes those words is not a listener. +A board without a live poll is inert by construction because the page has no agent listener. + +### 4. Browser boundary + +Consider browser-side exhaustion only after HTTP is fast, the current server lacks the listener-leak signature, and the exact board has a live poll. +A large Chrome, headless, or bridge process count is a lead, not proof. +Use `chrome-devtools-axi --help` and the relevant command help to inspect the exact board page and establish the browser failure before changing browser state. + +## Repair the proven layer + +Use the least-destructive branch that matches the evidence. + +### Server absent or not serving + +Re-serve each source file without opening another browser window: + +```sh +lavish-axi '' --no-open +``` + +The command starts the local server when needed and prints the session URL. +Re-run the HTTP check, then attach a poll for every open board. +If serving still fails, use `lavish-axi server --help` and the server log for the startup diagnosis rather than touching Chrome. + +### Poll absent + +Start the missing listener and leave it running: + +```sh +lavish-axi poll '' +``` + +Long-poll silence is normal. +Confirm the actual poll process for that exact file with the listener-presence check. +Do not restart the server or browser when attaching the missing poll repairs the board. + +### Current server has the listener leak + +List possible server processes without signaling them: + +```sh +pgrep -fl 'lavish-axi/dist/cli.mjs server' +``` + +This selector can also appear inside a crewmate brief, so its output is discovery evidence only. +Inspect every candidate with the safety-gate `ps` command. +Confirm the intended Node process has the Lavish server entry point and expected `server --port ` arguments. +When useful, corroborate the listener PID for the expected port with `lsof -nP -iTCP: -sTCP:LISTEN`. +If the candidates do not resolve to exactly one intended server, stop without signaling anything. + +Terminate only the verified numeric server PID: + +```sh +kill +``` + +Re-serve every affected source file with `lavish-axi '' --no-open`. +Reattach `lavish-axi poll ''` for every board and leave each poll running. +Repeat the HTTP, current-log, and listener checks before reporting recovery. + +Session URLs survive a server restart when the same source files are re-served because Lavish derives session identity from each canonical source-file path and retains it in state. +Do not treat the restart as URL loss or replace working links preemptively. + +### Proven browser failure + +Protect any unsubmitted captain input before reloading or reopening a page. +Use current `chrome-devtools-axi` help to target the exact board page, and prefer repairing or reopening only that page with its existing session URL. +Do not kill `chrome-devtools-axi` processes by name. +If an exact browser process must be terminated, apply the fleet-wide safety gate and act only on a verified explicit PID with a run-specific binary or `user-data-dir` identity. +Recheck the same URL and poll connection after the browser-side repair. + +## What survives a crew kill + +A process-kill incident is serious but does not erase landed or worktree-backed work. + +- Crewmate reports and completion artifacts under `data//` survive. +- Leased worktrees, their working files, and their commits survive. +- Pushed branches and PRs survive remotely. +- Only the live agent process and its in-memory session are lost. + +Recover a dead crewmate by respawning it from its recorded worktree and brief. +Load `stuck-crewmate-recovery` and `harness-adapters` before performing that recovery. diff --git a/AGENTS.md b/AGENTS.md index 0b6959e1e7..9d2f14c243 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -822,6 +822,7 @@ These skills are not captain-invocable; they are conditional operating reference - `firstmate-coding-guidelines` - load before changing firstmate's shared, tracked material, as defined by section 1's list, whether editing directly or briefing a crewmate for a firstmate-repo task. - `memory-hygiene` - load before writing, rewriting, pruning, deduplicating, or otherwise leaning `data/captain.md` or `data/learnings.md`. - `lavish-decision-boards` - load before creating or revising a captain-facing Lavish board that asks the captain to make decisions or provide structured feedback. +- `lavish-repair` - load when a served Lavish board remains loading, cannot exchange live feedback, reports no agent listener, or before restarting Lavish or browser processes to repair a surface. - `eks-usage` - load before running `kubectl` or Amazon EKS commands, on an EKS IAM, authenticator, TLS, or connectivity error, or whenever the active cluster or context is uncertain. ## 14. X mode diff --git a/tests/lavish-repair.test.sh b/tests/lavish-repair.test.sh new file mode 100755 index 0000000000..8b42746fa9 --- /dev/null +++ b/tests/lavish-repair.test.sh @@ -0,0 +1,91 @@ +#!/usr/bin/env bash +# Behavioral contract for the lavish-repair skill's routing, diagnosis order, +# fleet-wide kill guard, recovery sequence, and durable-state expectations. +set -euo pipefail + +ROOT=$(cd "$(dirname "$0")/.." && pwd) +SKILL="$ROOT/.agents/skills/lavish-repair/SKILL.md" +AGENTS="$ROOT/AGENTS.md" + +fail() { + printf 'FAIL: %s\n' "$1" >&2 + exit 1 +} + +pass() { + printf 'PASS: %s\n' "$1" +} + +assert_grep() { + local pattern=$1 + local file=$2 + local message=$3 + grep -Fq -- "$pattern" "$file" || fail "$message" +} + +test_internal_skill_and_trigger() { + local section global_count section_count + + [ -f "$SKILL" ] || fail "lavish-repair SKILL.md is missing" + assert_grep "name: lavish-repair" "$SKILL" "skill name is missing" + assert_grep "user-invocable: false" "$SKILL" "skill must not be captain-invocable" + assert_grep "metadata:" "$SKILL" "skill is missing internal metadata" + assert_grep "internal: true" "$SKILL" "skill is not marked internal" + + section=$(awk '/^## 13\. Agent-only reference skills$/ { capture=1; next } capture && /^## / { exit } capture' "$AGENTS") + global_count=$(grep -Fc '`lavish-repair`' "$AGENTS") + section_count=$(printf '%s\n' "$section" | grep -Fc '`lavish-repair`') + [ "$global_count" -eq 1 ] || fail "AGENTS.md must reference lavish-repair exactly once" + [ "$section_count" -eq 1 ] || fail "lavish-repair trigger must live in section 13" + assert_grep "served Lavish board remains loading" "$AGENTS" "trigger must cover a visibly wedged surface" + assert_grep "before restarting Lavish or browser processes" "$AGENTS" "trigger must fire before process repair" + pass "lavish-repair is internal and has one conditional trigger" +} + +test_cheapest_first_diagnosis() { + local http_line leak_line listener_line browser_line + + http_line=$(grep -nF '### 1. HTTP serving' "$SKILL" | cut -d: -f1) + leak_line=$(grep -nF '### 2. Live-channel listener leak' "$SKILL" | cut -d: -f1) + listener_line=$(grep -nF '### 3. Listener presence' "$SKILL" | cut -d: -f1) + browser_line=$(grep -nF '### 4. Browser boundary' "$SKILL" | cut -d: -f1) + [ "$http_line" -lt "$leak_line" ] || fail "HTTP must be diagnosed before the live channel" + [ "$leak_line" -lt "$listener_line" ] || fail "listener leak must be diagnosed before poll presence" + [ "$listener_line" -lt "$browser_line" ] || fail "browser diagnosis must remain last" + + assert_grep "curl -s -o /dev/null -w '%{http_code} %{time_total}\\n'" "$SKILL" "HTTP probe is missing" + assert_grep "grep -c MaxListenersExceededWarning ~/.lavish-axi/server.log" "$SKILL" "listener-leak probe is missing" + assert_grep "pgrep -fl 'lavish-axi.*poll'" "$SKILL" "poll-listener probe is missing" + assert_grep "A large Chrome, headless, or bridge process count is a lead, not proof" "$SKILL" "browser counts must not be treated as proof" + pass "lavish-repair preserves cheapest-first layer diagnosis" +} + +test_fleet_wide_kill_guard() { + assert_grep '`pkill -f ` is a FLEET-WIDE DESTRUCTIVE COMMAND' "$SKILL" "fleet-wide pkill warning is missing" + assert_grep "pgrep -fl ''" "$SKILL" "candidate listing is missing" + assert_grep 'ps -p -o pid=,ppid=,etime=,command=' "$SKILL" "candidate argv inspection is missing" + assert_grep 'Signal only one verified explicit PID with `kill `' "$SKILL" "explicit-PID signaling is missing" + assert_grep 'Never pipe unfiltered `pgrep` output into `kill`' "$SKILL" "unfiltered process signaling is not forbidden" + assert_grep 'Fifteen of sixteen live crewmate panes died' "$SKILL" "the concrete fleet-loss counter-example is missing" + assert_grep "pkill -f 'chrome-devtools-axi'" "$SKILL" "the dangerous concrete pattern is missing" + assert_grep 'user-data-dir=/tmp/' "$SKILL" "run-specific unavoidable-pattern guidance is missing" + pass "lavish-repair retains the fleet-wide destructive-command guard" +} + +test_repair_and_survival_contracts() { + assert_grep "pgrep -fl 'lavish-axi/dist/cli.mjs server'" "$SKILL" "server candidate discovery is missing" + assert_grep 'kill ' "$SKILL" "verified server termination is missing" + assert_grep "lavish-axi '' --no-open" "$SKILL" "server re-serve command is missing" + assert_grep "lavish-axi poll ''" "$SKILL" "listener reattachment is missing" + assert_grep 'Session URLs survive a server restart' "$SKILL" "stable session-URL behavior is missing" + assert_grep 'Crewmate reports and completion artifacts under `data//` survive' "$SKILL" "report survival is missing" + assert_grep 'Leased worktrees, their working files, and their commits survive' "$SKILL" "worktree survival is missing" + assert_grep 'Pushed branches and PRs survive remotely' "$SKILL" "remote work survival is missing" + assert_grep 'Only the live agent process and its in-memory session are lost' "$SKILL" "live-session loss boundary is missing" + pass "lavish-repair retains the repair and durable-state contracts" +} + +test_internal_skill_and_trigger +test_cheapest_first_diagnosis +test_fleet_wide_kill_guard +test_repair_and_survival_contracts From 6518920bc4f56e93fe41c208e23061dd5ad60215 Mon Sep 17 00:00:00 2001 From: Dongkeun Lee Date: Tue, 28 Jul 2026 12:21:27 -0400 Subject: [PATCH 2/5] no-mistakes(lint): Fix ShellCheck quoting in lavish repair assertions --- tests/lavish-repair.test.sh | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/tests/lavish-repair.test.sh b/tests/lavish-repair.test.sh index 8b42746fa9..529f058f5a 100755 --- a/tests/lavish-repair.test.sh +++ b/tests/lavish-repair.test.sh @@ -33,8 +33,8 @@ test_internal_skill_and_trigger() { assert_grep "internal: true" "$SKILL" "skill is not marked internal" section=$(awk '/^## 13\. Agent-only reference skills$/ { capture=1; next } capture && /^## / { exit } capture' "$AGENTS") - global_count=$(grep -Fc '`lavish-repair`' "$AGENTS") - section_count=$(printf '%s\n' "$section" | grep -Fc '`lavish-repair`') + global_count=$(grep -Fc "\`lavish-repair\`" "$AGENTS") + section_count=$(printf '%s\n' "$section" | grep -Fc "\`lavish-repair\`") [ "$global_count" -eq 1 ] || fail "AGENTS.md must reference lavish-repair exactly once" [ "$section_count" -eq 1 ] || fail "lavish-repair trigger must live in section 13" assert_grep "served Lavish board remains loading" "$AGENTS" "trigger must cover a visibly wedged surface" @@ -61,11 +61,11 @@ test_cheapest_first_diagnosis() { } test_fleet_wide_kill_guard() { - assert_grep '`pkill -f ` is a FLEET-WIDE DESTRUCTIVE COMMAND' "$SKILL" "fleet-wide pkill warning is missing" + assert_grep "\`pkill -f \` is a FLEET-WIDE DESTRUCTIVE COMMAND" "$SKILL" "fleet-wide pkill warning is missing" assert_grep "pgrep -fl ''" "$SKILL" "candidate listing is missing" assert_grep 'ps -p -o pid=,ppid=,etime=,command=' "$SKILL" "candidate argv inspection is missing" - assert_grep 'Signal only one verified explicit PID with `kill `' "$SKILL" "explicit-PID signaling is missing" - assert_grep 'Never pipe unfiltered `pgrep` output into `kill`' "$SKILL" "unfiltered process signaling is not forbidden" + assert_grep "Signal only one verified explicit PID with \`kill \`" "$SKILL" "explicit-PID signaling is missing" + assert_grep "Never pipe unfiltered \`pgrep\` output into \`kill\`" "$SKILL" "unfiltered process signaling is not forbidden" assert_grep 'Fifteen of sixteen live crewmate panes died' "$SKILL" "the concrete fleet-loss counter-example is missing" assert_grep "pkill -f 'chrome-devtools-axi'" "$SKILL" "the dangerous concrete pattern is missing" assert_grep 'user-data-dir=/tmp/' "$SKILL" "run-specific unavoidable-pattern guidance is missing" @@ -78,7 +78,7 @@ test_repair_and_survival_contracts() { assert_grep "lavish-axi '' --no-open" "$SKILL" "server re-serve command is missing" assert_grep "lavish-axi poll ''" "$SKILL" "listener reattachment is missing" assert_grep 'Session URLs survive a server restart' "$SKILL" "stable session-URL behavior is missing" - assert_grep 'Crewmate reports and completion artifacts under `data//` survive' "$SKILL" "report survival is missing" + assert_grep "Crewmate reports and completion artifacts under \`data//\` survive" "$SKILL" "report survival is missing" assert_grep 'Leased worktrees, their working files, and their commits survive' "$SKILL" "worktree survival is missing" assert_grep 'Pushed branches and PRs survive remotely' "$SKILL" "remote work survival is missing" assert_grep 'Only the live agent process and its in-memory session are lost' "$SKILL" "live-session loss boundary is missing" From fd94a5dc52dca8b31ef109795915ad740f71a810 Mon Sep 17 00:00:00 2001 From: Dongkeun Lee Date: Tue, 28 Jul 2026 12:43:40 -0400 Subject: [PATCH 3/5] no-mistakes: apply CI fixes --- tests/lavish-repair.test.sh | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/tests/lavish-repair.test.sh b/tests/lavish-repair.test.sh index 529f058f5a..80be7e5da7 100755 --- a/tests/lavish-repair.test.sh +++ b/tests/lavish-repair.test.sh @@ -45,10 +45,14 @@ test_internal_skill_and_trigger() { test_cheapest_first_diagnosis() { local http_line leak_line listener_line browser_line - http_line=$(grep -nF '### 1. HTTP serving' "$SKILL" | cut -d: -f1) - leak_line=$(grep -nF '### 2. Live-channel listener leak' "$SKILL" | cut -d: -f1) - listener_line=$(grep -nF '### 3. Listener presence' "$SKILL" | cut -d: -f1) - browser_line=$(grep -nF '### 4. Browser boundary' "$SKILL" | cut -d: -f1) + http_line=$(grep -nF '### 1. HTTP serving' "$SKILL" | cut -d: -f1) || + fail "HTTP diagnosis stage is missing" + leak_line=$(grep -nF '### 2. Live-channel listener leak' "$SKILL" | cut -d: -f1) || + fail "live-channel diagnosis stage is missing" + listener_line=$(grep -nF '### 3. Listener presence' "$SKILL" | cut -d: -f1) || + fail "listener-presence diagnosis stage is missing" + browser_line=$(grep -nF '### 4. Browser boundary' "$SKILL" | cut -d: -f1) || + fail "browser diagnosis stage is missing" [ "$http_line" -lt "$leak_line" ] || fail "HTTP must be diagnosed before the live channel" [ "$leak_line" -lt "$listener_line" ] || fail "listener leak must be diagnosed before poll presence" [ "$listener_line" -lt "$browser_line" ] || fail "browser diagnosis must remain last" From b634bf46869e6a94277ca8414876ec1735e73aa4 Mon Sep 17 00:00:00 2001 From: Dongkeun Lee Date: Tue, 28 Jul 2026 13:05:17 -0400 Subject: [PATCH 4/5] no-mistakes: apply CI fixes --- tests/fm-account-routing.test.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/fm-account-routing.test.sh b/tests/fm-account-routing.test.sh index 8bb02b29a9..37680e977c 100755 --- a/tests/fm-account-routing.test.sh +++ b/tests/fm-account-routing.test.sh @@ -2225,8 +2225,8 @@ test_enforced_secondmate_requires_routing_inheritance_and_capable_home() { out=$(FM_TEST_PANE_PATH="$sm" run_spawn "$id" "$sm" --secondmate) status=$? [ "$status" -ne 0 ] || fail "enforced secondmate launched from a pre-Agent-Fleet home" - assert_contains "$out" "$id" "capability refusal omitted the offending secondmate" - assert_contains "$out" "dirty working tree" "capability refusal did not stop at the freshness gate" + assert_contains "$out" "$sm" "capability refusal omitted the offending secondmate home" + assert_contains "$out" "lacks Agent Fleet routing support" "capability refusal omitted the failed capability" assert_not_grep '^new-window ' "$TMUX_LOG" "capability refusal created an endpoint" pass "enforced secondmates require inherited routing policy and Agent Fleet-capable homes" } From 8e6030c4a7bb738aa66d8fe4426619cad7d36fea Mon Sep 17 00:00:00 2001 From: Dongkeun Lee Date: Tue, 28 Jul 2026 13:21:42 -0400 Subject: [PATCH 5/5] Remove unrelated routing test assertions --- tests/fm-account-routing.test.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/fm-account-routing.test.sh b/tests/fm-account-routing.test.sh index 37680e977c..8bb02b29a9 100755 --- a/tests/fm-account-routing.test.sh +++ b/tests/fm-account-routing.test.sh @@ -2225,8 +2225,8 @@ test_enforced_secondmate_requires_routing_inheritance_and_capable_home() { out=$(FM_TEST_PANE_PATH="$sm" run_spawn "$id" "$sm" --secondmate) status=$? [ "$status" -ne 0 ] || fail "enforced secondmate launched from a pre-Agent-Fleet home" - assert_contains "$out" "$sm" "capability refusal omitted the offending secondmate home" - assert_contains "$out" "lacks Agent Fleet routing support" "capability refusal omitted the failed capability" + assert_contains "$out" "$id" "capability refusal omitted the offending secondmate" + assert_contains "$out" "dirty working tree" "capability refusal did not stop at the freshness gate" assert_not_grep '^new-window ' "$TMUX_LOG" "capability refusal created an endpoint" pass "enforced secondmates require inherited routing policy and Agent Fleet-capable homes" }