Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
166 changes: 166 additions & 0 deletions .agents/skills/lavish-repair/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,166 @@
---
name: lavish-repair
description: >-
Agent-only recovery playbook for diagnosing and repairing served Lavish Editor boards that remain loading, lose live feedback, report that the agent is not listening, or prompt consideration of a Lavish or browser process restart.
Use before touching Lavish, Chrome, or chrome-devtools-axi processes during a surface incident.
Do not use it to design or revise a working decision board; use `lavish-decision-boards` for that.
user-invocable: false
metadata:
internal: true
---

# lavish-repair

Prove the failing layer before changing any process.
Use `lavish-axi --help` and its command help as the authority for current serving and polling mechanics.
Consult the `data/learnings.md` content already included in the session-start digest for relevant prior incidents without violating the digest's read-once rule.

## Fleet-wide process safety gate

From a firstmate session, `pkill -f <pattern>` is a FLEET-WIDE DESTRUCTIVE COMMAND.
The `-f` matcher searches each process's entire command line, and a crewmate launcher carries its full brief in that command line.
An ordinary English phrase or tool name in a pattern can therefore select unrelated crew processes.

Before sending any process signal:

1. List candidates with `pgrep -fl '<candidate-pattern>'` and read every row.
2. Inspect each candidate with `ps -p <pid> -o pid=,ppid=,etime=,command=`.
3. Reject every shell, harness, crewmate launcher, or process whose role is not proven.
4. Signal only one verified explicit PID with `kill <pid>`.
5. Re-list and verify the intended result before considering another signal.

Never pipe unfiltered `pgrep` output into `kill`.
If an external control path leaves no explicit-PID option, constrain its matcher to a runtime-specific shape that ordinary prose cannot contain, such as the full resolved binary path plus exact operational arguments or a freshly generated `user-data-dir=/tmp/<opaque-id>` path.
List and inspect candidates again immediately before acting, and abort if any crew or shell appears.
A full path copied into a brief is no longer a safe matcher.

The concrete counter-example is the 2026-07-28 incident.
Firstmate ran `pkill -f 'chrome-devtools-axi'` after several crewmate briefs had mentioned that tool in ordinary instructions.
Fifteen of sixteen live crewmate panes died, and a Codex account was pushed to a re-authentication prompt.
The browser count was not proof of the failing layer, and the pattern matched the briefs rather than only browser bridges.

## Diagnose in cheapest-first order

Gather the source file and session URL for every affected board before changing anything.
Run every stage below and keep its evidence.

### 1. HTTP serving

Run this against each affected `/session/<id>` URL:

```sh
curl -s -o /dev/null -w '%{http_code} %{time_total}\n' '<session-url>'
```

A fast local `200` proves that the HTTP serving path responds, not that the live channel works.
It moves the fault boundary to the live channel, listener, or browser.
Do not touch Chrome merely because the visible page looks slow.
A refusal, timeout, or non-`200` keeps the server layer in scope.

### 2. Live-channel listener leak

Check the default detached-server log:

```sh
grep -c MaxListenersExceededWarning ~/.lavish-axi/server.log
```

When `LAVISH_AXI_STATE_DIR` is set, use its `server.log` as documented by `lavish-axi server --help`.
A nonzero count while multiple boards are simultaneously wedged is the known highest-probability live-channel failure.
The signature is warnings for 11 `reload`, `agent-reply`, and `agent-presence` listeners against the default EventEmitter limit of 10.
The log appends across restarts, so distinguish historical warnings from the current server before claiming an active leak.
Inspect recent warning lines and compare their `(node:<pid>)` value with the server PID verified under the repair procedure below.

### 3. Listener presence

List poll candidates:

```sh
pgrep -fl 'lavish-axi.*poll'
```

Apply the process safety gate to the output.
For each affected board, verify an actual Node process whose command is `lavish-axi poll <source-file>` for that board's source path.
A shell or crewmate command that merely quotes those words is not a listener.
A board without a live poll is inert by construction because the page has no agent listener.

### 4. Browser boundary

Consider browser-side exhaustion only after HTTP is fast, the current server lacks the listener-leak signature, and the exact board has a live poll.
A large Chrome, headless, or bridge process count is a lead, not proof.
Use `chrome-devtools-axi --help` and the relevant command help to inspect the exact board page and establish the browser failure before changing browser state.

## Repair the proven layer

Use the least-destructive branch that matches the evidence.

### Server absent or not serving

Re-serve each source file without opening another browser window:

```sh
lavish-axi '<source-file>' --no-open
```

The command starts the local server when needed and prints the session URL.
Re-run the HTTP check, then attach a poll for every open board.
If serving still fails, use `lavish-axi server --help` and the server log for the startup diagnosis rather than touching Chrome.

### Poll absent

Start the missing listener and leave it running:

```sh
lavish-axi poll '<source-file>'
```

Long-poll silence is normal.
Confirm the actual poll process for that exact file with the listener-presence check.
Do not restart the server or browser when attaching the missing poll repairs the board.

### Current server has the listener leak

List possible server processes without signaling them:

```sh
pgrep -fl 'lavish-axi/dist/cli.mjs server'
```

This selector can also appear inside a crewmate brief, so its output is discovery evidence only.
Inspect every candidate with the safety-gate `ps` command.
Confirm the intended Node process has the Lavish server entry point and expected `server --port <port>` arguments.
When useful, corroborate the listener PID for the expected port with `lsof -nP -iTCP:<port> -sTCP:LISTEN`.
If the candidates do not resolve to exactly one intended server, stop without signaling anything.

Terminate only the verified numeric server PID:

```sh
kill <server-pid>
```

Re-serve every affected source file with `lavish-axi '<source-file>' --no-open`.
Reattach `lavish-axi poll '<source-file>'` for every board and leave each poll running.
Repeat the HTTP, current-log, and listener checks before reporting recovery.

Session URLs survive a server restart when the same source files are re-served because Lavish derives session identity from each canonical source-file path and retains it in state.
Do not treat the restart as URL loss or replace working links preemptively.

### Proven browser failure

Protect any unsubmitted captain input before reloading or reopening a page.
Use current `chrome-devtools-axi` help to target the exact board page, and prefer repairing or reopening only that page with its existing session URL.
Do not kill `chrome-devtools-axi` processes by name.
If an exact browser process must be terminated, apply the fleet-wide safety gate and act only on a verified explicit PID with a run-specific binary or `user-data-dir` identity.
Recheck the same URL and poll connection after the browser-side repair.

## What survives a crew kill

A process-kill incident is serious but does not erase landed or worktree-backed work.

- Crewmate reports and completion artifacts under `data/<id>/` survive.
- Leased worktrees, their working files, and their commits survive.
- Pushed branches and PRs survive remotely.
- Only the live agent process and its in-memory session are lost.

Recover a dead crewmate by respawning it from its recorded worktree and brief.
Load `stuck-crewmate-recovery` and `harness-adapters` before performing that recovery.
1 change: 1 addition & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -822,6 +822,7 @@ These skills are not captain-invocable; they are conditional operating reference
- `firstmate-coding-guidelines` - load before changing firstmate's shared, tracked material, as defined by section 1's list, whether editing directly or briefing a crewmate for a firstmate-repo task.
- `memory-hygiene` - load before writing, rewriting, pruning, deduplicating, or otherwise leaning `data/captain.md` or `data/learnings.md`.
- `lavish-decision-boards` - load before creating or revising a captain-facing Lavish board that asks the captain to make decisions or provide structured feedback.
- `lavish-repair` - load when a served Lavish board remains loading, cannot exchange live feedback, reports no agent listener, or before restarting Lavish or browser processes to repair a surface.
- `eks-usage` - load before running `kubectl` or Amazon EKS commands, on an EKS IAM, authenticator, TLS, or connectivity error, or whenever the active cluster or context is uncertain.

## 14. X mode
Expand Down
95 changes: 95 additions & 0 deletions tests/lavish-repair.test.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
#!/usr/bin/env bash
# Behavioral contract for the lavish-repair skill's routing, diagnosis order,
# fleet-wide kill guard, recovery sequence, and durable-state expectations.
set -euo pipefail

ROOT=$(cd "$(dirname "$0")/.." && pwd)
SKILL="$ROOT/.agents/skills/lavish-repair/SKILL.md"
AGENTS="$ROOT/AGENTS.md"

fail() {
printf 'FAIL: %s\n' "$1" >&2
exit 1
}

pass() {
printf 'PASS: %s\n' "$1"
}

assert_grep() {
local pattern=$1
local file=$2
local message=$3
grep -Fq -- "$pattern" "$file" || fail "$message"
}

test_internal_skill_and_trigger() {
local section global_count section_count

[ -f "$SKILL" ] || fail "lavish-repair SKILL.md is missing"
assert_grep "name: lavish-repair" "$SKILL" "skill name is missing"
assert_grep "user-invocable: false" "$SKILL" "skill must not be captain-invocable"
assert_grep "metadata:" "$SKILL" "skill is missing internal metadata"
assert_grep "internal: true" "$SKILL" "skill is not marked internal"

section=$(awk '/^## 13\. Agent-only reference skills$/ { capture=1; next } capture && /^## / { exit } capture' "$AGENTS")
global_count=$(grep -Fc "\`lavish-repair\`" "$AGENTS")
section_count=$(printf '%s\n' "$section" | grep -Fc "\`lavish-repair\`")
[ "$global_count" -eq 1 ] || fail "AGENTS.md must reference lavish-repair exactly once"
[ "$section_count" -eq 1 ] || fail "lavish-repair trigger must live in section 13"
assert_grep "served Lavish board remains loading" "$AGENTS" "trigger must cover a visibly wedged surface"
assert_grep "before restarting Lavish or browser processes" "$AGENTS" "trigger must fire before process repair"
pass "lavish-repair is internal and has one conditional trigger"
}

test_cheapest_first_diagnosis() {
local http_line leak_line listener_line browser_line

http_line=$(grep -nF '### 1. HTTP serving' "$SKILL" | cut -d: -f1) ||
fail "HTTP diagnosis stage is missing"
leak_line=$(grep -nF '### 2. Live-channel listener leak' "$SKILL" | cut -d: -f1) ||
fail "live-channel diagnosis stage is missing"
listener_line=$(grep -nF '### 3. Listener presence' "$SKILL" | cut -d: -f1) ||
fail "listener-presence diagnosis stage is missing"
browser_line=$(grep -nF '### 4. Browser boundary' "$SKILL" | cut -d: -f1) ||
fail "browser diagnosis stage is missing"
[ "$http_line" -lt "$leak_line" ] || fail "HTTP must be diagnosed before the live channel"
[ "$leak_line" -lt "$listener_line" ] || fail "listener leak must be diagnosed before poll presence"
[ "$listener_line" -lt "$browser_line" ] || fail "browser diagnosis must remain last"

assert_grep "curl -s -o /dev/null -w '%{http_code} %{time_total}\\n'" "$SKILL" "HTTP probe is missing"
assert_grep "grep -c MaxListenersExceededWarning ~/.lavish-axi/server.log" "$SKILL" "listener-leak probe is missing"
assert_grep "pgrep -fl 'lavish-axi.*poll'" "$SKILL" "poll-listener probe is missing"
assert_grep "A large Chrome, headless, or bridge process count is a lead, not proof" "$SKILL" "browser counts must not be treated as proof"
pass "lavish-repair preserves cheapest-first layer diagnosis"
}

test_fleet_wide_kill_guard() {
assert_grep "\`pkill -f <pattern>\` is a FLEET-WIDE DESTRUCTIVE COMMAND" "$SKILL" "fleet-wide pkill warning is missing"
assert_grep "pgrep -fl '<candidate-pattern>'" "$SKILL" "candidate listing is missing"
assert_grep 'ps -p <pid> -o pid=,ppid=,etime=,command=' "$SKILL" "candidate argv inspection is missing"
assert_grep "Signal only one verified explicit PID with \`kill <pid>\`" "$SKILL" "explicit-PID signaling is missing"
assert_grep "Never pipe unfiltered \`pgrep\` output into \`kill\`" "$SKILL" "unfiltered process signaling is not forbidden"
assert_grep 'Fifteen of sixteen live crewmate panes died' "$SKILL" "the concrete fleet-loss counter-example is missing"
assert_grep "pkill -f 'chrome-devtools-axi'" "$SKILL" "the dangerous concrete pattern is missing"
assert_grep 'user-data-dir=/tmp/<opaque-id>' "$SKILL" "run-specific unavoidable-pattern guidance is missing"
pass "lavish-repair retains the fleet-wide destructive-command guard"
}

test_repair_and_survival_contracts() {
assert_grep "pgrep -fl 'lavish-axi/dist/cli.mjs server'" "$SKILL" "server candidate discovery is missing"
assert_grep 'kill <server-pid>' "$SKILL" "verified server termination is missing"
assert_grep "lavish-axi '<source-file>' --no-open" "$SKILL" "server re-serve command is missing"
assert_grep "lavish-axi poll '<source-file>'" "$SKILL" "listener reattachment is missing"
assert_grep 'Session URLs survive a server restart' "$SKILL" "stable session-URL behavior is missing"
assert_grep "Crewmate reports and completion artifacts under \`data/<id>/\` survive" "$SKILL" "report survival is missing"
assert_grep 'Leased worktrees, their working files, and their commits survive' "$SKILL" "worktree survival is missing"
assert_grep 'Pushed branches and PRs survive remotely' "$SKILL" "remote work survival is missing"
assert_grep 'Only the live agent process and its in-memory session are lost' "$SKILL" "live-session loss boundary is missing"
pass "lavish-repair retains the repair and durable-state contracts"
}

test_internal_skill_and_trigger
test_cheapest_first_diagnosis
test_fleet_wide_kill_guard
test_repair_and_survival_contracts
Loading