|
| 1 | +--- |
| 2 | +name: lavish-repair |
| 3 | +description: >- |
| 4 | + Agent-only recovery playbook for diagnosing and repairing served Lavish Editor boards that remain loading, lose live feedback, report that the agent is not listening, or prompt consideration of a Lavish or browser process restart. |
| 5 | + Use before touching Lavish, Chrome, or chrome-devtools-axi processes during a surface incident. |
| 6 | + Do not use it to design or revise a working decision; use `lavish-decisions` for that. |
| 7 | +user-invocable: false |
| 8 | +metadata: |
| 9 | + internal: true |
| 10 | +--- |
| 11 | + |
| 12 | +# lavish-repair |
| 13 | + |
| 14 | +Prove the failing layer before changing any process. |
| 15 | +Use `lavish-axi --help` and its command help as the authority for current serving and polling mechanics. |
| 16 | +Consult the `data/learnings.md` content already included in the session-start digest for relevant prior incidents without violating the digest's read-once rule. |
| 17 | + |
| 18 | +## Fleet-wide process safety gate |
| 19 | + |
| 20 | +From a firstmate session, `pkill -f <pattern>` is a FLEET-WIDE DESTRUCTIVE COMMAND. |
| 21 | +The `-f` matcher searches each process's entire command line, and a crewmate launcher carries its full brief in that command line. |
| 22 | +An ordinary English phrase or tool name in a pattern can therefore select unrelated crew processes. |
| 23 | + |
| 24 | +Before sending any process signal: |
| 25 | + |
| 26 | +1. List candidates with `pgrep -fl '<candidate-pattern>'` and read every row. |
| 27 | +2. Inspect each candidate with `ps -p <pid> -o pid=,ppid=,etime=,command=`. |
| 28 | +3. Reject every shell, harness, crewmate launcher, or process whose role is not proven. |
| 29 | +4. Signal only one verified explicit PID with `kill <pid>`. |
| 30 | +5. Re-list and verify the intended result before considering another signal. |
| 31 | + |
| 32 | +Never pipe unfiltered `pgrep` output into `kill`. |
| 33 | +If an external control path leaves no explicit-PID option, constrain its matcher to a runtime-specific shape that ordinary prose cannot contain, such as the full resolved binary path plus exact operational arguments or a freshly generated `user-data-dir=/tmp/<opaque-id>` path. |
| 34 | +List and inspect candidates again immediately before acting, and abort if any crew or shell appears. |
| 35 | +A full path copied into a brief is no longer a safe matcher. |
| 36 | + |
| 37 | +The concrete counter-example is the 2026-07-28 incident. |
| 38 | +Firstmate ran `pkill -f 'chrome-devtools-axi'` after several crewmate briefs had mentioned that tool in ordinary instructions. |
| 39 | +Fifteen of sixteen live crewmate panes died, and a Codex account was pushed to a re-authentication prompt. |
| 40 | +The browser count was not proof of the failing layer, and the pattern matched the briefs rather than only browser bridges. |
| 41 | + |
| 42 | +## Diagnose in cheapest-first order |
| 43 | + |
| 44 | +Gather the source file and session URL for every affected board before changing anything. |
| 45 | +Run every stage below and keep its evidence. |
| 46 | + |
| 47 | +### 1. HTTP serving |
| 48 | + |
| 49 | +Run this against each affected `/session/<id>` URL: |
| 50 | + |
| 51 | +```sh |
| 52 | +curl -s -o /dev/null -w '%{http_code} %{time_total}\n' '<session-url>' |
| 53 | +``` |
| 54 | + |
| 55 | +A fast local `200` proves that the HTTP serving path responds, not that the live channel works. |
| 56 | +It moves the fault boundary to the live channel, listener, or browser. |
| 57 | +Do not touch Chrome merely because the visible page looks slow. |
| 58 | +A refusal, timeout, or non-`200` keeps the server layer in scope. |
| 59 | + |
| 60 | +### 2. Live-channel listener leak |
| 61 | + |
| 62 | +Check the default detached-server log: |
| 63 | + |
| 64 | +```sh |
| 65 | +grep -c MaxListenersExceededWarning ~/.lavish-axi/server.log |
| 66 | +``` |
| 67 | + |
| 68 | +When `LAVISH_AXI_STATE_DIR` is set, use its `server.log` as documented by `lavish-axi server --help`. |
| 69 | +A nonzero count while multiple boards are simultaneously wedged is the known highest-probability live-channel failure. |
| 70 | +The signature is warnings for 11 `reload`, `agent-reply`, and `agent-presence` listeners against the default EventEmitter limit of 10. |
| 71 | +The log appends across restarts, so distinguish historical warnings from the current server before claiming an active leak. |
| 72 | +Inspect recent warning lines and compare their `(node:<pid>)` value with the server PID verified under the repair procedure below. |
| 73 | + |
| 74 | +### 3. Listener presence |
| 75 | + |
| 76 | +List poll candidates: |
| 77 | + |
| 78 | +```sh |
| 79 | +pgrep -fl 'lavish-axi.*poll' |
| 80 | +``` |
| 81 | + |
| 82 | +Apply the process safety gate to the output. |
| 83 | +For each affected board, verify an actual Node process whose command is `lavish-axi poll <source-file>` for that board's source path. |
| 84 | +A shell or crewmate command that merely quotes those words is not a listener. |
| 85 | +A board without a live poll is inert by construction because the page has no agent listener. |
| 86 | + |
| 87 | +### 4. Browser boundary |
| 88 | + |
| 89 | +Consider browser-side exhaustion only after HTTP is fast, the current server lacks the listener-leak signature, and the exact board has a live poll. |
| 90 | +A large Chrome, headless, or bridge process count is a lead, not proof. |
| 91 | +Use `chrome-devtools-axi --help` and the relevant command help to inspect the exact board page and establish the browser failure before changing browser state. |
| 92 | + |
| 93 | +## Repair the proven layer |
| 94 | + |
| 95 | +Use the least-destructive branch that matches the evidence. |
| 96 | + |
| 97 | +### Server absent or not serving |
| 98 | + |
| 99 | +Re-serve each source file without opening another browser window: |
| 100 | + |
| 101 | +```sh |
| 102 | +lavish-axi '<source-file>' --no-open |
| 103 | +``` |
| 104 | + |
| 105 | +The command starts the local server when needed and prints the session URL. |
| 106 | +Re-run the HTTP check, then attach a poll for every open board. |
| 107 | +If serving still fails, use `lavish-axi server --help` and the server log for the startup diagnosis rather than touching Chrome. |
| 108 | + |
| 109 | +### Poll absent |
| 110 | + |
| 111 | +Start the missing listener and leave it running: |
| 112 | + |
| 113 | +```sh |
| 114 | +lavish-axi poll '<source-file>' |
| 115 | +``` |
| 116 | + |
| 117 | +Long-poll silence is normal. |
| 118 | +Confirm the actual poll process for that exact file with the listener-presence check. |
| 119 | +Do not restart the server or browser when attaching the missing poll repairs the board. |
| 120 | + |
| 121 | +### Current server has the listener leak |
| 122 | + |
| 123 | +List possible server processes without signaling them: |
| 124 | + |
| 125 | +```sh |
| 126 | +pgrep -fl 'lavish-axi/dist/cli.mjs server' |
| 127 | +``` |
| 128 | + |
| 129 | +This selector can also appear inside a crewmate brief, so its output is discovery evidence only. |
| 130 | +Inspect every candidate with the safety-gate `ps` command. |
| 131 | +Confirm the intended Node process has the Lavish server entry point and expected `server --port <port>` arguments. |
| 132 | +When useful, corroborate the listener PID for the expected port with `lsof -nP -iTCP:<port> -sTCP:LISTEN`. |
| 133 | +If the candidates do not resolve to exactly one intended server, stop without signaling anything. |
| 134 | + |
| 135 | +Terminate only the verified numeric server PID: |
| 136 | + |
| 137 | +```sh |
| 138 | +kill <server-pid> |
| 139 | +``` |
| 140 | + |
| 141 | +Re-serve every affected source file with `lavish-axi '<source-file>' --no-open`. |
| 142 | +Reattach `lavish-axi poll '<source-file>'` for every board and leave each poll running. |
| 143 | +Repeat the HTTP, current-log, and listener checks before reporting recovery. |
| 144 | + |
| 145 | +Session URLs survive a server restart when the same source files are re-served because Lavish derives session identity from each canonical source-file path and retains it in state. |
| 146 | +Do not treat the restart as URL loss or replace working links preemptively. |
| 147 | + |
| 148 | +### Proven browser failure |
| 149 | + |
| 150 | +Protect any unsubmitted captain input before reloading or reopening a page. |
| 151 | +Use current `chrome-devtools-axi` help to target the exact board page, and prefer repairing or reopening only that page with its existing session URL. |
| 152 | +Do not kill `chrome-devtools-axi` processes by name. |
| 153 | +If an exact browser process must be terminated, apply the fleet-wide safety gate and act only on a verified explicit PID with a run-specific binary or `user-data-dir` identity. |
| 154 | +Recheck the same URL and poll connection after the browser-side repair. |
| 155 | + |
| 156 | +## What survives a crew kill |
| 157 | + |
| 158 | +A process-kill incident is serious but does not erase landed or worktree-backed work. |
| 159 | + |
| 160 | +- Crewmate reports and completion artifacts under `data/<id>/` survive. |
| 161 | +- Leased worktrees, their working files, and their commits survive. |
| 162 | +- Pushed branches and PRs survive remotely. |
| 163 | +- Only the live agent process and its in-memory session are lost. |
| 164 | + |
| 165 | +Recover a dead crewmate by respawning it from its recorded worktree and brief. |
| 166 | +Load `stuck-crewmate-recovery` and `harness-adapters` before performing that recovery. |
0 commit comments