feat: monitor certsync health on heartbeat

[Freudator86]

Intent

Wire certsync into the firstmate watcher heartbeat so certsync health is monitored without adding a separate escalation path. The certsync project's real health interface is its documented Docker Compose certsync status command, which returns JSON containing healthy and reason fields; the watcher should run that bounded command from the heartbeat only when the certsync project/compose files are present. A confirmed healthy:false JSON reading should enqueue an existing check wake for certsync-health with the unhealthy reason. Healthy, missing, unreadable, invalid JSON, missing docker/jq, or absent certsync deployment must be treated as unknown/quiet so the watcher does not spam, crash, or slow normal heartbeat behavior. Add colocated watcher coverage for healthy, unhealthy, and unknown branches and preserve existing supervision behavior.

What Changed

• Adds a bounded certsync heartbeat health probe that runs the documented Docker Compose certsync status command when a home-local deployment is present.
• Queues the existing durable check wake for confirmed healthy: false JSON results, including during AFK mode, while keeping healthy, missing, invalid, or unreadable states quiet.
• Documents the new FM_CERTSYNC_* configuration knobs and extends watcher triage coverage for healthy, unhealthy, AFK unhealthy, and unknown certsync status paths.

Risk Assessment

✅ Low: The change is narrowly scoped to heartbeat certsync probing and colocated watcher coverage, with no material correctness, security, or error-handling issues found, captain.

Testing

Captain, I exercised the changed watcher heartbeat path with the focused watcher triage suite plus a reviewer-readable end-to-end certsync heartbeat matrix; unhealthy JSON surfaced through the existing check wake queue, and healthy/unknown readings produced no wake or stdout.

Evidence: Focused watcher triage test log

ok - signal_reason_is_actionable: benign absorbed, captain verbs and coalesced batches surfaced
ok - stale_is_terminal: terminal status surfaces, non-terminal and no-status are benign
ok - scan_captain_relevant_statuses lists only captain-relevant statuses
ok - classifier primitives: keyed decisions and activity phases, captain relevance, window-to-task, and overrides
ok - crew_is_provably_working: only working+run-step/pane is provable; idle/finished/parked/failed/unknown surface
ok - status_is_paused: only the leading paused verb matches, and paused is not captain-relevant
ok - crew_absorb_class: working/paused/none from one read; crew_is_paused and crew_is_provably_working agree
ok - signal_crew_provably_working: benign only when every referenced crew is provably working
ok - a no-verb signal whose crew is provably working is absorbed (no exit, no queue, suppressor advanced, beacon present)
ok - a bare turn-end whose crew is provably working (busy pane) is absorbed
ok - a bare turn-end whose crew is not provably working is surfaced (the swallowed-finish fix)
ok - a no-verb working: note whose crew is idle with no running pipeline is surfaced
ok - captain-relevant signal is surfaced (queue + exit) and marked surfaced
ok - a two-file single crewmate turn enqueues exactly one wake while both markers advance
ok - away mode collapses a two-file crewmate turn to one wake while both markers advance
ok - two crewmates signalling in one poll still produce one wake each (dedup never crosses tasks)
ok - a turn-end marker moving alone still keys its own wake
ok - a stale pane sitting on a terminal status is surfaced (queue + exit)
ok - a relayed terminal task absorbs parked pane churn and re-surfaces on the bounded cadence
ok - a real status write wakes immediately and clears parked tracking
ok - a metadata change clears parked tracking before stale classification
ok - mark-parked wrapper: creates the marker for a recorded window, refuses an unrecognized one, never engages the watcher lock
ok - mark-parked wrapper: refuses a kind=secondmate window, leaving the pause-tracking path untouched
ok - a stale terminal-looking status is absorbed while its run works, holds the ladder, and escalates once the run stops
ok - provably-working non-terminal stale is absorbed, holds the ladder while its run is active, and escalates once the run stops
ok - a healthy static-pane codex worker with an alive process is absorbed (agent-liveness backstop), and the wedge timer still arms
ok - a codex pane whose process is dead surfaces immediately (the backstop never masks a crash)
ok - the agent-liveness absorb is scoped to codex: a claude worker with an alive process still surfaces immediately
ok - a follow-on stale repeating an already-surfaced terminal line is absorbed, and a real wedge behind it still escalates
ok - a stale whose captain-relevant line has not been surfaced yet still wakes firstmate at once
ok - consecutive wedge escalations on the same pane accumulate and demand deep inspection at the threshold
ok - a ladder hold that lasts a full recheck window surfaces one bounded recheck without climbing the ladder
ok - a pane becoming active again resets the consecutive wedge-escalation counter
ok - a not-provably-working non-terminal stale is surfaced immediately (never left to wait out the timer)
ok - a declared pause is absorbed on first sight, then re-surfaced as a recheck past the threshold, never wedge-escalated
ok - exited declared-pause and captain-held panes use bounded pause cadence while a live decision gate still surfaces once
ok - a declared paused secondmate re-surfaces on the bounded normal-mode cadence
ok - a non-paused secondmate retains normal stale suppression
ok - a resumed secondmate clears pause and stale tracking before stale exemption
ok - unchanged stale hashes reclassify when a crew enters or leaves pause
ok - a declared pause is periodically rechecked against authoritative active-run state
ok - a paused status overridden by authoritative working keeps its wedge timer and holds the ladder while the run is active
ok - a ladder hold on the paused-then-working path surfaces one bounded recheck without climbing the ladder
ok - matching non-terminal stale suppressors repair missing or corrupt stale-since timers
ok - triage log capping handles wc byte counts with leading spaces
ok - a heartbeat with no captain-relevant change is absorbed and backs off the cadence
ok - heartbeat backstop fail-safe surfaces a captain-relevant status the per-wake path missed
ok - heartbeat absorbs healthy certsync status
ok - heartbeat surfaces confirmed unhealthy certsync through the check wake path
ok - afk heartbeat surfaces confirmed unhealthy certsync through the check wake path
ok - heartbeat treats unreadable certsync status as unknown without escalation
ok - the liveness beacon stays fresh while the watcher absorbs benign wakes (fm-guard never false-alarms)
ok - with .afk present the watcher reverts to one-shot so the daemon owns triage (no double-triage)
ok - AFK changed paused panes hand off plain stale identities for daemon-owned pause triage

Evidence: Certsync heartbeat end-to-end matrix

## unhealthy
input_status={"healthy":false,"reason":"state DB missing"}
watch_exit=0
watch_stdout=check: certsync health: unhealthy: state DB missing|
drain_output=1785770603	1	check	certsync-health	check: certsync health: unhealthy: state DB missing|
docker_argv=compose -f /tmp/no-mistakes-evidence/01KZ40R867KJ81ASD9R1P5A1YT/manual-certsync-1785770602/unhealthy/home/projects/hlr-certsync/docker-compose.yml -f /tmp/no-mistakes-evidence/01KZ40R867KJ81ASD9R1P5A1YT/manual-certsync-1785770602/unhealthy/home/projects/hlr-certsync/docker-compose.graph-pem.yml exec -T certsync certsync status --state-db /var/lib/hlr-certsync/certsync-state.sqlite3 --heartbeat-file /var/lib/hlr-certsync/certsync-heartbeat.json --daemon-state running|

## healthy
input_status={"healthy":true,"reason":"ok"}
watch_alive_after_2.5s=yes
watch_stdout_bytes=0
wake_queue=empty
triage_log=[2026-08-03T15:23:25+0000] absorbed heartbeat (no captain-relevant change)|
docker_argv=compose -f /tmp/no-mistakes-evidence/01KZ40R867KJ81ASD9R1P5A1YT/manual-certsync-1785770602/healthy/home/projects/hlr-certsync/docker-compose.yml -f /tmp/no-mistakes-evidence/01KZ40R867KJ81ASD9R1P5A1YT/manual-certsync-1785770602/healthy/home/projects/hlr-certsync/docker-compose.graph-pem.yml exec -T certsync certsync status --state-db /var/lib/hlr-certsync/certsync-state.sqlite3 --heartbeat-file /var/lib/hlr-certsync/certsync-heartbeat.json --daemon-state running|

## invalid-json
input_status=not-json
watch_alive_after_2.5s=yes
watch_stdout_bytes=0
wake_queue=empty
triage_log=[2026-08-03T15:23:28+0000] certsync health unknown (invalid status JSON)|[2026-08-03T15:23:28+0000] absorbed heartbeat (no captain-relevant change)|
docker_argv=compose -f /tmp/no-mistakes-evidence/01KZ40R867KJ81ASD9R1P5A1YT/manual-certsync-1785770602/invalid-json/home/projects/hlr-certsync/docker-compose.yml -f /tmp/no-mistakes-evidence/01KZ40R867KJ81ASD9R1P5A1YT/manual-certsync-1785770602/invalid-json/home/projects/hlr-certsync/docker-compose.graph-pem.yml exec -T certsync certsync status --state-db /var/lib/hlr-certsync/certsync-state.sqlite3 --heartbeat-file /var/lib/hlr-certsync/certsync-heartbeat.json --daemon-state running|

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

🔧 **Review** - 1 issue found → auto-fixed (2) ✅

• 🚨 bin/fm-watch.sh:986 - The default certsync deployment path is resolved from FM_ROOT, but project clones belong to the active FM_HOME. In any home where the tracked code root and operational home differ, the watcher will look in the shared code checkout instead of the home’s projects directory, treat certsync as absent, and never monitor an installed certsync clone unless FM_CERTSYNC_PROJECT is manually set. Default this to $FM_HOME/projects/hlr-certsync (or the home’s project override if that is intended) and add a test with FM_HOME != FM_ROOT_OVERRIDE.

🔧 Fix: Captain: Default certsync discovery to FM_HOME
1 warning still open:

• ⚠️ bin/fm-watch.sh:1668 - When AFK mode is active, the heartbeat path queues a generic heartbeat before reaching certsync_health_reason, so certsync is never checked and a confirmed healthy:false JSON does not enqueue the requested check certsync-health wake. The away daemon self-handles generic heartbeat wakes and only runs its status-file catch-all, so this health signal is missed until AFK is cleared.

🔧 Fix: Captain: Check certsync before AFK heartbeat
✅ Re-checked - no issues remain.

✅ **Test** - passed

✅ No issues found.

• ./tests/fm-watch-triage.test.sh > /tmp/no-mistakes-evidence/01KZ40R867KJ81ASD9R1P5A1YT/fm-watch-triage.test.log 2>&1
• Manual evidence run using bin/fm-watch.sh with a fake Docker Compose certsync status command for healthy, unhealthy, and invalid-JSON payloads, followed by bin/fm-wake-drain.sh for the unhealthy case; output saved to /tmp/no-mistakes-evidence/01KZ40R867KJ81ASD9R1P5A1YT/certsync-heartbeat-matrix.txt

✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.