feat: monitor certsync health on heartbeat

AGENTS.md

@@ -132,7 +132,7 @@ state/               volatile runtime signals; gitignored
   .stow-receipt .context-reset.log  proof that this session's durable knowledge was filed, bound to the transcript position it was filed at, and the durable record of every context-reset refusal or completed reset (docs/context-reset.md)
   .parked-<window-key>  firstmate-owned declaration that a relayed terminal task waits only on external human action; created via `bin/fm-mark-parked.sh <window>` (bin/fm-watch.sh's `mark_parked`), never by hand
   .watch.lock .wake-queue.lock watcher singleton and queue serialization locks
-  .hash-* .count-* .stale-* .stale-since-* .paused-* .parkedmeta-* .parkedresurfaced-* .wedge-escalations-* .wedgeheld-* .seen-* .hb-surfaced-* .last-* .heartbeat-streak .bridge-* .context-ceiling-surfaced   watcher internals; never touch
+  .hash-* .count-* .stale-* .stale-since-* .paused-* .parkedmeta-* .parkedresurfaced-* .wedge-escalations-* .wedgeheld-* .seen-* .hb-surfaced-* .last-* .heartbeat-streak .bridge-* .context-ceiling-surfaced .certsync-health-surfaced   watcher internals; never touch
   .watch-triage.log  watcher's absorbed-wake debug log (size-capped); never relied on, safe to delete
   .last-watcher-beat watcher liveness beacon, touched every poll (including while absorbing benign wakes); guard scripts read it
   .subsuper-* .supervise-daemon.*   sub-supervisor internals; never touch
@@ -372,7 +372,7 @@ Handle actionable wakes as follows:
 
 1. For `signal:`, read the listed event lines first, then reconcile current state only where action depends on it.
 2. For `stale:`, inspect the recorded endpoint and load `stuck-crewmate-recovery` for a stopped, looping, confused, or unresponsive worker; a deep-inspection reason also requires current-state and validation-log inspection.
-3. For `check:`, act on the named poll result, including merges, Bridge inbox traffic, X-mode events, and a context-ceiling wake whose payload carries its own next step: either run `/stow` and then, in that same turn, the receipt and reset commands it names, or ask the captain first because a reset must never happen during a live conversation.
+3. For `check:`, act on the named poll result, including merges, Bridge inbox traffic, X-mode events, certsync health, and a context-ceiling wake whose payload carries its own next step: either run `/stow` and then, in that same turn, the receipt and reset commands it names, or ask the captain first because a reset must never happen during a live conversation.
 4. For `heartbeat:`, review the whole fleet from the structured fleet view, reconcile suspicious tasks and PR state, update the backlog, and never report an unchanged fleet as progress.
 
 When any wake reports a merged PR for a project cloned in this home, refresh that clone through the guarded fleet-sync path.

bin/fm-watch.sh

@@ -50,6 +50,10 @@
 #                          an unenforced one. See docs/context-reset.md.
 #   heartbeat              fleet-scan backstop found an unsurfaced captain-relevant
 #                          status, unless afk is active
+#   check: certsync health: unhealthy: <reason>
+#                          heartbeat found a confirmed unhealthy certsync status
+#                          JSON reading and surfaced it through the ordinary
+#                          durable check wake path
 # For normal supervision, resume the session-start primary-harness protocol
 # after each printed reason. Direct duplicate invocations of this script still
 # no-op through the watcher singleton lock.
@@ -140,6 +144,8 @@ HEARTBEAT=${FM_HEARTBEAT:-600}        # base seconds between heartbeat scans
 HEARTBEAT_MAX=${FM_HEARTBEAT_MAX:-7200}  # heartbeat backoff cap
 CHECK_INTERVAL=${FM_CHECK_INTERVAL:-300}  # seconds between *.check.sh sweeps
 CHECK_TIMEOUT=${FM_CHECK_TIMEOUT:-30}     # seconds allowed per *.check.sh
+CERTSYNC_HEALTH_TIMEOUT=${FM_CERTSYNC_HEALTH_TIMEOUT:-5}  # seconds allowed for certsync heartbeat health
+CERTSYNC_HEALTH_RESURFACE=${FM_CERTSYNC_HEALTH_RESURFACE:-3600}  # seconds before repeating unchanged unhealthy certsync
 CONTEXT_CHECK_INTERVAL=${FM_CONTEXT_CHECK_INTERVAL:-300}  # seconds between context-ceiling reads
 # How long an UNCHANGED context-ceiling report stays quiet before it says so
 # again. Long, because every branch of that check describes a standing condition
@@ -963,6 +969,70 @@ heartbeat_scan_finds_actionable() {
   return 1
 }
 
+FM_CERTSYNC_HEALTH_REASON=
+FM_CERTSYNC_HEALTH_SIGNATURE=
+
+certsync_health_mark_surfaced() {
+  [ -n "$FM_CERTSYNC_HEALTH_SIGNATURE" ] || return 0
+  printf '%s\n' "$FM_CERTSYNC_HEALTH_SIGNATURE" > "$STATE/.certsync-health-surfaced" 2>/dev/null || true
+}
+
+# Read certsync's own monitoring interface and produce one bounded wake reason
+# only for a confirmed unhealthy JSON payload.
+certsync_health_reason() {
+  local project compose graph_compose marker previous timeout_previous out healthy summary
+  FM_CERTSYNC_HEALTH_REASON=
+  FM_CERTSYNC_HEALTH_SIGNATURE=
+  project=${FM_CERTSYNC_PROJECT:-$FM_HOME/projects/hlr-certsync}
+  compose=${FM_CERTSYNC_COMPOSE_FILE:-$project/docker-compose.yml}
+  graph_compose=${FM_CERTSYNC_GRAPH_COMPOSE_FILE:-$project/docker-compose.graph-pem.yml}
+  marker="$STATE/.certsync-health-surfaced"
+
+  [ -d "$project" ] || return 1
+  [ -f "$compose" ] || return 1
+  command -v docker >/dev/null 2>&1 || { triage_log "certsync health unknown (docker missing)"; return 1; }
+  command -v jq >/dev/null 2>&1 || { triage_log "certsync health unknown (jq missing)"; return 1; }
+
+  timeout_previous=$CHECK_TIMEOUT
+  CHECK_TIMEOUT=$CERTSYNC_HEALTH_TIMEOUT
+  if [ -f "$graph_compose" ]; then
+    out=$(run_bounded docker compose -f "$compose" -f "$graph_compose" exec -T certsync certsync status \
+      --state-db "${FM_CERTSYNC_STATE_DB:-/var/lib/hlr-certsync/certsync-state.sqlite3}" \
+      --heartbeat-file "${FM_CERTSYNC_HEARTBEAT_FILE:-/var/lib/hlr-certsync/certsync-heartbeat.json}" \
+      --daemon-state "${FM_CERTSYNC_DAEMON_STATE:-running}")
+  else
+    out=$(run_bounded docker compose -f "$compose" exec -T certsync certsync status \
+      --state-db "${FM_CERTSYNC_STATE_DB:-/var/lib/hlr-certsync/certsync-state.sqlite3}" \
+      --heartbeat-file "${FM_CERTSYNC_HEARTBEAT_FILE:-/var/lib/hlr-certsync/certsync-heartbeat.json}" \
+      --daemon-state "${FM_CERTSYNC_DAEMON_STATE:-running}")
+  fi
+  CHECK_TIMEOUT=$timeout_previous
+
+  [ -n "$out" ] || { triage_log "certsync health unknown (no status output)"; return 1; }
+  healthy=$(printf '%s' "$out" | jq -r 'if (.healthy == true or .healthy == false) then .healthy else empty end' 2>/dev/null) \
+    || { triage_log "certsync health unknown (invalid status JSON)"; return 1; }
+  [ -n "$healthy" ] || { triage_log "certsync health unknown (missing healthy boolean)"; return 1; }
+  if [ "$healthy" = true ]; then
+    rm -f "$marker" 2>/dev/null || true
+    return 1
+  fi
+
+  summary=$(printf '%s' "$out" | jq -r '.reason // "unhealthy"' 2>/dev/null \
+    | tr '\n\t' '  ' \
+    | sed 's/[[:space:]]*$//' \
+    | cut -c1-300)
+  [ -n "$summary" ] || summary=unhealthy
+  FM_CERTSYNC_HEALTH_REASON="check: certsync health: unhealthy: $summary"
+  FM_CERTSYNC_HEALTH_SIGNATURE=$FM_CERTSYNC_HEALTH_REASON
+  previous=$(cat "$marker" 2>/dev/null || true)
+  if [ "$previous" = "$FM_CERTSYNC_HEALTH_SIGNATURE" ] \
+    && [ "$(age_of "$marker")" -lt "$CERTSYNC_HEALTH_RESURFACE" ]; then
+    triage_log "absorbed certsync health (unchanged unhealthy status)"
+    return 1
+  fi
+  return 0
+}
+
 # event_wait_or_sleep: the terminal wait of each supervision cycle. For a home
 # with push-capable windows (herdr), it replaces the blind `sleep POLL` with a
 # bounded wait on the backend's native transition stream, so a crew going
@@ -1588,9 +1658,14 @@ EOF
     # Triage: in always-on mode a heartbeat is benign unless the cheap fleet-scan
     # turns up a captain-relevant status the per-wake path missed. Absorb the
     # no-change case (advance the schedule and back off exactly as wake() would,
-    # without exiting); the away-mode daemon, when present, owns triage and wants
-    # every heartbeat.
-    if afk_present; then
+    # without exiting).
+    if certsync_health_reason; then
+      fm_wake_append check certsync-health "$FM_CERTSYNC_HEALTH_REASON" || exit 1
+      touch "$STATE/.last-heartbeat"
+      certsync_health_mark_surfaced
+      wake "$FM_CERTSYNC_HEALTH_REASON"
+      [ "$WAKE_PENDING" -eq 0 ] || continue
+    elif afk_present; then
       fm_wake_append heartbeat heartbeat heartbeat || exit 1
       touch "$STATE/.last-heartbeat"
       wake "heartbeat"

docs/architecture.md

@@ -9,7 +9,7 @@ firstmate's always-loaded operating contract and routing index for conditional p
 ## Event-driven supervision
 
 A zero-token bash watcher (`bin/fm-watch.sh`) sleeps on the fleet, classifies detected wakes in bash, and wakes the first mate only when something is actionable.
-Actionable wakes include captain-relevant status signals, no-verb signals whose crew is not provably working, authenticated check output such as PR merge polling or an X-mode mention, stale panes whose crew is not provably working and whose captain-relevant status the signal path has not already surfaced, whether their status log looks terminal or non-terminal, absorbed stale panes that persist past `FM_STALE_ESCALATE_SECS` and no longer read as provably working at that moment, declared external waits (a crew's `paused:` status or a firstmate-declared `.parked-<window-key>` marker) that remain past `FM_PAUSE_RESURFACE_SECS`, and heartbeat backstop hits.
+Actionable wakes include captain-relevant status signals, no-verb signals whose crew is not provably working, authenticated check output such as PR merge polling or an X-mode mention, confirmed unhealthy certsync health from the heartbeat path, stale panes whose crew is not provably working and whose captain-relevant status the signal path has not already surfaced, whether their status log looks terminal or non-terminal, absorbed stale panes that persist past `FM_STALE_ESCALATE_SECS` and no longer read as provably working at that moment, declared external waits (a crew's `paused:` status or a firstmate-declared `.parked-<window-key>` marker) that remain past `FM_PAUSE_RESURFACE_SECS`, and heartbeat backstop hits.
 Repeated stale escalations on the same unchanged pane add an escalation count to the wake reason and, at `FM_WEDGE_DEMAND_INSPECT_COUNT`, a `demand-deep-inspection` marker.
 Those actionable wakes are written to a durable local queue (`state/.wake-queue`) before detector state advances, so losing a model-facing delivery wait cannot lose the wake.
 No-verb wakes, such as `working:` notes and bare turn-ended signals, are benign only when `bin/fm-crew-state.sh` reports positive evidence that the crew is still working: an actively running no-mistakes step for that crew's branch or a backend busy signature.
@@ -53,6 +53,8 @@ Optional X mode integrates with the watcher only after explicit opt-in; [configu
 The optional per-home Bridge inbox check has a separate plain-shell frequency monitor for the first configured vessel and retains the original watcher path as a slower fallback.
 Both paths use one shared, lock-protected implementation to bounded-fetch and read unacknowledged envelopes from the Bridge clone's `origin/main`, durably enqueue one wake per new signature, and never acknowledge mail.
 [configuration.md](configuration.md#bridge-frequency-monitor-service) owns the service and consent mechanics, while [configuration.md](configuration.md#bridge-inbox-check-fm_bridge_) owns inbox detection and cadence.
+The optional certsync heartbeat check reuses the same durable `check` wake path instead of adding a separate escalation channel.
+[configuration.md](configuration.md#certsync-health-check-fm_certsync_) owns the deployment discovery, bounded Docker Compose status command, quiet unknown cases, and re-surface cadence.
 
 At session start, `bin/fm-session-start.sh` emits exactly one primary-harness delivery block rendered by `bin/fm-supervision-instructions.sh` from `docs/supervision-protocols/`.
 The watcher loop is external to the model harness and runs continuously in an enabled `systemd --user` template instance, or in a detached home-scoped tmux keeper when the user manager is unavailable.

docs/configuration.md

@@ -569,6 +569,17 @@ The watcher caches each vessel's fetched tree signature and derived priority sep
 The frequency monitor deliberately narrows that compatibility list to its first vessel and fetches it every `FM_FREQUENCY_MONITOR_INTERVAL` seconds.
 Both paths share the same signature and marker implementation, so the slow fallback and fast service cannot drift in their definition of new mail or duplicate one signature during a concurrent check.
 
+## Certsync health check (FM_CERTSYNC_*)
+
+`bin/fm-watch.sh` folds certsync health into the ordinary heartbeat path when a certsync deployment is present under the home.
+The default deployment path is `$FM_HOME/projects/hlr-certsync` with `docker-compose.yml`; `FM_CERTSYNC_PROJECT`, `FM_CERTSYNC_COMPOSE_FILE`, and `FM_CERTSYNC_GRAPH_COMPOSE_FILE` override the project and compose files.
+If the optional graph compose file exists, the watcher adds it to the `docker compose` command; if it is absent, the base compose file is enough.
+The heartbeat runs `docker compose ... exec -T certsync certsync status` with `FM_CERTSYNC_STATE_DB`, `FM_CERTSYNC_HEARTBEAT_FILE`, and `FM_CERTSYNC_DAEMON_STATE` mapped to the status command's `--state-db`, `--heartbeat-file`, and `--daemon-state` arguments.
+Only a confirmed JSON object with `healthy: false` becomes a durable `check` wake keyed as `certsync-health`, with the `reason` field trimmed and bounded in the wake text.
+A `healthy: true` reading clears the unchanged-unhealthy marker and stays quiet.
+Missing project or compose files, missing `docker` or `jq`, empty output, invalid JSON, a missing boolean `healthy` field, and all other unreadable states are unknown and quiet except for the local `state/.watch-triage.log` debug note.
+The command is bounded by `FM_CERTSYNC_HEALTH_TIMEOUT` rather than the general check timeout, and unchanged unhealthy readings re-surface only after `FM_CERTSYNC_HEALTH_RESURFACE`.
+
 ## Environment variables
 
 Runtime tuning via environment variables (defaults shown):
@@ -607,6 +618,14 @@ FM_HEARTBEAT=600        # base seconds between heartbeat scans; no-change heartb
 FM_HEARTBEAT_MAX=7200   # heartbeat backoff cap
 FM_CHECK_INTERVAL=300   # seconds between slow checks (authenticated merge polls, custom checks, or X-mode dispatch)
 FM_CHECK_TIMEOUT=30     # seconds allowed per slow check script
+FM_CERTSYNC_PROJECT=$FM_HOME/projects/hlr-certsync   # certsync deployment directory watched from the heartbeat path when its compose file exists
+FM_CERTSYNC_COMPOSE_FILE=$FM_CERTSYNC_PROJECT/docker-compose.yml   # base Docker Compose file for the certsync status command
+FM_CERTSYNC_GRAPH_COMPOSE_FILE=$FM_CERTSYNC_PROJECT/docker-compose.graph-pem.yml   # optional extra certsync graph compose file; used only when present
+FM_CERTSYNC_STATE_DB=/var/lib/hlr-certsync/certsync-state.sqlite3   # certsync status --state-db argument
+FM_CERTSYNC_HEARTBEAT_FILE=/var/lib/hlr-certsync/certsync-heartbeat.json   # certsync status --heartbeat-file argument
+FM_CERTSYNC_DAEMON_STATE=running   # certsync status --daemon-state argument
+FM_CERTSYNC_HEALTH_TIMEOUT=5   # seconds allowed for the heartbeat's certsync status command
+FM_CERTSYNC_HEALTH_RESURFACE=3600   # seconds before an unchanged unhealthy certsync status is queued again
 FM_CONTEXT_CEILING=300000   # captain-decided token ceiling for the primary session's own context; above it, at a quiet boundary, the watcher queues the stow-then-clear or ask wake (docs/context-reset.md)
 FM_CONTEXT_CAPTAIN_IDLE_SECS=1800   # silence since the last genuine captain prompt below which the captain counts as in live conversation: the watcher asks instead of ordering a reset, and bin/fm-context-reset.sh refuses
 FM_CONTEXT_RECEIPT_MAX_AGE=900   # seconds a state/.stow-receipt stays fresh; the receipt and the reset are meant to happen in one turn