Watcher and daemon stranded unhealthy forever after updating across the fm_pid_identity format change (a26b37c / #752)

[jaime-oviedo]

Summary

a26b37c ("fix(bin): make watcher process identity immune to Linux wall-clock changes", #752) changed the output format of fm_pid_identity from <lstart> <command> to linux-starttime=<ticks> cmdline-hex=<hex>.

It did not change fm_watcher_lock_matches_pid, which compares the recorded identity against a fresh read with strict string equality, and it added no handling for an unrecognised or legacy record.

The identity is written once, at lock-claim time, and never refreshed:

• bin/fm-watch.sh:667 writes pid-identity before entering the poll loop
• bin/fm-supervise-daemon.sh:1309 writes pid-identity before its main loop

So a watcher or daemon that is already running when its home updates keeps a record in the old format for the rest of that process's life. Every subsequent health check re-derives the new format, compares it against the old record, and finds a mismatch.

Observed behaviour

The live, beating lock holder is classified unhealthy — permanently, not transiently.

Downstream of that:

• bin/fm-continuity-pretool-check.sh refuses fm-send, fm-peek, fm-spawn, fm-brief, fm-crew-state and fm-decision-hold with no live watcher holds this home lock
• bin/fm-turnend-guard.sh blocks turn end with TURN WOULD END BLIND - SUPERVISION IS OFF while reporting a beacon age of a few seconds in the same message

That self-contradiction — fresh beacon, rejected fingerprint — is the diagnostic tell.

Expected behaviour

A holder that is genuinely alive and beating should not be permanently unhealthy merely because the identity format changed underneath it.

Reproduction

Against a tree containing a26b37c:

1. Start a watcher and let it claim state/.watch.lock.
2. Overwrite state/.watch.lock/pid-identity with the pre-a26b37c value for the same pid — i.e. what an older watcher would have written:

   LC_ALL=C ps -p <lock pid> -o lstart= -o command= | sed 's/^[[:space:]]*//'
   

3. Call fm_watcher_healthy with the home and watcher path recorded in the lock.

It returns unhealthy, and keeps returning unhealthy indefinitely, because nothing re-stamps the record.

Standalone confirmation of the comparison itself, with a live pid and a fresh beacon:

legacy record  : Tue Jul 28 09:30:30 2026 sleep 25
new-format now : linux-starttime=8508874 cmdline-hex=736c65657000323500
RESULT: UNHEALTHY -- live beating watcher rejected because its record is legacy-format

Affected

• Introduced by: a26b37c (fix(bin): make watcher process identity immune to Linux wall-clock changes #752, 2026-07-20)
• Versions: every revision with a26b37c as an ancestor; confirmed present at origin/main = fa0d85d
• Platforms: Linux, where a26b37c changed the emitted format. Hosts taking the ps fallback keep the old format and are not affected by the format change itself.

Note on severity

This fires precisely when a home syncs — which is the operation that delivers the fix. The more successfully a26b37c propagates, the more homes hit this on the way in.

Possible fix

We addressed it locally with a holder-only, ownership-gated fm_lock_refresh_identity called from both long-lived loops, leaving the comparison itself untouched so pid-reuse detection is preserved. Re-stamping is only safe when performed by the verified holder; a non-holder re-stamp would defeat the check entirely.

We are not opening a PR — this instance tracks upstream read-only — but we're happy to share the patch if it's useful.

Possibly related

#1179 mentions "watcher cycles also exit 1 without reason since update" at fa0d85d, which may be a different symptom of this same update boundary. Not confirmed.