Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .agents/skills/bootstrap-diagnostics/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ When any diagnostic needs captain attention, report the plain consequence and re
- `FLEET_SYNC: <repo>: skipped: <reason>` - a benign one-off skip (offline, no origin, local-only); bootstrap continued, investigate only if it blocks work.
A skip can also report the bounded fleet-refresh timeout (`FM_FLEET_SYNC_BOOTSTRAP_TIMEOUT`, or a fleet-size-aware default with a 20 second floor); a timeout never blocks startup.
- `FLEET_SYNC: <repo>: recovered: <detail>` - the clone had drifted onto a clean detached HEAD holding no unique commits and the sync self-healed it (re-attached the default branch and fast-forwarded); no action needed, it is reported only so the self-heal is visible.
- `FLEET_SYNC: <repo>: STUCK: on <state>, N commits behind <base> - needs attention` - the clone is dirty, on a non-default branch, detached with unique commits, or diverged, so the sync left it untouched (never forcing or discarding); it will keep falling behind until you look.
- `FLEET_SYNC: <repo>: STUCK: on <state>, N commits behind <base> - needs attention` - the clone is dirty beyond the lone untracked root `CAPTAINS-LOG.md` the sync tolerates on the default branch, on a non-default branch, detached with unique commits, or diverged, so the sync left it untouched (never forcing or discarding); it will keep falling behind until you look.
A loud STUCK, especially a growing N across bootstraps, means that clone needs hands-on attention; dispatch a crewmate or resolve it before it strands work.
- `PR_CHECK_MIGRATION: canonical polls rebuilt and armed; resume supervision for this home` - the non-executing migration rebuilt canonical task polls from validated metadata, and those polls are already armed.
Independently verify the private per-task outcome record, then resume the emitted supervision protocol after finishing the session-start wake handling.
Expand Down
49 changes: 41 additions & 8 deletions bin/fm-fleet-sync.sh
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,13 @@
# Every other off-default state - a non-default named branch, a detached HEAD with
# unique commits, a dirty tree, or a diverged default - may hold real work, so it
# is left untouched and reported as a quantified, loud "STUCK: ... N commits behind
# ... - needs attention" warning rather than a quiet drift. Nothing is ever forced,
# stashed, or discarded.
# ... - needs attention" warning rather than a quiet drift. The sole cleanliness
# exception is exactly one untracked root `CAPTAINS-LOG.md`, and it only relaxes
# ordinary fast-forward eligibility for a clone already on its default branch -
# detached-HEAD re-attachment still demands a fully clean tree. Git keeps the
# untracked file visible and retains its overwrite protection; when that protection
# blocks the fast-forward (origin publishes a tracked CAPTAINS-LOG.md) the clone is
# reported STUCK, not skipped. Nothing is ever forced, stashed, or discarded.
# Still skips (benignly) local-only/no-origin projects, missing remotes/branches,
# and fetch failures.
# Pruning never deletes the checked-out branch or a branch that still has a
Expand All @@ -36,6 +41,8 @@ PROJECTS="${FM_PROJECTS_OVERRIDE:-$FM_HOME/projects}"
# shellcheck source=bin/fm-lock-lib.sh
. "$SCRIPT_DIR/fm-lock-lib.sh"
FM_LOCK_LOG_PREFIX=fleet-sync
# The one repo-root path a Captain may keep untracked without blocking fleet sync.
CAPTAINS_LOG=CAPTAINS-LOG.md
"$FM_ROOT/bin/fm-guard.sh" || true

# Bounded recovery for an orphaned .git/packed-refs.lock. A git ref rewrite
Expand Down Expand Up @@ -127,13 +134,29 @@ first_line() {
printf '%s\n' "$1" | sed -n '1s/[[:space:]]\{1,\}/ /g;1p'
}

# These two matchers read git's human-facing error text, which gettext localizes
# under the operator's LANG/LC_ALL. Every git call whose output they inspect is
# therefore pinned to LC_ALL=C at the call site, the same way bin/fm-wake-lib.sh
# pins ps for lstart; without that pin a non-English fleet silently loses the
# recovery and the STUCK escalation these drive.

# True when git stderr shows the packed-refs.lock "File exists" race. The lock
# path can appear anywhere in the message (git prefixes it with the failed ref op,
# e.g. "could not delete reference ...:"). Other "File exists" errors must not match.
is_packed_refs_lock_error() {
printf '%s\n' "$1" | grep -Eq "Unable to create ['\"].*packed-refs\\.lock['\"]: File exists"
}

# True when git refused the merge because it would overwrite the untracked
# CAPTAINS-LOG.md that the cleanliness exception admitted - i.e. origin now tracks
# the file. Unlike a transient skip this state persists until a human resolves it,
# so callers escalate it to a loud, quantified STUCK. Git lists each blocked path
# on its own indented line, matched literally so a similarly named path cannot.
is_captains_log_overwrite_refusal() {
printf '%s\n' "$1" | grep -q 'untracked working tree files would be overwritten' \
&& printf '%s\n' "$1" | sed 's/^[[:space:]]*//' | grep -Fxq -- "$CAPTAINS_LOG"
}

# Absolute path to $PROJ's packed-refs.lock, or empty when it cannot be resolved.
packed_refs_lock_path() {
local lock abs
Expand Down Expand Up @@ -161,7 +184,7 @@ packed_refs_lock_path() {
# a session-start refresh (which discards fleet-sync stderr) still surfaces it.
fetch_with_packed_refs_lock_guard() {
local rc attempt=0 lock lock_desc
FETCH_OUTPUT=$(git -C "$PROJ" fetch origin --prune --quiet 2>&1); rc=$?
FETCH_OUTPUT=$(LC_ALL=C git -C "$PROJ" fetch origin --prune --quiet 2>&1); rc=$?
[ "$rc" -eq 0 ] && return 0
is_packed_refs_lock_error "$FETCH_OUTPUT" || return "$rc"

Expand All @@ -171,7 +194,7 @@ fetch_with_packed_refs_lock_guard() {
attempt=$(( attempt + 1 ))
echo "$label: fetch blocked by packed-refs lock ($lock_desc); waiting ${FLEET_SYNC_PACKED_REFS_LOCK_RETRY_WAIT_SECS}s and retrying ($attempt/${FLEET_SYNC_PACKED_REFS_LOCK_RETRIES}) (owning process may be exiting)" >&2
sleep "$FLEET_SYNC_PACKED_REFS_LOCK_RETRY_WAIT_SECS"
FETCH_OUTPUT=$(git -C "$PROJ" fetch origin --prune --quiet 2>&1); rc=$?
FETCH_OUTPUT=$(LC_ALL=C git -C "$PROJ" fetch origin --prune --quiet 2>&1); rc=$?
if [ "$rc" -eq 0 ]; then
echo "$label: fetch succeeded on retry; packed-refs lock cleared on its own" >&2
# One stdout summary so a session-start refresh (which discards fleet-sync
Expand All @@ -195,7 +218,7 @@ fetch_with_packed_refs_lock_guard() {
return "$rc"
fi
echo "$label: removed provably-stale packed-refs lock $lock (age >= ${FLEET_SYNC_PACKED_REFS_LOCK_AGE_SECS}s, no live holder) and retrying fetch" >&2
FETCH_OUTPUT=$(git -C "$PROJ" fetch origin --prune --quiet 2>&1); rc=$?
FETCH_OUTPUT=$(LC_ALL=C git -C "$PROJ" fetch origin --prune --quiet 2>&1); rc=$?
if [ "$rc" -eq 0 ]; then
echo "$label: fetch succeeded after stale packed-refs lock cleanup" >&2
echo "$label: recovered: removed a stale packed-refs lock (no live holder)"
Expand Down Expand Up @@ -334,8 +357,14 @@ sync_project() {
fi

cur=$(git -C "$PROJ" symbolic-ref --short HEAD 2>/dev/null || echo "")
status_porcelain=$(git -C "$PROJ" status --porcelain 2>/dev/null || true)
dirty=no
[ -z "$(git -C "$PROJ" status --porcelain 2>/dev/null | head -1)" ] || dirty=yes
[ -z "$status_porcelain" ] || dirty=yes
# The Captain's log exception, kept separate from $dirty so it only ever relaxes
# the on-default fast-forward gate below - never detached-HEAD recovery, which
# mutates the working tree and so still requires a fully clean one.
lone_captains_log=no
[ "$status_porcelain" != "?? $CAPTAINS_LOG" ] || lone_captains_log=yes
recovered=no

if [ "$cur" != "$DEFAULT" ]; then
Expand All @@ -361,7 +390,7 @@ sync_project() {
report_stuck "$(stuck_state)"
return 0
fi
elif [ "$dirty" = yes ]; then
elif [ "$dirty" = yes ] && [ "$lone_captains_log" = no ]; then
# On the default branch but with uncommitted changes we must not disturb.
report_stuck "$(stuck_state)"
return 0
Expand Down Expand Up @@ -397,7 +426,11 @@ sync_project() {
echo "$label: skipped: cannot read local $DEFAULT"
return 0
}
if ! merge_output=$(git -C "$PROJ" merge --ff-only "$BASE" 2>&1); then
if ! merge_output=$(LC_ALL=C git -C "$PROJ" merge --ff-only "$BASE" 2>&1); then
if [ "$lone_captains_log" = yes ] && is_captains_log_overwrite_refusal "$merge_output"; then
report_stuck "branch $DEFAULT with untracked $CAPTAINS_LOG blocking fast-forward"
return 0
fi
reason="fast-forward failed"
if [ -n "$merge_output" ]; then
reason="$reason: $(first_line "$merge_output")"
Expand Down
1 change: 1 addition & 0 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -247,6 +247,7 @@ The locked session-start bootstrap step, PR-based teardown, and merged-PR wake h
Wake-time refreshes can target a single clone by project name, so the primary home also catches up when a secondmate reports a merge from its own home.
Clean default-branch clones fast-forward to `origin/<default>`, and a clean detached HEAD that holds no unique commits is re-attached to the default branch before the same fast-forward path runs.
Dirty clones, non-default branches, detached HEADs with unique commits, diverged defaults, and default branches checked out in another worktree are reported as `STUCK:` with their behind count and left untouched.
The one cleanliness exception is a clone already on its default branch whose entire working-tree status is a single untracked root `CAPTAINS-LOG.md`: it still fast-forwards, the log stays untracked and keeps Git's overwrite protection, and a fast-forward that protection blocks is reported `STUCK:` rather than skipped; the header of `bin/fm-fleet-sync.sh` owns the exact eligibility rule.
Fetches blocked by an orphaned `.git/packed-refs.lock` use bounded retries and remove the lock only when the shared staleness proof can prove it abandoned; [configuration.md](configuration.md#toolchain) owns the recovery details and tuning knobs.
Local-only projects, clones without an origin remote, and fetch failures remain benign skips.
The refresh also prunes local branches whose remote is gone and that no worktree still needs.
Expand Down
Loading
Loading