feat(team): leader-only warmup with lazy teammate wakeup and per-member attach#670
Merged
Conversation
added 10 commits
July 21, 2026 15:56
…urce-gated leader notify Add TeamSession::ensure_member_runtime_lazy, hooked into both delivery entry points (human enqueue and agent->teammate send) before publishing the runtime constraint so the registry snapshot flips to Attaching synchronously, closing the race window in spec 5.2. Thread notify_leader_on_failure through spawn_attach_agent_process_bg and attach_member_runtime: only agent-triggered failures wake the leader. On attach failure, preserve unread mailbox rows (skip mark_read_batch) so a later retry re-drains them via reconcile_mailbox. The delivery-based lazy-wakeup tests require dormant teammates and are added in the leader-only-first-start task where that state exists.
Cold start now attaches only the lead via the single member attach path and publishes the session before attaching (dropping the startup guards first to avoid re-entrant deadlock with refresh/cleanup). Teammates stay dormant (Absent) and are broadcast as dormant; a leader attach failure bubbles to a session-level Failed. The batch rebuild machine (bounded concurrency + staggered starts + full-kill rollback) is removed, collapsing all attach orchestration onto one path. - refresh_member_runtime_status treats Absent as dormant (non-blocking for Ready); only in-flight Attaching/Removing block. - reconciliation skips dormant (Absent non-lead) members so a second ensure_session cannot punch through lazy warmup. - first-start recovery drain is restricted to the lead slot. Existing tests updated to the leader-only model (repair/idempotent/rollback/ concurrent/fence/serialization now assert leader-only warmup and dormant teammates).
POST /api/teams/{id}/agents/{slot_id}/attach and TeamSessionService::
attach_agent_runtime reuse the single attach path (reserve_attach with
retry_failed=true + background attach) to wake a dormant teammate or retry a
failed one. auth + CSRF are enforced by the shared team router middleware
layer, same as add/remove/send.
Tests: directed attach wakes a dormant teammate to ready; cross-user is
Forbidden; unknown slot is AgentNotFound. Also adds the deferred lazy-attach
coverage: a failed human-direct lazy attach preserves the unread delivery and
does not notify the leader (spec 5.4).
The leader-notify fork (spec 5.4c) was only covered on its negative side: two tests assert human-direct and manual-add failures do NOT wake the leader. Add the missing positive case required by the fix plan (Task A2) and spec section 6: a leader-initiated spawn whose attach fails flows through the single attach path with notify_leader_on_failure=true and must write a 'failed to start its runtime' message to the leader mailbox so it can re-delegate. Deterministic (no active-run dependency).
The full-screen "waking up the team" overlay is driven by session-level
TeamSessionStatus (useTeamWarmup maps `starting` -> warming). After the
leader-only warmup change, every member attach still broadcast a
session-level `Starting` (attach_member_runtime) and refresh_member_runtime_status
aggregated any teammate Attaching/Failed into session Starting/Failed. So a
dormant teammate's lazy wakeup, an add-member, or a teammate attach failure
resurfaced the overlay / full-screen failure card, even though the leader was
already ready and the team was usable.
Per spec 5.2/5.4/5.5 the session-level status is leader-scoped ("Ready =
leader ready") and only the leader may raise the overlay; teammate lifecycle
(dormant/pending/ready/failed) is surfaced inline per-member via
agentRuntimeStatusChanged.
- attach_member_runtime: only publish session `Starting` when the attaching
agent is the lead; a teammate attach relies on its inline pending status.
- refresh_member_runtime_status: compute session status from the leader's
snapshot only; teammate states no longer flip the session status.
- complete_member_runtime_reconciliation: only raise `Starting` when the
reconcile work set includes the lead.
Tests: add waking_dormant_teammate_does_not_resurface_session_starting and
failed_teammate_wakeup_does_not_flip_session_to_failed; update the two
dynamic-add-failure tests to assert the new inline (leader-scoped) semantics.
A whole-team `ensure_session` is invoked far beyond first start: on page mount, on every model switch, and before sends (via `warmupSession`). Its reconciliation retries non-dormant members, and a failed teammate retried this way still broadcast session-level `Failed` and returned `MemberRuntimeFailed`, which the frontend surfaced as the full-screen failure card. So a single broken teammate turned any unrelated interaction (switch another member's model, send in another column) into a whole-team failure overlay. The full-screen overlay is leader-scoped (spec 5.4/5.5): only a leader failure blocks the team. Scope `complete_member_runtime_reconciliation` accordingly — a teammate attach failure during reconciliation stays inline (its `agentRuntimeStatusChanged=failed` already fired) and `ensure_session` returns Ok (leader ready = team usable). A leader reconciliation failure still fails the team. Dormant teammates are already skipped by reconciliation, so they are never woken by these re-ensures. Tests: add reensure_with_failed_teammate_keeps_team_usable_and_inline; update the former conflict test to assert the new inline contract (re-ensure returns Ok, removal still restores Ready).
Slot work state (which drives the send-box "processing" spinner + timer) reached the frontend only bundled inside team.run* events, and TeamRunManager.apply_work_summary drops any summary whose team_run_id is not the currently-tracked run. So a slot doing run-less work — e.g. the leader self-waking to drain its mailbox for a membership-change notification, which creates background intents with no team_run_id — emitted neither run events nor child-turn events on its running->idle transition. The frontend's slotWorkBySlot[slot] stayed at the last "running" snapshot until a full reconcile (which is why switching views cleared the stuck spinner). Add a per-slot event, team.slotWorkChanged (TeamSlotWorkChangedPayload), and publish the affected slot's current snapshot at every batch-lifecycle transition (enqueue commit/abort, claim, mark_started, retry_start, runtime constraint change, terminalize) regardless of run association, via a new RunCausalityPort::publish_slot_work implemented by TeamRunManager. Run-less transitions are now observable, so the frontend can clear per-slot state without a reconcile. Tests: coordinator asserts a run-less batch lifecycle publishes per-slot snapshots ending Idle; emitter asserts the event name/payload shape.
These two broadcasts drive the team UI's two state surfaces — the full-screen warmup overlay (session status) and the per-member column badge / send-box gate (agent runtime status) — and were previously invisible in logs, which made the overlay/stuck-state issues hard to diagnose from production logs. Both are low-volume, important lifecycle state changes, so log at info (production runs at info; debug would be invisible there). Fields are non-sensitive: team_id, slot_id, status, phase, and the already-sanitized public failure reason. The high-frequency mechanism events (team.run*, childTurn, slotWorkChanged) stay at debug.
Merged
18 tasks
piorpua
pushed a commit
that referenced
this pull request
Jul 23, 2026
🤖 I have created a release *beep* *boop* --- ## [0.1.51](v0.1.50...v0.1.51) (2026-07-23) ### Features * **project:** add project-bind foundation (db + aionui-project) ([#672](#672)) ([70eae04](70eae04)) * **session-port:** route claude/codex through the direct-CLI SessionAgentTask ([#609](#609)) ([14efff8](14efff8)) * **team:** leader-only warmup with lazy teammate wakeup and per-member attach ([#670](#670)) ([f542da8](f542da8)) ### Bug Fixes * **acp:** harden grok startup environment and npx recovery ([#662](#662)) ([10cdd57](10cdd57)) * **ci:** restore main migration immutability guard ([7603d44](7603d44)) * **cron:** use host timezone for conversation cron ([#652](#652)) ([19b36f8](19b36f8)) * **skills:** repair butler cron and doc drift (2026-07-22 audit) ([#664](#664)) ([273ac47](273ac47)) * **system:** release keep-awake on shutdown ([#666](#666)) ([6e0b3a7](6e0b3a7)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Lazify team warmup. Previously, entering a team session warmed up every member via a batch
rebuild_agent_processes(bounded concurrency + staggered start), showing a full-screen "waking up team" overlay. This was slow, race-prone, and all-or-nothing: a single member's attach failure failed the whole first start andkill_and_wait-ed the members that had already started.This PR adopts spec approach A — reuse the existing single-member attach primitive and collapse the team runtime onto one attach path, with teammates started lazily.
Paired with the AionUi frontend PR on the same branch, which consumes the new
dormantruntime status, thePOST /api/teams/{id}/agents/{slot_id}/attachroute, and theteam.slotWorkChangedevent.Key changes
reserve_attach+attach_member_runtime. On leader success it broadcasts leaderReady; other teammates broadcast the newDormantstatus and stayAbsentwith no event loop; then sessionReady(server_count = total member count)is broadcast.rebuild_agent_processes,spawn_rebuild_agent_process,cleanup_bootstrap_runtime_tasksand the concurrency/stagger logic.seed_readyis kept but gated#[cfg(test)]. The whole codebase now has a single attach orchestration path shared by leader cold-start, reconciliation,add_agent,spawn_agent, lazy wakeup, and directed retry.ensure_member_runtime_lazy): hooked into both delivery entry points (human→teammateenqueue_user_message, agent→teammatesend_agent_message_from_agent) beforepublish_runtime_constraint. If the event loop is running it returns; otherwise it resolves service/published-session first (to avoid a danglingAttachinglease), does a synchronousreserve_attach(flips the snapshot toAttachingto close the race window), broadcastsPending, and spawns the attach in the background. The lease deduplicates concurrent deliveries into a single attach.attach_member_runtime/spawn_attach_agent_process_bgtake anotify_leader_on_failure: bool. A failed attach converges runtime state toFailedbut no longer callsmark_read_batchonterminal_message_ids, so the triggering message staysread=0and a retry re-drains it viareconcile_mailbox. The leader is notified only whennotify_leader_on_failure = true(agent-triggered delivery and leader-initiatedspawn_agent), not for human-direct deliveries,add_agent, reconciliation, leader cold-start, or the directed retry route.reserve_member_runtime_reconciliationskips non-leadAbsentmembers so a secondensure_session(re-warmup / model switch / retry) does not wake all dormant teammates; leader, Ready-repair, Failed-retry, and in-flightAttaching/Removingstill reconcile.refresh_member_runtime_statusis leader-scoped: it reads only the leader snapshot (leaderReady⇒ teamReady(server_count = total)); teammate dormant/pending/ready/failed is expressed per-member viaagentRuntimeStatusChangedand no longer flips session status. This keeps the overlay from being raised by teammate state and a single teammate failure from raising the full-screen failure card.Startingoverlay is strictly leader-scoped; teammate reconciliation failures stay inline instead of failing the whole team; first-start recovery drain scans only the leader (teammate unread is deferred to lazy start +reconcile_mailbox).POST /api/teams/{id}/agents/{slot_id}/attach:attach_agenthandler inroutes.rs,attach_agent_runtimeinservice.rs(load_owned_team→ idempotentensure_session_inner→reserve_attach(slot, true)→ backgroundspawn_attach_agent_process_bg(..., false)). State-changing; auth + CSRF via the same team router middleware as add/remove/send.team.slotWorkChangedevent +TeamSlotWorkChangedPayload: broadcasts per-slot work changes for run-less slots (e.g. the leader self-waking to drain its mailbox), which the run-scopedteam.run*events do not cover. Naming followsdomain.camelCaseAction.agentRuntimeStatusChangedand session-status broadcasts raised toinfo; lazy-wakeup boundary logs atinfo(team_id/slot_id/trigger=human|agent, no sensitive payloads).Semantic drift called out for reviewers (spec-approved, not a bug)
The full-screen warmup overlay and the multi-member failure card now reflect the leader only.
server_countstill equals total member count but no longer implies all members are ready.Database / migrations
None.
requires_database_schema_change = false; no migration files added. Mailbox delivery semantics are unchanged (writesread=0, messages are never lost).Testing
cargo fmt --all -- --check— cleancargo clippy --workspace -- -D warnings— clean (push gate flags)just pushfull workspacecargo nextest run --workspace— 6916 passed, 18 skipped, 0 failedaionui-teamintegration suite (105 tests) covers: leader-only first start, lazy wakeup happy path, preserved-unread failure withblocked_reason=runtime_failed, source-gated leader notify (agent-triggered notifies / human-direct does not), Ready gating (Absent does not block / Attaching blocks), reconciliation skipping dormant, the per-member attach route (wake dormant / cross-userForbidden/ unknown-slotAgentNotFound), andteam.slotWorkChangedpayload shape.Coverage gap for reviewers: the attach route's service-level cross-user
Forbiddenand unknown-slotAgentNotFoundare asserted, but an HTTP-layer 401 (unauthenticated) and missing/invalid-CSRF assertion is not explicit (it relies on the shared team router middleware). An app-layer HTTP security assertion is worth adding.Notes
Backend correctness hinges on ordering:
ensure_member_runtime_lazydoes the synchronousreserve_attachbefore the background spawn, and resolves service/published-session first to avoid a danglingAttachinglease. This is a hard-to-observe path, hence theinfo-level boundary logs.Related PR
Frontend companion: iOfficeAI/AionUi#3712