Skip to content

feat(team): leader-only warmup with lazy teammate wakeup and per-member attach#670

Merged
piorpua merged 10 commits into
mainfrom
aionissue/task-20260721-001-001
Jul 23, 2026
Merged

feat(team): leader-only warmup with lazy teammate wakeup and per-member attach#670
piorpua merged 10 commits into
mainfrom
aionissue/task-20260721-001-001

Conversation

@piorpua

@piorpua piorpua commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Summary

Lazify team warmup. Previously, entering a team session warmed up every member via a batch rebuild_agent_processes (bounded concurrency + staggered start), showing a full-screen "waking up team" overlay. This was slow, race-prone, and all-or-nothing: a single member's attach failure failed the whole first start and kill_and_wait-ed the members that had already started.

This PR adopts spec approach A — reuse the existing single-member attach primitive and collapse the team runtime onto one attach path, with teammates started lazily.

Paired with the AionUi frontend PR on the same branch, which consumes the new dormant runtime status, the POST /api/teams/{id}/agents/{slot_id}/attach route, and the team.slotWorkChanged event.

Key changes

  • Leader-only first start: cold start attaches only the lead slot via reserve_attach + attach_member_runtime. On leader success it broadcasts leader Ready; other teammates broadcast the new Dormant status and stay Absent with no event loop; then session Ready(server_count = total member count) is broadcast.
  • Remove the batch machine: delete rebuild_agent_processes, spawn_rebuild_agent_process, cleanup_bootstrap_runtime_tasks and the concurrency/stagger logic. seed_ready is kept but gated #[cfg(test)]. The whole codebase now has a single attach orchestration path shared by leader cold-start, reconciliation, add_agent, spawn_agent, lazy wakeup, and directed retry.
  • Lazy wakeup (ensure_member_runtime_lazy): hooked into both delivery entry points (human→teammate enqueue_user_message, agent→teammate send_agent_message_from_agent) before publish_runtime_constraint. If the event loop is running it returns; otherwise it resolves service/published-session first (to avoid a dangling Attaching lease), does a synchronous reserve_attach (flips the snapshot to Attaching to close the race window), broadcasts Pending, and spawns the attach in the background. The lease deduplicates concurrent deliveries into a single attach.
  • Preserved-unread failure + source-gated leader notify: attach_member_runtime / spawn_attach_agent_process_bg take a notify_leader_on_failure: bool. A failed attach converges runtime state to Failed but no longer calls mark_read_batch on terminal_message_ids, so the triggering message stays read=0 and a retry re-drains it via reconcile_mailbox. The leader is notified only when notify_leader_on_failure = true (agent-triggered delivery and leader-initiated spawn_agent), not for human-direct deliveries, add_agent, reconciliation, leader cold-start, or the directed retry route.
  • Reconciliation skips dormant: reserve_member_runtime_reconciliation skips non-lead Absent members so a second ensure_session (re-warmup / model switch / retry) does not wake all dormant teammates; leader, Ready-repair, Failed-retry, and in-flight Attaching/Removing still reconcile.
  • refresh_member_runtime_status is leader-scoped: it reads only the leader snapshot (leader Ready ⇒ team Ready(server_count = total)); teammate dormant/pending/ready/failed is expressed per-member via agentRuntimeStatusChanged and no longer flips session status. This keeps the overlay from being raised by teammate state and a single teammate failure from raising the full-screen failure card.
  • Session-level Starting overlay is strictly leader-scoped; teammate reconciliation failures stay inline instead of failing the whole team; first-start recovery drain scans only the leader (teammate unread is deferred to lazy start + reconcile_mailbox).
  • New directed retry route POST /api/teams/{id}/agents/{slot_id}/attach: attach_agent handler in routes.rs, attach_agent_runtime in service.rs (load_owned_team → idempotent ensure_session_innerreserve_attach(slot, true) → background spawn_attach_agent_process_bg(..., false)). State-changing; auth + CSRF via the same team router middleware as add/remove/send.
  • New team.slotWorkChanged event + TeamSlotWorkChangedPayload: broadcasts per-slot work changes for run-less slots (e.g. the leader self-waking to drain its mailbox), which the run-scoped team.run* events do not cover. Naming follows domain.camelCaseAction.
  • Observability: per-member agentRuntimeStatusChanged and session-status broadcasts raised to info; lazy-wakeup boundary logs at info (team_id/slot_id/trigger=human|agent, no sensitive payloads).

Semantic drift called out for reviewers (spec-approved, not a bug)

The full-screen warmup overlay and the multi-member failure card now reflect the leader only. server_count still equals total member count but no longer implies all members are ready.

Database / migrations

None. requires_database_schema_change = false; no migration files added. Mailbox delivery semantics are unchanged (writes read=0, messages are never lost).

Testing

  • cargo fmt --all -- --check — clean
  • cargo clippy --workspace -- -D warnings — clean (push gate flags)
  • just push full workspace cargo nextest run --workspace6916 passed, 18 skipped, 0 failed
  • aionui-team integration suite (105 tests) covers: leader-only first start, lazy wakeup happy path, preserved-unread failure with blocked_reason=runtime_failed, source-gated leader notify (agent-triggered notifies / human-direct does not), Ready gating (Absent does not block / Attaching blocks), reconciliation skipping dormant, the per-member attach route (wake dormant / cross-user Forbidden / unknown-slot AgentNotFound), and team.slotWorkChanged payload shape.

Coverage gap for reviewers: the attach route's service-level cross-user Forbidden and unknown-slot AgentNotFound are asserted, but an HTTP-layer 401 (unauthenticated) and missing/invalid-CSRF assertion is not explicit (it relies on the shared team router middleware). An app-layer HTTP security assertion is worth adding.

Notes

Backend correctness hinges on ordering: ensure_member_runtime_lazy does the synchronous reserve_attach before the background spawn, and resolves service/published-session first to avoid a dangling Attaching lease. This is a hard-to-observe path, hence the info-level boundary logs.

Related PR

Frontend companion: iOfficeAI/AionUi#3712

zynx added 10 commits July 21, 2026 15:56
…urce-gated leader notify

Add TeamSession::ensure_member_runtime_lazy, hooked into both delivery
entry points (human enqueue and agent->teammate send) before publishing
the runtime constraint so the registry snapshot flips to Attaching
synchronously, closing the race window in spec 5.2.

Thread notify_leader_on_failure through spawn_attach_agent_process_bg and
attach_member_runtime: only agent-triggered failures wake the leader.
On attach failure, preserve unread mailbox rows (skip mark_read_batch) so
a later retry re-drains them via reconcile_mailbox.

The delivery-based lazy-wakeup tests require dormant teammates and are
added in the leader-only-first-start task where that state exists.
Cold start now attaches only the lead via the single member attach path and
publishes the session before attaching (dropping the startup guards first to
avoid re-entrant deadlock with refresh/cleanup). Teammates stay dormant
(Absent) and are broadcast as dormant; a leader attach failure bubbles to a
session-level Failed. The batch rebuild machine (bounded concurrency +
staggered starts + full-kill rollback) is removed, collapsing all attach
orchestration onto one path.

- refresh_member_runtime_status treats Absent as dormant (non-blocking for
  Ready); only in-flight Attaching/Removing block.
- reconciliation skips dormant (Absent non-lead) members so a second
  ensure_session cannot punch through lazy warmup.
- first-start recovery drain is restricted to the lead slot.

Existing tests updated to the leader-only model (repair/idempotent/rollback/
concurrent/fence/serialization now assert leader-only warmup and dormant
teammates).
POST /api/teams/{id}/agents/{slot_id}/attach and TeamSessionService::
attach_agent_runtime reuse the single attach path (reserve_attach with
retry_failed=true + background attach) to wake a dormant teammate or retry a
failed one. auth + CSRF are enforced by the shared team router middleware
layer, same as add/remove/send.

Tests: directed attach wakes a dormant teammate to ready; cross-user is
Forbidden; unknown slot is AgentNotFound. Also adds the deferred lazy-attach
coverage: a failed human-direct lazy attach preserves the unread delivery and
does not notify the leader (spec 5.4).
The leader-notify fork (spec 5.4c) was only covered on its negative side:
two tests assert human-direct and manual-add failures do NOT wake the
leader. Add the missing positive case required by the fix plan (Task A2)
and spec section 6: a leader-initiated spawn whose attach fails flows
through the single attach path with notify_leader_on_failure=true and
must write a 'failed to start its runtime' message to the leader mailbox
so it can re-delegate. Deterministic (no active-run dependency).
The full-screen "waking up the team" overlay is driven by session-level
TeamSessionStatus (useTeamWarmup maps `starting` -> warming). After the
leader-only warmup change, every member attach still broadcast a
session-level `Starting` (attach_member_runtime) and refresh_member_runtime_status
aggregated any teammate Attaching/Failed into session Starting/Failed. So a
dormant teammate's lazy wakeup, an add-member, or a teammate attach failure
resurfaced the overlay / full-screen failure card, even though the leader was
already ready and the team was usable.

Per spec 5.2/5.4/5.5 the session-level status is leader-scoped ("Ready =
leader ready") and only the leader may raise the overlay; teammate lifecycle
(dormant/pending/ready/failed) is surfaced inline per-member via
agentRuntimeStatusChanged.

- attach_member_runtime: only publish session `Starting` when the attaching
  agent is the lead; a teammate attach relies on its inline pending status.
- refresh_member_runtime_status: compute session status from the leader's
  snapshot only; teammate states no longer flip the session status.
- complete_member_runtime_reconciliation: only raise `Starting` when the
  reconcile work set includes the lead.

Tests: add waking_dormant_teammate_does_not_resurface_session_starting and
failed_teammate_wakeup_does_not_flip_session_to_failed; update the two
dynamic-add-failure tests to assert the new inline (leader-scoped) semantics.
A whole-team `ensure_session` is invoked far beyond first start: on page
mount, on every model switch, and before sends (via `warmupSession`). Its
reconciliation retries non-dormant members, and a failed teammate retried this
way still broadcast session-level `Failed` and returned `MemberRuntimeFailed`,
which the frontend surfaced as the full-screen failure card. So a single broken
teammate turned any unrelated interaction (switch another member's model, send
in another column) into a whole-team failure overlay.

The full-screen overlay is leader-scoped (spec 5.4/5.5): only a leader failure
blocks the team. Scope `complete_member_runtime_reconciliation` accordingly —
a teammate attach failure during reconciliation stays inline (its
`agentRuntimeStatusChanged=failed` already fired) and `ensure_session` returns
Ok (leader ready = team usable). A leader reconciliation failure still fails the
team. Dormant teammates are already skipped by reconciliation, so they are
never woken by these re-ensures.

Tests: add reensure_with_failed_teammate_keeps_team_usable_and_inline; update
the former conflict test to assert the new inline contract (re-ensure returns
Ok, removal still restores Ready).
Slot work state (which drives the send-box "processing" spinner + timer)
reached the frontend only bundled inside team.run* events, and
TeamRunManager.apply_work_summary drops any summary whose team_run_id is not
the currently-tracked run. So a slot doing run-less work — e.g. the leader
self-waking to drain its mailbox for a membership-change notification, which
creates background intents with no team_run_id — emitted neither run events nor
child-turn events on its running->idle transition. The frontend's
slotWorkBySlot[slot] stayed at the last "running" snapshot until a full
reconcile (which is why switching views cleared the stuck spinner).

Add a per-slot event, team.slotWorkChanged (TeamSlotWorkChangedPayload), and
publish the affected slot's current snapshot at every batch-lifecycle
transition (enqueue commit/abort, claim, mark_started, retry_start, runtime
constraint change, terminalize) regardless of run association, via a new
RunCausalityPort::publish_slot_work implemented by TeamRunManager. Run-less
transitions are now observable, so the frontend can clear per-slot state
without a reconcile.

Tests: coordinator asserts a run-less batch lifecycle publishes per-slot
snapshots ending Idle; emitter asserts the event name/payload shape.
These two broadcasts drive the team UI's two state surfaces — the full-screen
warmup overlay (session status) and the per-member column badge / send-box gate
(agent runtime status) — and were previously invisible in logs, which made the
overlay/stuck-state issues hard to diagnose from production logs.

Both are low-volume, important lifecycle state changes, so log at info
(production runs at info; debug would be invisible there). Fields are
non-sensitive: team_id, slot_id, status, phase, and the already-sanitized
public failure reason. The high-frequency mechanism events (team.run*,
childTurn, slotWorkChanged) stay at debug.
@piorpua
piorpua merged commit f542da8 into main Jul 23, 2026
6 checks passed
@piorpua
piorpua deleted the aionissue/task-20260721-001-001 branch July 23, 2026 08:00
piorpua pushed a commit that referenced this pull request Jul 23, 2026
🤖 I have created a release *beep* *boop*
---


##
[0.1.51](v0.1.50...v0.1.51)
(2026-07-23)


### Features

* **project:** add project-bind foundation (db + aionui-project)
([#672](#672))
([70eae04](70eae04))
* **session-port:** route claude/codex through the direct-CLI
SessionAgentTask
([#609](#609))
([14efff8](14efff8))
* **team:** leader-only warmup with lazy teammate wakeup and per-member
attach ([#670](#670))
([f542da8](f542da8))


### Bug Fixes

* **acp:** harden grok startup environment and npx recovery
([#662](#662))
([10cdd57](10cdd57))
* **ci:** restore main migration immutability guard
([7603d44](7603d44))
* **cron:** use host timezone for conversation cron
([#652](#652))
([19b36f8](19b36f8))
* **skills:** repair butler cron and doc drift (2026-07-22 audit)
([#664](#664))
([273ac47](273ac47))
* **system:** release keep-awake on shutdown
([#666](#666))
([6e0b3a7](6e0b3a7))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant