Fleet architecture design: multi-robot operations from day one

[MJohnson459]

Fleet architecture design doc: docs/design/fleet.md (design only, no implementation).

Answers the seven design questions with a recommendation + rationale each, grounded in existing repo seams (file:line citations) and verifiable tech claims (docs linked; unverified items flagged):

1. Topology — per-robot mote_agent (bridge/reporter, never in the control loop; robot stays autonomous offline) + central fleet server; fleet server and GPU inference server kept as SEPARATE roles (opposite uptime/compute/trust profiles).
2. Transport — layered: Tailscale (WireGuard mesh) as the substrate so off-LAN/NAT/firewall/identity/encryption are one problem not N; Foxglove (foxglove_bridge) adopted for rich per-robot view + teleop; MQTT (Mosquitto) for the fleet control plane (LWT/retained/namespacing = the exact semantics needed). rmw_zenoh deferred (not production-ready on Jazzy, targets Kilted); zenoh-plugin-ros2dds kept as documented fallback; rosbridge superseded.
3. Identity — new ~/.mote/robot.yaml (id/name/site) as the primary key, decoupling from the drifting auldbot/mote hostname; per-robot state in ~/.mote vs shared code in the package; mote enroll flow.
4. Site/map registry — fleet server as source of truth; distribution reuses the existing immutable-revision + atomic-symlink-flip machinery (sites.py:248-252); server-gated publish; two-mapper conflict resolved by operator promotion, never auto-merge.
5. Ops UI — hybrid: adopt Foxglove for deep single-robot view, build only a thin fleet roster+dispatch+map-overlay web app (maps are PNG by design, so pose-on-map is a dot on a PNG).
6. Updates — consume the pending prefix.dev mote channel (one update mechanism, not two): install-alongside + health-check + rollback + ring rollout; ~/.mote outside the package means updates can't clobber identity.
7. Security — proportionate: WireGuard gives free end-to-end encryption + no public exposure (makes the unauthenticated inference wire safe across the internet), Tailscale ACLs + per-channel auth on top; no custom PKI for v1.

Includes a mermaid component diagram, explicit v1 non-goals (no traffic coordination, no map auto-merge, no cross-robot task allocation, no public-exposed services), and an 8-milestone breakdown (M0-M7) sized so each is one dispatchable follow-up task, with a v0 line (one robot fully remotely operable off-LAN) and a v1 line (second robot enrolled). Appendix lists the (verify) items.

Verified: doc committed on branch design/fleet-architecture (commit a71b0eb); pre-commit hooks pass. No implementation, per task scope.

[MJohnson459]

Overall a solid plan. I need more details on how this scales as that's an important consideration. A lot of the choices just have the positives, but lets be clear about the costs as well. Finally this introduces a solid pipeline for the robots, lets also consider what the pipeline looks like for the servers and other components as well.

[MJohnson459]

Tailscale sounds good. What are the limitations, costs, and tradeoffs?

[MJohnson459]

Added a Cost & tradeoffs block under Layer 0 (Tailscale): per-user pricing but robots are tagged devices ($1/device/mo past 50), the coordination plane is cloud-locked, DERP-relay latency when a direct path can't be punched, and a single flat tailnet doesn't multi-tenant. Escape hatch documented: self-hosted Headscale.

[MJohnson459]

What is the realistic cost scaling? This is an example project so how would this scale to say 10,000 robots across different sites/customers?

[MJohnson459]

Added a Scaling section with a per-component regime table and cost estimates. Concretely for 10k robots across sites/customers: Tailscale tagged-device overage lands around **$10k/mo**, which is exactly what forces the Regime-C pivot — drop the full mesh, have robots hold only an outbound TLS connection to a per-region EMQX/HiveMQ broker cluster (scales to millions), self-host Headscale, and move the registry to object storage + CDN. The point of the section: these are swaps behind the same seams (robot_id keying, per-robot topic tree, object-storable bundles), not a redesign.

[MJohnson459]

The same question here - how would the cost scale to 10,000 robots etc?

[MJohnson459]

Covered in Foxglove's Cost & tradeoffs note and the Scaling table: free tier is 3 users / 5 devices (the 5-device cap is the first wall we hit), then per-seat/per-device paid tiers, Enterprise quote-only. At platform scale the option is Foxglove Enterprise or Rerun-offline + a custom multi-tenant live panel — and because Foxglove is adopted at a seam (it only consumes the robot's ROS graph), swapping it doesn't touch robot code.

[MJohnson459]

Foxglove also looks good, what are the tradeoffs? Is this a full replacement for rviz? What alternatives were considered?

[MJohnson459]

Added Is it a full RViz replacement? No — RViz's interactive-marker primitives stay RViz-only, so we keep pixi run rviz for local dev and use Foxglove for remote/fleet viewing; they don't compete. Also added an Alternatives considered list: RViz (local only), Rerun (open source, but no live multi-user/cloud sessions — wrong for live teleop), rosbridge (superseded), build-our-own (no payoff). Plus the tradeoffs: vendor dependency + per-seat cost + camera/pointcloud bandwidth.

[MJohnson459]

Same question, what are the downsides/tradeoffs for MQTT? Can a web UI actually connect to an MQTT broker directly, assuming we want position updates etc to be displayed?

[MJohnson459]

Added MQTT limitations: pub/sub only (request/response is a reply-topic+correlation-id convention we implement), no message history/replay (last retained value only — add a DB subscriber or Kafka bridge later), payloads must stay small (maps/frames go over HTTP/Foxglove instead), and Mosquitto doesn't cluster (swap to EMQX/HiveMQ at scale). And yes — a web UI connects to the broker directly via MQTT-over-WebSockets (MQTT.js): the dashboard subscribes to mote/+/pose and mote/+/health over WS and renders live positions with no polling and no middle service. That's now spelled out in Layer 2 and Q5.

[MJohnson459]

This works for the raw map, but then we will have things like the robot positions, zone areas, etc. Is this still good? What would the limitations be?

[MJohnson459]

Addressed in Q5 with the actual coordinate math (pose/zone metres → pixel via map.yaml resolution + origin, heading arrow from the quaternion) and an honest limits paragraph: it's a static top-down 2D raster overlay — fine for a roster thumbnail and one/a few robots, but not for dense multi-robot scenes (marker overlap), 3D/point clouds, live costmap/laser layers, or very large sites (needs pan/zoom). The rule: the thin overlay is the fleet-glance view; the moment an operator needs depth they deep-link into Foxglove. We deliberately don't grow it into a second Foxglove.

[MJohnson459]

Just to reiterate on an above point - does MQTT have any limitations in what clients can connect, or what data can be sent?

[MJohnson459]

Added to Layer 2. Clients: any MQTT client, including browsers over MQTT-over-WebSockets; who may connect is broker auth/ACL (per-robot creds, topic-prefix ACLs — Q7), not anonymous. Data limits: pub/sub only (no native request/response), only the last retained value per topic (no history/replay — that's Kafka's job; add a DB subscriber if we want a time-series), and payloads should stay small (control/telemetry, not maps or frames — those go over HTTP/Foxglove).

[MJohnson459]

This is sensible, but we would need to understand disk and network costs for this strategy, and possibly ways to minimise it.

[MJohnson459]

Agreed, and added the numbers + mitigations to M5/Q6. Disk: two envs is gigabytes, so keep exactly two slots (current + one rollback) and rely on conda/pixi hard-linking identical packages across envs — a point release costs disk only for changed packages (megabytes, not gigabytes). Network: conda fetches per-package, so an update pulls only the lockfile diff, not the whole env; at scale put a per-site channel mirror/CDN so N robots don't each pull from prefix.dev over WAN. Staging is a separate prefix, so a failed download never touches the running robot. (Flagged the hardlink/per-package behaviour as verify-on-Pi.)

[MJohnson459]

This make sense, but only covers the robot OTA. What about the inference server and fleet server?

[MJohnson459]

Added a whole "The other pipelines: fleet server & inference server" section. Inference server: same prefix.dev channel, own (torch, no-ROS) env, stateless so updates are blue/green on a second port then flip inference_host port; not availability-critical (robots idle-degrade when it's down). Fleet server: not on the robot OTA channel — it's server software, provisioned via IaC/container-compose and updated by normal blue/green image deploy; its state (retained broker state + registry) is what to back up; robot autonomy survives its downtime. Net: three pipelines, one release channel for the two ROS-software roles.

[MJohnson459]

These look good as tasks - be explicit if there are any dependencies between them. Consider scoping so we can dispatch many tasks in parallel.

[MJohnson459]

Added an explicit dependency graph at the top of Milestones plus per-milestone Depends/Blocks/Parallel-with annotations, and scoped for parallelism: after M0, M1 / M2 / Ms run in parallel (independent transports/roles); M4 and M5 run in parallel after M1; M7 is cross-cutting from M0. Also added Ms (server pipelines) as its own dispatchable task.

[MJohnson459]

Thanks — addressed the three overall points across the revision (commit b46317c, ready to push):

• Costs, not just positives. Every transport choice now carries a Cost & tradeoffs note (Tailscale per-device pricing + cloud-locked coordinator + relay latency; Foxglove per-seat + 5-device free cap + bandwidth; MQTT pub/sub-only + no replay + Mosquitto doesn't cluster; inference WAN-latency).
• Scaling. New Scaling: one robot → ten thousand section — a per-component regime table (homelab → small fleet → 1k–10k platform), cost estimates (incl. the ~$10k/mo Tailscale-tag figure at 10k), the three real breaking points, and why each pivot is a swap behind a seam rather than a redesign.
• Pipelines for the servers, not just robots. New The other pipelines section covering provisioning + updates + failure posture for the inference server (same channel, own stateless env, blue/green) and the fleet server (IaC/container deploy, not the robot OTA channel), plus an Ms server-pipelines milestone.

Individual replies are on each inline thread. Note: I could not push (remote-write guard) — the commit is ready on design/fleet-architecture for you to push, after which the inline replies will line up with the new content.

[MJohnson459]

When does this happen? If the robot needs a Tailscale key to join the network, how can it talk to the fleet server?

[MJohnson459]

Good catch — that was a real bootstrap trap. Fixed by making the Tailscale key a provisioning-time secret baked into the image, not a runtime fetch: at imaging time a single-use pre-auth key (and a short-lived enrollment token) are written into cloud-init user-data. On first boot the Pi runs tailscale up with that baked-in key and joins the tailnet before it ever contacts the fleet server; only then does mote enroll reach the server over the tailnet to get its id/site. So the robot is never in the position of needing the network to fetch the key that gets it onto the network.

[MJohnson459]

Is there no way to limit DDS to one machine and avoid this problem?

[MJohnson459]

Yes — and that's the better answer, so I've switched to it. ROS 2 Jazzy has ROS_AUTOMATIC_DISCOVERY_RANGE=LOCALHOST (replaces the deprecated ROS_LOCALHOST_ONLY), which confines DDS discovery to the local machine. Since the agent and foxglove_bridge run on the same Pi (the agent is the sole egress), nothing off-box needs the robot's DDS graph, so we pin discovery to localhost. That dissolves the domain-ID problem entirely — two robots on one LAN can't see each other regardless of domain, so there's nothing to allocate or manage, and domain_id is dropped from robot.yaml. The one caveat (bench dev where a workstation shares the LAN graph) is a dev-time flag, not fleet ops.

[MJohnson459]

This shouldn't be a new section - fix the process above, don't repeat it.

[MJohnson459]

Agreed — merged. There's now a single "Provisioning a new robot — one flow" subsection; the duplicate expanded version is gone.

[MJohnson459]

How does this fit with our current raspberry pi image? We use the raspberry pi imager to write an image and use raspberry pi connect for peace of mind.

[MJohnson459]

Reconciled it with the actual workflow rather than inventing a bespoke image. Raspberry Pi Imager's OS customization now writes cloud-init user-data to the boot partition (Imager 2.0+), which can install packages and run first-boot commands headlessly — so the "golden image" is just our stock Raspberry Pi OS image + a user-data template (Tailscale key, enroll token, first-boot provisioning), not a custom image to maintain. And Raspberry Pi Connect stays as the independent break-glass path (remote shell/screen over Pi's own relay, not dependent on our tailnet or fleet server) — so a robot with a botched key or broken agent is still recoverable. Tailscale = fleet data plane; Pi Connect = peace-of-mind fallback, exactly as today.

[MJohnson459]

I agree no laser or point clouds, but what about hundreds of robots? And large sites? We definitely want pan and zoom, the ability to follow a robot, etc.

[MJohnson459]

Fair — I'd under-scoped it. Q5 now describes a real interactive 2D fleet map, not a static thumbnail: the PNG is just the basemap, with a pannable/zoomable canvas (Canvas2D → WebGL as marker count grows), click-to-follow-a-robot, marker clustering + viewport culling so hundreds of robots stay smooth (positions are tiny MQTT payloads, so it's a rendering problem, not a data one), and tiled/pyramid basemaps for large sites (pre-rendered per map revision, which the registry already stores immutably), plus a floor switcher. What still deep-links to Foxglove is only the single-robot depth view — 3D, point clouds, live costmap/laser layers, teleop — not the 2D fleet picture.

[MJohnson459]

Will we have CPU space to run two versions of the stack at the same time? How would that work with hardware connections? Or do we need to stop the old stack first, even if we keep the env "active".

[MJohnson459]

Important clarification, now in Q6: "install-alongside" is two envs on disk, never two stacks running. The Pi doesn't have the CPU for two stacks, and the hardware is exclusive anyway (one process set holds the servo/lidar/camera ports — mote_hardware opens the port in on_activate). So the new env is only staged on disk while the old stack runs; the cutover stops the old stack, then starts the new one, and because that's a brief downtime it's gated to when the robot is idle/charging, not mid-mission. Rollback = restart the old env we kept on disk. "Keep the old env active" meant installed-for-rollback, not running.

[MJohnson459]

This is very manual and prone to error - how can we smooth this out?

[MJohnson459]

Agreed, "update it whenever" was too loose. The inference server is now a managed compute node: it runs the same mote_agent in a compute-node mode, reports its version/health to the fleet server (shows up in the roster), and its updates are orchestrated/gated/reported like a robot's — not hand-run over SSH. Because it's stateless it's actually easier to automate than the robot: automated blue/green — stand up the new version on a second port, run a probe request as the health gate, flip the served port on success (and push the new inference_host port to its robots), auto-rollback on failure. No hardware exclusivity, so both versions really can run side-by-side during the check.

[MJohnson459]

Design review (voro #166)

Verdict: approve with comments. All seven questions answered with recommendation + rationale, component diagram, non-goals, v0/v1 milestones sized as dispatchable tasks, verification ledger — every acceptance criterion met. Issues below are fixable in-place; nothing structural.

Citation check

All 26 repo citations were checked against main — 21 exact, 5 have stale line numbers (content accurate in every case):

Doc cites	Actually at
sites.py:248-252 (atomic flip)	sites.py:254-258
sites.py:390-400 (artifact check)	sites.py:396-406
sites.py:426-434 (use-map)	sites.py:432-441
task_server.py:56-57 (String topics)	task_server.py:67-68
task_server.py:83-97 (grammar/status)	docstring (1-15) + lines 93/110/121/124

Worth fixing since the doc's credibility rests on file:line grounding (they also appear in the M0–M6 seam lists).

Two of the doc's (verify) items resolve positively and can be struck from the ledger:

• ros-jazzy-foxglove-bridge exists on robostack-jazzy: 3.3.0, with a linux-aarch64 build (June 2026) — the Pi is covered.
• ROS_AUTOMATIC_DISCOVERY_RANGE is supported by rmw_cyclonedds — merged in rmw_cyclonedds#429 (April 2023, so in Iron and Jazzy). The Q3 answer stands.

Substantive comments (ordered by importance)

1. New (verify) item for the ledger: participant cap under LOCALHOST discovery. The rmw_cyclonedds implementation of discovery-range defaults MaxAutoParticipantIndex to 32 — roughly 32 discoverable ROS processes per host in localhost-peer mode. A full stack (bringup + Nav2's nodes + SLAM + perception + task server + agent + foxglove_bridge) may flirt with that. Verify the process count on the robot, or note the CYCLONEDDS_URI bump. Q3's whole answer leans on this setting.

2. The exactly-once claim is overstated (Q1: "a dropped command is neither lost nor double-applied"). task/command is a bare std_msgs/String with no correlation id at the ROS seam — accepted:/rejected: can't be attributed to a specific command if two are in flight. The property is achievable, but only as a convention the agent implements (single in-flight command per robot, treat the next accepted:/rejected: as the ack, MQTT correlation-id upstream of that). The doc mentions the reply-topic convention under MQTT tradeoffs but should state the single-inflight rule explicitly, since it's what makes the Q1 claim true.

3. M0 smuggles in a fleet-server component. "Server-allocated id/name" plus mote enroll needs an enrollment endpoint and a registry row store — i.e., some fleet server exists before M1 builds the control plane, yet M0 says "Depends on: nothing" and Ms is listed as parallel-with-M1. Either (a) v0's M0 uses operator-assigned IDs written straight to robot.yaml, with server allocation deferred to M1, or (b) M0 explicitly includes a minimal enrollment service and is sized accordingly. As written, M0 is the heaviest milestone while looking like the lightest.

4. Server-side map re-validation implies sharing sites.py logic outside the robot package. The fleet server (a container, no ROS) re-runs the four-artifact invariant + map.yaml sanity checks. That's either an extracted non-ROS shared lib or duplicated logic — worth one sentence in Q4/M4 so the implementer doesn't discover it mid-task.

5. Dispatch authorization when the browser talks straight to the broker. Q5's dashboard publishes task/command directly over MQTT-over-WS, which means dispatch authz lives entirely in broker ACLs for the operator credential — the fleet API never sees the command. Q7 lists per-robot creds but not the operator/browser broker role. Either define that ACL (read mote/+/health|pose, publish mote/+/task/command) or route dispatch through the fleet API and keep the browser read-only on the broker.

6. The roster promises battery, the robot doesn't publish it. Nothing in the tree publishes battery state today. Fine to keep in the health schema as a field, but flag it as future, and M1's health payload deserves a defined schema either way.

7. PR description drift. The PR body says robot.yaml is "id/name/domain_id" — the doc dropped domain_id when it moved to localhost-pinned discovery (its robot.yaml is id/name/site). Update the PR body to match.

Nits

• "The agent buffers health/telemetry" (Q1) sits oddly next to the no-time-series non-goal and MQTT's retained-last-value-only. Either specify a small reconnect buffer or soften to "re-publishes current state on reconnect."
• M5 assumes the prefix.dev channel serves linux-aarch64 robot packages — a requirement on the parallel release task worth stating in M5's dependency line.
• Cost arithmetic checked out (200 robots → (200−50)×$1 = $150/mo ✓).

What's strong (keep exactly as-is)

• Localhost-pinned DDS dissolving the domain-ID question — turns an allocation/management problem into a non-problem, and makes agent-as-sole-egress load-bearing rather than decorative.
• The layered transport answer with per-layer "Cost & tradeoffs" — especially the Zenoh treatment, which names the real consolidation it would buy and still correctly defers it.
• Map distribution reusing the existing revision+flip machinery — near-zero new invention; the no-auto-merge stance is correct given the frame-origin docstring.
• The scaling table with actual pricing breakpoints (Tailscale tag overages, Mosquitto non-clustering, Foxglove 5-device cap) makes homelab-vs-platform concrete instead of hand-wavy.

[MJohnson459]

Thanks — thorough review. Addressed all of it (commit df2359b, ready to push):

Citations — fixed all 5 stale line numbers (verified against main), in the body and the M0–M6 seam lists: sites.py flip 254-258, artifact-check 396-406, use-map 432-441; task_server.py topics 67-68, grammar/status → docstring 1-15. Struck the two resolved (verify) items (foxglove-bridge 3.3.0/aarch64, discovery-range in rmw_cyclonedds) from the ledger.

Substantive

1. Participant cap — added as a (verify) item in Q3 and the ledger (MaxAutoParticipantIndex ~32, count on-robot, CYCLONEDDS_URI bump), noting Q3 leans on it. Good catch.
2. Exactly-once overstated — replaced with the explicit single-in-flight-command rule in Q1 (hold until terminal status, attribute the next accepted/rejected as the ack, MQTT correlation id upstream), since task/command is a bare String.
3. M0 smuggled a server — fixed: M0 now uses an operator-set id (no server, truly zero-dependency); the enrollment endpoint + server allocation move to M1 where the registry exists. Q3 notes the sequencing.
4. Shared validation logic — Q4/M4 now say the artifact/map.yaml invariant must be an extracted ROS-free module shared with sites.py, not duplicated in the no-ROS fleet server.
5. Dispatch authz — defined the operator broker ACL in Q7 (subscribe mote/+/health|pose, publish only mote/+/task/command) and noted the route-through-API alternative in Q5.
6. Battery — flagged as a future health field in Q5; M1 now calls for a defined health schema.
7. PR body drift — updated the PR description robot.yaml to id/name/site.

Nits — "buffers telemetry" softened to "re-publishes current state on reconnect"; M5 dependency line now names the linux-aarch64 build requirement; cost arithmetic confirmed.

Verdict noted (approve with comments) — nothing structural changed, all in-place.

[MJohnson459]

Follow-up to the review — three milestone-level decisions for the revision:

1. M3: settle review comment 5 as "dispatch through the fleet API." The dashboard stays read-only on the broker (MQTT-over-WS for roster/pose); task/command publishes go through the API. Rationale as in the review: dispatch authorization and an audit trail need a mediating service; broker ACLs alone can't attribute or gate per-operator dispatch.
2. M0: take the operator-assigned branch from review comment 3. v0 identity is a hand-written robot.yaml; mote enroll and server allocation move to M1 where the fleet server first exists. M0 becomes as light as its dependency line claims.
3. All milestones: add "interface contract documented as a versioned schema" to acceptance where the milestone creates a surface (M1 topic tree + health payload, M3 dispatch API, M4 bundle/registry API). A professional fleet layer's interfaces are specs, not incidental implementation details — and it keeps future consumers (dashboards, tooling, other agents) off the implementation internals.

[MJohnson459]

All three decisions applied (commit 56de04b, ready to push):

1. M3 dispatch through the fleet API — settled that way. The browser is now read-only on the broker (roster/pose over MQTT-over-WS); the dispatch box POSTs to the fleet API, which authorizes the operator, writes an audit record, then publishes task/command. Updated Q5 (dispatch section rewritten), Q7 (operator broker credential is now subscribe-only; dispatch authz + audit live in the API), and M3 (renamed "Thin fleet UI + dispatch API").
2. M0 operator-assigned identity — already landed in df2359b: M0 uses a hand-written robot.yaml with no server; mote enroll + server allocation are in M1. Confirmed consistent.
3. Versioned interface schemas in acceptance — added the principle to the Milestones preamble ("any milestone that creates a consumed surface must ship it as a documented, versioned schema/spec") and to the three acceptance criteria: M1 (topic tree + health payload), M3 (dispatch API), M4 (bundle/registry API).

Also fixed two shorthand citations in the M4 seam list that the earlier line-number pass missed (:390-400→:396-406, :426-434→:432-441).