On-robot reliability: systemd hardening, health monitoring, startup self-checks

CLAUDE.md

@@ -87,6 +87,8 @@ Contains `urdf/mote.urdf.xacro` and `config/robot.yaml`. The xacro loads robot.y
 ### `mote_bringup` (Python/ament)
 Launch files, config, udev rules, NetworkManager drop-ins, and systemd services.
 
+**On-robot reliability** (see `mote_bringup/README.md`): the systemd services restart with backoff and never permanently give up (`Restart=always`, `RestartSec`/`RestartSteps`/`RestartMaxDelaySec`, `StartLimitIntervalSec=0`), order after the udev-tagged `dev-mote_*.device` units, and bound the journal. A pre-flight self-check (`self_check.py`, run as `mote-bringup`'s `ExecStartPre`; `pixi run self-check`) gates bringup on servo ping + lidar/camera/disk/clock/config and keeps the robot idle with a clear reason on failure. A health monitor (`health_monitor.py`, `mote-health.service` with a `Type=notify` watchdog; `pixi run health`) publishes per-subsystem `diagnostic_msgs/DiagnosticArray` on `/diagnostics_agg` and a single OK/DEGRADED/FAULT summary on `/health`. Driver and nav2 nodes are `respawn=True` for per-node recovery under the whole-service systemd restart. Battery voltage is **not** software-measurable (the power bank exposes no telemetry); `system_monitor` reports the Pi under-voltage flag as the only power signal.
+
 **Launch hierarchy:** the two mission launches (`mapping_launch.py`, `robot_launch.py`) each take a `base` arg (default true) that includes the hardware base, and a `use_sim_time` arg they forward to everything they include. The sim runs these *same* files with `base:=false`, supplying a Gazebo base in place of the drivers — so the missions are defined once and the sim exercises the real launch files.
 - `robot_launch.py` — nav mission: `mote_launch.py` (if `base`) + `nav2_launch.py` (drive a saved map). Forwards a `map` arg, defaulting to the active site's map (see Sites).
 - `mapping_launch.py` — mapping mission: `mote_launch.py` (if `base`) + `slam_launch.py` + `nav2_launch.py` (`localisation:=false`) + `record_launch.py` (`streams:=mapping`, unless `record:=false`): build/extend a map with SLAM *and* drive to goals autonomously while doing so, recording the session for map provenance.

mote_bringup/README.md

@@ -0,0 +1,111 @@
+# mote_bringup
+
+Launch files, config, udev rules, NetworkManager drop-ins, and systemd services
+for the robot. The package layout and launch hierarchy are documented in the
+top-level `CLAUDE.md`; this README covers the **on-robot reliability stack** —
+how the robot survives unattended operation.
+
+## systemd services
+
+Installed by `pixi run setup` (→ `systemd/install.sh`), which fills in the
+invoking user/home and enables them. Boot order:
+
+```
+mote-bringup  →  mote-slam  →  mote-nav
+  │  (self-check runs here      ↘  mote-record
+  │   as ExecStartPre)          ↘  mote-health
+```
+
+`mote-bringup` runs the self-check as its `ExecStartPre` gate — there is no
+separate self-check service; gating bringup gates everything downstream.
+
+| Service          | Runs                | Notes |
+|------------------|---------------------|-------|
+| `mote-bringup`   | `pixi run launch`   | Hardware base. `ExecStartPre` runs the self-check gate first. |
+| `mote-slam`      | `pixi run slam`     | `BindsTo`/`PartOf` bringup — restarts with it. |
+| `mote-nav`       | `pixi run nav`      | `BindsTo`/`PartOf` slam. |
+| `mote-record`    | `pixi run record`   | `Wants`/`PartOf` bringup — a recorder crash never takes down the drive stack. |
+| `mote-health`    | `pixi run health`   | Health monitor; `Type=notify` + `WatchdogSec` watchdog. |
+
+**Hardening** (all services):
+
+- `Restart=always` with backoff (`RestartSec=2`, `RestartSteps=5`,
+  `RestartMaxDelaySec=30`): a crashed service restarts, backing off from 2 s to a
+  30 s ceiling instead of hammering.
+- `StartLimitIntervalSec=0`: never permanently give up. An unattended robot must
+  self-heal when reconnected hardware reappears; the backoff prevents a busy
+  loop, so there is no reason to latch into a failed state.
+- **Device ordering**: the udev rules tag the servo/lidar devices
+  `TAG+="systemd"`, so systemd synthesises `dev-mote_servos.device` /
+  `dev-mote_lidar.device`. `mote-bringup` orders `After=` / `Wants=` them, so
+  bringup waits for the hardware to enumerate but a mid-run flap does not
+  force-kill the stack (the self-check and health monitor own that).
+- **journald sizing**: `systemd/journald-mote.conf` bounds the persistent
+  journal (`SystemMaxUse=500M`, `SystemKeepFree=1G`, `MaxRetentionSec=2week`) so
+  always-restarting services can never fill the SD card.
+
+**Two layers of process recovery** (see also `test/chaos/`):
+
+1. **Node crash** → the launch system relaunches just that node
+   (`respawn=True` on the drivers in `mote_launch.py` and the nav2 servers in
+   `nav2_launch.py`; the nav2 lifecycle managers reconnect the respawned node).
+2. **Launch/process crash** → systemd restarts the whole service.
+3. **Health-monitor hang** → the `WatchdogSec` watchdog restarts `mote-health`.
+
+## Startup self-check — `self_check.py`
+
+Runs as `mote-bringup`'s `ExecStartPre` (and by hand: `pixi run self-check`).
+Fast, static pre-flight checks — no ROS graph — that gate the launch:
+
+- **servos**: bus device present *and* an SCServo ping answers (`servo_ping`, the
+  drive IDs from `robot.yaml`) — CRITICAL.
+- **lidar**: device present and openable — CRITICAL.
+- **camera**: device present and openable — advisory.
+- **disk**: free space on `MOTE_HOME` (< 500 MB CRITICAL, < 2 GB warn).
+- **clock**: system time looks NTP-synced (an RTC-less Pi boots at the epoch) —
+  advisory.
+- **config**: `robot.yaml` parses (CRITICAL); an active site is resolved
+  (advisory — mapping runs without one).
+
+Any failed **CRITICAL** check → non-zero exit → the `ExecStartPre` fails →
+bringup does not start (robot stays in safe idle) and systemd retries with
+backoff, so replugging the lidar recovers on its own. The verdict is written to
+`$MOTE_HOME/self_check_status.yaml` and printed to journald.
+
+Runtime data-flow liveness (is `/scan` *publishing*?) is deliberately **not**
+checked here — the drivers are not up yet. That is the health monitor's job.
+
+## Health monitor — `health_monitor.py`
+
+Runs as `mote-health.service` (or `pixi run health`). Watches subsystem liveness
+and publishes, every second:
+
+- **`/diagnostics_agg`** (`diagnostic_msgs/DiagnosticArray`) — one
+  `DiagnosticStatus` per subsystem (scan, filtered scan, joint states, camera,
+  odom TF, localisation TF), the host status folded in from `system_monitor`'s
+  `/diagnostics`, the last self-check verdict, and a rolled-up `mote` status. The
+  standard form the fleet layer can lift later.
+- **`/health`** (`std_msgs/String`) — a single human-readable summary line:
+  `OK` / `DEGRADED: camera stale` / `FAULT: scan stale (…)`. Easy to eyeball:
+
+  ```bash
+  pixi run -- ros2 topic echo /health
+  ```
+
+**Criticality → roll-up**: a stale *critical* subsystem (scan, filtered scan,
+joint states, odom TF) is a **FAULT**; a stale non-critical one (camera,
+localisation TF) or a fresh-but-slow one is **DEGRADED**. Expectations live in
+`config/health.yaml`, overridable per-robot at `~/.mote/health.yaml`.
+
+The monitor is also the systemd watchdog feeder: it sends `READY=1` once up and
+pets the watchdog on every publish (`sd_notify.py`, a dependency-free
+`$NOTIFY_SOCKET` client that no-ops outside systemd).
+
+## Known gap: battery voltage
+
+The USB-C power bank exposes **no state-of-charge or voltage telemetry**, so the
+robot cannot see its own battery in software. The only power signal available is
+the Raspberry Pi firmware's `get_throttled` under-voltage bitfield, which
+`system_monitor` already reports (a brown-out shows as `DEGRADED`). True battery
+sensing needs a hardware change (a fuel-gauge / INA-class sensor on the power
+rail) and is tracked as a follow-up — see the reliability follow-up task.

mote_bringup/config/health.yaml

@@ -0,0 +1,58 @@
+# Health monitor liveness expectations.
+#
+# The health_monitor node checks each entry for freshness (a message seen within
+# `timeout` seconds) and, when `min_rate` is set, for sustained rate over the
+# reporting window. `critical: true` subsystems drive the robot-level FAULT
+# state; non-critical ones only drive DEGRADED. A per-robot ~/.mote/health.yaml
+# overrides this packaged default (same pattern as perception.yaml).
+
+# Aggregation + publish period, seconds.
+period: 1.0
+
+topics:
+  - name: scan
+    topic: /scan
+    type: sensor_msgs/msg/LaserScan
+    min_rate: 5.0
+    timeout: 2.0
+    critical: true
+  - name: scan_filtered
+    topic: /scan_filtered
+    type: sensor_msgs/msg/LaserScan
+    min_rate: 5.0
+    timeout: 2.0
+    critical: true
+  - name: joint_states
+    topic: /joint_states
+    type: sensor_msgs/msg/JointState
+    min_rate: 5.0
+    timeout: 2.0
+    critical: true
+  # Camera feeds perception, not the safety-critical drive loop, so a camera
+  # dropout is DEGRADED, not FAULT. The compressed stream is subscribed to
+  # avoid pulling the full raw image just to measure liveness.
+  - name: camera
+    topic: /image_raw/compressed
+    type: sensor_msgs/msg/CompressedImage
+    min_rate: 5.0
+    timeout: 5.0
+    critical: false
+
+# TF edges to watch. odom->base_footprint is the live odometry corrector
+# (kinematic_icp); map->odom appears only once localised (slam or AMCL), so it
+# is non-critical — its absence means "not yet localised", not "broken".
+tf:
+  - name: odometry
+    parent: odom
+    child: base_footprint
+    timeout: 2.0
+    critical: true
+  - name: localization
+    parent: map
+    child: odom
+    timeout: 5.0
+    critical: false
+
+# Fold the host status published by system_monitor on /diagnostics (CPU, temp,
+# under-voltage) into the robot-level summary so a brown-out shows as DEGRADED.
+subscribe_diagnostics: true

mote_bringup/launch/mote_launch.py

@@ -51,16 +51,25 @@ def generate_launch_description():
     )
     wheel_params_file.close()
 
+    # respawn=True gives per-node recovery: if a driver process crashes, the
+    # launch system relaunches it within respawn_delay. This is the inner layer;
+    # systemd restarts the whole service only if `pixi run launch` itself dies.
+    # The spawners are one-shot and re-run via the OnProcessStart handler when
+    # controller_manager respawns, so they are not marked respawn.
+    respawn = {"respawn": True, "respawn_delay": 2.0}
+
     robot_state_publisher = Node(
         package="robot_state_publisher",
         executable="robot_state_publisher",
         parameters=[robot_description],
+        **respawn,
     )
 
     controller_manager = Node(
         package="controller_manager",
         executable="ros2_control_node",
         parameters=[robot_description, controller_config, wheel_params_file.name],
+        **respawn,
     )
 
     joint_state_broadcaster_spawner = Node(
@@ -90,6 +99,7 @@ def generate_launch_description():
                 "scan_mode": "Standard",
             }
         ],
+        **respawn,
     )
 
     laser_filter = Node(
@@ -100,6 +110,7 @@ def generate_launch_description():
             ("scan", "/scan"),
             ("scan_filtered", "/scan_filtered"),
         ],
+        **respawn,
     )
 
     cam = cfg["camera"]
@@ -124,11 +135,13 @@ def generate_launch_description():
         executable="v4l2_camera_node",
         name="camera",
         parameters=[camera_params],
+        **respawn,
     )
 
     system_monitor = Node(
         package="mote_bringup",
         executable="system_monitor",
+        **respawn,
     )
 
     localization = IncludeLaunchDescription(

mote_bringup/launch/nav2_launch.py

@@ -91,12 +91,18 @@ def generate_launch_description():
 
     cmd_vel_remap = ("/cmd_vel", "/diff_drive_controller/cmd_vel")
 
+    # respawn=True relaunches a crashed nav2 server; the lifecycle managers
+    # (attempt_respawn_reconnection defaults true) then reconnect their bond and
+    # re-activate it, so a single server crash recovers without restarting nav.
+    respawn = {"respawn": True, "respawn_delay": 2.0}
+
     map_server = Node(
         package="nav2_map_server",
         executable="map_server",
         parameters=[nav2_params, {"yaml_filename": LaunchConfiguration("map")}],
         condition=IfCondition(localisation),
         output="screen",
+        **respawn,
     )
 
     amcl = Node(
@@ -105,6 +111,7 @@ def generate_launch_description():
         parameters=[nav2_params],
         condition=IfCondition(localisation),
         output="screen",
+        **respawn,
     )
 
     controller_server = Node(
@@ -113,20 +120,23 @@ def generate_launch_description():
         parameters=[nav2_params, critic_params_file.name],
         remappings=[cmd_vel_remap],
         output="screen",
+        **respawn,
     )
 
     smoother_server = Node(
         package="nav2_smoother",
         executable="smoother_server",
         parameters=[nav2_params],
         output="screen",
+        **respawn,
     )
 
     planner_server = Node(
         package="nav2_planner",
         executable="planner_server",
         parameters=[nav2_params],
         output="screen",
+        **respawn,
     )
 
     behavior_server = Node(
@@ -135,20 +145,23 @@ def generate_launch_description():
         parameters=[nav2_params],
         remappings=[cmd_vel_remap],
         output="screen",
+        **respawn,
     )
 
     bt_navigator = Node(
         package="nav2_bt_navigator",
         executable="bt_navigator",
         parameters=[nav2_params],
         output="screen",
+        **respawn,
     )
 
     waypoint_follower = Node(
         package="nav2_waypoint_follower",
         executable="waypoint_follower",
         parameters=[nav2_params],
         output="screen",
+        **respawn,
     )
 
     lifecycle_manager_localization = Node(
@@ -160,6 +173,7 @@ def generate_launch_description():
                 "autostart": True,
                 "node_names": LOCALIZATION_NODES,
                 "bond_timeout": 10.0,
+                "attempt_respawn_reconnection": True,
             }
         ],
         condition=IfCondition(localisation),
@@ -175,6 +189,7 @@ def generate_launch_description():
                 "autostart": True,
                 "node_names": NAVIGATION_NODES,
                 "bond_timeout": 10.0,
+                "attempt_respawn_reconnection": True,
             }
         ],
         output="screen",