On-robot reliability: systemd hardening, health monitoring, startup self-checks

[MJohnson459]

On-robot reliability: systemd hardening, continuous health monitoring, and a pre-flight startup self-check so the robot survives unattended operation. Branch reliability-hardening (1 commit, pixi run build clean, 36 mote_bringup tests pass, lint/shellcheck green, systemd-analyze verify clean on all units).

What changed

systemd hardening (mote-bringup/slam/nav/record + new mote-health)

• Restart=always with backoff (RestartSec=2, RestartSteps=5, RestartMaxDelaySec=30) and StartLimitIntervalSec=0 — recovers from clean exits too and never latches into a permanent failed state, so the robot self-heals when reconnected hardware reappears, while backoff prevents bus hammering.
• Device ordering: udev rules now TAG+="systemd", so bringup orders After=/Wants= dev-mote_servos.device/dev-mote_lidar.device (waits for enumeration, no hard-bind on mid-run flaps).
• BindsTo/PartOf so slam→nav follow bringup; record/health are non-fatal to the drive stack.
• journald-mote.conf bounds the persistent journal (500M/2wk) so restart loops can't fill the SD card. Installed by systemd/install.sh — still one pixi run setup.

Health monitor (health_monitor.py → mote-health.service, pixi run health)

• Per-subsystem liveness/rate checks: scan, filtered scan, joint states, camera, odom TF, localisation TF — folding in system_monitor's host status and the last self-check verdict.
• Publishes diagnostic_msgs/DiagnosticArray on /diagnostics_agg (fleet-consumable) and a single OK/DEGRADED/FAULT line on /health (std_msgs/String, easy ros2 topic echo). Criticality drives the roll-up: stale critical subsystem → FAULT, stale non-critical/slow → DEGRADED. Config in config/health.yaml (~/.mote override).
• Type=notify + WatchdogSec=15 watchdog, fed by sd_notify.py (dependency-free $NOTIFY_SOCKET client, no-ops outside systemd) — a hung monitor is restarted.

Startup self-check (self_check.py, pixi run self-check)

• Runs as mote-bringup's ExecStartPre gate: servo bus ping (new servo_ping C++ tool, IDs from robot.yaml), lidar/camera device presence + openability, disk space, clock sanity (RTC-less Pi at epoch), config/site load.
• A failed CRITICAL check exits non-zero → blocks the launch (safe idle, clear reason) → systemd retries with backoff so replugging the lidar recovers on its own. Writes $MOTE_HOME/self_check_status.yaml, surfaced by the health monitor.

Per-node recovery — respawn=True on the drivers (mote_launch.py) and nav2 servers (nav2_launch.py; lifecycle managers get attempt_respawn_reconnection), so a single crashed node is relaunched inside the whole-service systemd restart.

Chaos validation (mote_bringup/test/chaos/)

• chaos_policy_demo.sh — hardware-free, proves the systemd restart policy locally via a transient --user unit. Committed chaos_log.txt shows a SIGKILLed service back with a new PID in ~2s.
• chaos_restart.sh (pixi run chaos) — kills ros2_control_node/sllidar_node/controller_server on the robot and verifies respawn within 30s. Safe by construction (matches executable names, excludes own PID).

Acceptance criteria

• ✅ Killing ros2_control_node / lidar / nav2 nodes → automatic recovery within a bounded time: respawn (nodes) + systemd restart (service), both scripted; local systemd-layer recovery evidenced in chaos_log.txt.
• ✅ /health shows OK/DEGRADED/FAULT transitions (verified: monitor publishes a rolled-up FAULT with all critical subsystems stale).
• ✅ Self-check failure (lidar unplugged) blocks readiness with a clear diagnostic (verified locally: exit 1, "NOT READY — mission blocked").
• ✅ Setup stays one command; docs updated (mote_bringup/README.md, CLAUDE.md, tools README).
• ✅ Battery-sensing gap recorded → follow-up task 187.

Needs bench with Michael (hardware-gated, not runnable from CI)

• chaos_restart.sh on auldbot (per-node respawn end-to-end) — capture its chaos_log.txt.
• /health OK↔DEGRADED↔FAULT transition by physically unplugging a sensor.
• Real servo_ping against the drive bus (validated as a clean build only; no hardware here).
• One pixi run setup on the Pi to install the new mote-health unit + journald drop-in.

Note: not pushed (per the no-remote-writes rule); voro pr will open the PR from this summary.