stackman

A menu-driven terminal operations console for the local bot-fleet stack — hermes-agent, gajae-code (gjc), and clawhip (plus the gjc-relay and gjc-dlq-watch support units). It gives an operator a calm, live view of the whole stack and safe, dependency-aware control over it: start, stop, restart, inspect status, and tail logs, per service or across the entire stack.

What it manages

The console is driven by a data registry (src/server_script/services.toml) that mirrors the real systemd topology on this host:

Service	Control	Health signal	Depends on
gjc-relay	gjc-relay.service	GET /healthz + port 127.0.0.1:25295	—
clawhip	clawhip.service	GET /health + port 127.0.0.1:25294	gjc-relay
hermes-gateway	hermes-gateway.service	gateway_state.json + cron ticker_heartbeat	—
gjc-dlq-watch	gjc-dlq-watch.service	dlq-watch.sh process	clawhip
gajae-code	managed child (no unit)	gjc mcp-serve coordinator process	hermes-gateway

gajae-code has no unit of its own — it runs on demand as the coordinator-MCP child of hermes-gateway, so the console observes it (status + logs) but does not start or stop it directly. Start/stop ordering for the whole stack is derived from the depends_on graph, matching the units' real After=/Wants= relationships (relay → clawhip → dlq-watch).

Requirements

• Python 3.14+ and uv
• systemd (the managed services are system-level units under /etc/systemd/system/)
• Passwordless sudo for systemctl (lifecycle actions run sudo -n systemctl …; status and log reads need no privileges)

Launch

uv run server-script

(uv run stackman is an alias for the same entry point.)

Keys

Key	Action
↑ / ↓	Move between services
s / x / r	Start / stop / restart the selected service
l	Open the live log viewer for the selected service
u / d / b	Stack up (start all) / down (stop all) / restart (restart all)
h	Show the keyboard legend (all keys)
f5	Refresh now (status also auto-refreshes)
q	Quit

Stop and restart actions (per-service and stack-wide) ask for confirmation first.

Architecture

The code is organised as small, single-purpose layers so services, health checks, and UI panels can be added or removed without rewriting control flow:

• models.py — immutable domain types (service specs, snapshots, probe/command results)
• config.py — loads and validates the TOML registry; computes dependency ordering
• command.py — safe async subprocess execution behind an injectable CommandRunner
• systemd.py — systemctl show status parsing and sudo -n systemctl lifecycle verbs
• health.py — concurrent TCP / HTTP / state-file / heartbeat / process probes
• logs.py — bounded recent logs and cancellable live tailing (journal or file)
• orchestrator.py — the shared control layer: snapshots and dependency-ordered stack operations with progress events (the UI and any future CLI call this)
• logging_setup.py — the manager's own rotating audit log (see below)
• theme.py, app.tcss, screens.py, app.py — the Textual console

A lifecycle action flows one way: the UI calls an orchestrator method → the orchestrator consults service metadata for the unit and ordering → it runs the systemd verb through the command runner → it waits for the unit to settle and re-probes health → progress events stream back to the operation log and the next status refresh repaints the board.

Logging

The manager keeps its own audit log, separate from the managed services' journals. Every run records startup and shutdown, and each lifecycle action (per-service and stack-wide) is logged with its outcome — including partial failures — so a restart that reported trouble can be reconstructed after the fact.

• Location: stackman.log under the platform state dir (~/.local/state/stackman/ on Linux).
• Rotation is built in: a RotatingFileHandler rolls the file at ~1 MB, keeping five older generations (stackman.log.1 … .5). The footprint stays bounded (~6 MB) with no external logrotate cron to configure.
• Records never reach the Textual screen (the package logger does not propagate to root), so the console stays quiet while the trail is still written.

A failed unit state after a stop is treated as a successful stop (the service is no longer running) and logged as a warning noting the unclean shutdown — it does not fail the surrounding stack restart.

Adding a service

Append a [[services]] block to services.toml with its unit, depends_on, health probes, and log source. No code changes are needed; ordering and the UI update automatically.

Development

uv run pytest              # unit + headless UI smoke tests
uvx basedpyright@latest    # strict type gate (recommended mode, zero warnings)
uv run ruff check          # lint
uv run ruff format         # format

prek.toml wires these as pre-commit hooks (run prek install to enable them). The [tool.basedpyright] table is intentionally minimal and guarded by tools/check_basedpyright_config.py — warnings are fixed in code, never silenced globally.

Operational assumptions

• The five services above are the current fleet; the runbook and architecture docs predate gjc-relay/gjc-dlq-watch, so this tool's registry (derived from the live unit files) is the authoritative source for them.
• Health endpoints (/health, /healthz) and loopback ports are those observed live on this host; if a service is reconfigured, update its probes in services.toml.
• Lifecycle control assumes the units keep their current names and that passwordless sudo systemctl remains available; missing privileges surface as an actionable hint rather than a crash.