[FEATURE] Add admin CLI

[Vicente-Cheng]

Background

Arctic Wolf currently has no admin interface — operators can't inspect runtime state, change settings without restarting, or coordinate shutdown without a hard signal. This issue tracks the v1 admin CLI: a daemon-side Unix domain socket server, a separate arcticwolfctl client binary, basic observability, and runtime mutation of the export list that landed in #26.

The multi-export work (#26 / PR #27) is the prerequisite — admin operations on exports only make sense once the FSAL supports more than one. With that merged, the admin CLI can build on top of ExportRegistry/NfsBackend cleanly.

Goals

1. Runtime export mutation — add, remove, and toggle read-only on exports without restarting the daemon. In-memory only in v1.
2. Runtime configuration knobs — log level adjustment, status inspection, graceful shutdown.
3. Metrics — counters and histograms exposed through the admin protocol for operators and (later) scrapers.
4. Audit trail — every admin write recorded in a JSONL audit log so operators can answer "who changed what, when?"

Architecture summary

• New arcticwolfctl binary (separate [[bin]] from arcticwolf daemon), follows the systemctl / kubectl pattern.
• Daemon listens on a Unix domain socket (default /run/arcticwolf/admin.sock, mode 0600). No TCP transport in v1 — cross-pod admin in Kubernetes is expected to go via kubectl exec.
• Wire protocol: length-prefixed JSON. Each message is one AdminRequest → one AdminResponse. Reasoning: framing is explicit; payloads containing newlines (log messages, multi-line errors) don't require escaping; tokio_util::codec::LengthDelimitedCodec is ready-made. Debug ergonomics covered by an arcticwolfctl raw subcommand.
• The daemon's tokio::select! gains an admin task alongside the three RPC servers; a CancellationToken is shared so SIGTERM/SIGINT and the admin shutdown command all flow through the same path.
• MultiExportFilesystem is refactored from a build-once HashMap to an arc-swap::ArcSwap<ExportsSnapshot>. Read path stays wait-free (every NFS RPC reads); admin write rebuilds a small snapshot (typically <10 exports) and atomically swaps. This avoids the cross-await lock-guard hazards of RwLock.

Runtime export semantics

• All export mutation is in-memory only. The config file is never rewritten by the admin CLI in v1. After daemon restart the in-memory state reverts to the file.
• exports update accepts only --read-only=<bool> in v1. Changing name, uid, backend, or path is rejected (use exports remove + exports add to migrate, accepting the file-handle stale cost).
• When an export is removed, its uid is moved into a retired_uids: HashSet<u32> and cannot be reused by a subsequent exports add until the daemon restarts. This prevents in-flight client handles whose first 4 bytes still encode that uid from being silently re-routed to a new, unrelated export. The set is cleared on startup since the configuration file is re-read fresh.
• After exports remove, clients holding handles for that export will see NFS3ERR_STALE on their next operation — this is the existing behavior of MultiExportFilesystem::entry_for_handle plus classify_handle_error and requires no new code.

Scope

In scope (v1)

Daemon-side

• Unix domain socket admin server with length-prefixed JSON framing.
• New [admin] config section: enabled (bool, default false), socket_path, socket_mode.
• MultiExportFilesystem migration to ArcSwap<ExportsSnapshot> with retired_uids.
• tracing_subscriber::reload::Layer so log level can be changed without restart.
• CancellationToken-based graceful shutdown that drains in-flight RPCs.
• SIGTERM and SIGINT trigger the same graceful path.
• Metrics registry (metrics + metrics-exporter-prometheus) instrumented in the NFS dispatcher and FSAL: per-procedure RPC counters and latency histograms, per-export bytes counters, error counts by nfsstat3, active connection gauges by service.
• JSONL audit log (default /var/log/arcticwolf/audit.jsonl, configurable). Each admin write captures peer pid/uid via SO_PEERCRED, command, args, result, and duration. Append-only; rotation is left to the system (e.g. logrotate).
• [audit] config section (path; enabled flag).

Client-side (arcticwolfctl)

• status — uptime, bind address, actual listening ports, daemon version, current log level, number of exports.
• version — client and daemon versions (build commit / Rust version via vergen).
• log-level get / log-level set <level> — adjust global tracing filter at runtime.
• config show — emit the live in-memory Config as JSON. No masking in v1 (no sensitive fields exist; revisit when S3 backend lands).
• exports list — list configured exports (name, uid, backend, path, read_only).
• exports add --name <path> --uid <u32> --backend local --path <path> [--read-only] [--dry-run] — add a new export. --dry-run validates the candidate (path resolution, uid uniqueness, name uniqueness, no collision with retired_uids) without committing.
• exports remove <name|uid> — remove an export; its uid is retired until restart.
• exports update <name|uid> --read-only=<bool> [--dry-run] — toggle the read-only flag.
• metrics [--format json|prometheus] — render the metrics registry; JSON by default.
• shutdown [--timeout=<seconds>] — request graceful shutdown.
• raw '<json>' — send a raw admin request; debug aid.
• Global flag --socket <path> to override the default socket location.

Out of scope (tracked separately)

• HTTP server (/metrics, /healthz, /readyz) — defer; Prometheus scraping is expected to use kubectl exec arcticwolfctl metrics --format prometheus in v1.
• Configuration persistence — admin writes are never reflected back to the config file. Operators who want changes to survive restart edit the file directly.
• exports update --name/--uid/--backend/--path — requires extra handle-rewrite or migration semantics; revisit only if a concrete need surfaces.
• config show masking for sensitive fields — implement when the first backend with secrets (S3, Ceph) is introduced.
• Per-module tracing filter (debug enable-tracing --filter <env_filter>) — v1 only supports global level.
• Mount-table tracking and clients list — requires non-trivial state; large enough for its own issue.
• TCP / mTLS admin transport — Unix socket plus kubectl exec covers v1 deployment.
• Role-based admin authorization — single-tier privilege in v1 (whoever can open() the socket is admin).
• config reload — explicit per-resource commands cover the v1 mutation surface; "reload from file" introduces ambiguity (file vs in-memory) that we don't yet need.
• open_handles{export=...} metric — requires a persistent handle table; ties into [TASK] Persistent file handle for LocalFilesystem #10 (if applicable) or a future tracking system.

Acceptance criteria

• arcticwolfctl binary builds from cargo build --release --bin arcticwolfctl and is included in the release artifact path; symlink/install instructions are documented in README.md.
• Daemon refuses to start with [admin] enabled = true if socket_path's parent directory does not exist or is not writable; error is explicit.
• When [admin] enabled = false (the default), no admin socket is created and admin code paths never run; make test still passes.
• arcticwolfctl status returns daemon_version, uptime_seconds, bind_address, the actual nfs_port/mount_port, log_level, and export_count.
• arcticwolfctl version shows the client version, the daemon's reported version, and includes a build_commit field derived from vergen.
• arcticwolfctl log-level set debug changes the live filter; a subsequent NFS request logs at debug level without a daemon restart. log-level set tomato returns an error and the filter is unchanged.
• arcticwolfctl config show returns valid JSON that reflects the in-memory config (including any runtime export mutations applied so far in the current process).
• arcticwolfctl exports list lists every configured export with name, uid, backend, path, and read_only.
• arcticwolfctl exports add --name /new --uid 9 --backend local --path /srv/new makes the export visible to subsequent arcticwolfctl exports list calls and to mount -t nfs server:/new from a Linux client.
• arcticwolfctl exports add --uid 9 ... is rejected when uid 9 is currently in use, when 9 is in retired_uids, or when the candidate fails any of the existing Config::validate() invariants (e.g. relative path, name not starting with /).
• arcticwolfctl exports add --dry-run ... validates but does not commit; the daemon's state is unchanged and exports list does not include the candidate.
• arcticwolfctl exports remove <name> removes the export; a client holding a handle for that export receives NFS3ERR_STALE on its next request.
• After exports remove /old --uid 7, a subsequent exports add --uid 7 ... is rejected with a "uid retired this run" error until the daemon restarts.
• arcticwolfctl exports update /data --read-only=true flips the export to read-only; subsequent WRITE requests against that export return NFS3ERR_ROFS.
• arcticwolfctl exports update /data --name=/renamed is rejected in v1 with a clear "field not mutable in v1" message.
• arcticwolfctl metrics returns JSON containing at minimum arcticwolf_nfs_rpc_total{proc=...}, arcticwolf_nfs_errors_total{proc=...,nfsstat=...}, arcticwolf_nfs_bytes_total{export=...,direction=...}, and arcticwolf_rpc_connections_active{service=...}. --format prometheus returns the same data in Prometheus exposition format.
• arcticwolfctl shutdown returns success when the daemon process exits cleanly within the timeout; in-flight RPCs are allowed to complete.
• SIGTERM and SIGINT to the daemon trigger the same graceful path as arcticwolfctl shutdown.
• Every admin command that mutates state (exports add/remove/update, log-level set, shutdown) writes one JSONL record to the audit log containing ts, peer_pid, peer_uid, command, args, result, and duration_ms. Read-only commands (status, version, exports list, config show, metrics) do not write audit records.
• If the audit log path cannot be written (disk full, perms), the admin command still completes; the daemon logs a single warning. The admin response includes audit_warning: "..." in that case.
• arcticwolfctl --socket /tmp/test.sock status connects to a non-default socket location.
• arcticwolfctl raw '{"command": "status"}' succeeds and prints the raw JSON response.
• make test passes. make nfstest passes including new admin-vs-NFS interaction tests (notably: exports remove followed by an in-flight client returns NFS3ERR_STALE; exports update --read-only=true is observed by an NFS client as EROFS).
• README.md documents how to enable the admin server, the available arcticwolfctl commands, and the Kubernetes kubectl exec workflow.

Implementation phases

Each phase is one squashed commit. make build, make test, make lint must pass on every phase; make nfstest must pass on the phases marked NFS below.

1. Admin transport scaffolding. New src/admin/ module: protocol.rs (length-prefixed JSON codec with tokio_util::codec::LengthDelimitedCodec), request.rs/response.rs enums, server.rs accepting connections and routing to a not_implemented dispatcher, context.rs carrying shared state. New [admin] config section with enabled = false by default. main.rs spawns the admin task into tokio::select!. Adds tokio-util and nix (for SO_PEERCRED) to Cargo.toml. Tests: codec round-trip; unknown command error shape.

2. arcticwolfctl binary + status + version. New src/bin/arcticwolfctl.rs with a clap-driven subcommand tree. Implements status and version commands end-to-end. Adds vergen as a build dep for build_commit/rustc info. Tests: spawn daemon + ctl in tokio::test; assert JSON shape.

3. Log level reload (log-level get/set). Migrates tracing_subscriber::fmt().init() to EnvFilter + reload::Layer. Reload handle stored on AdminContext. Tests: set → get round-trip; invalid level string returns an error.

4. MultiExportFilesystem arc-swap refactor. Switches exports/name_index storage to ArcSwap<ExportsSnapshot> where the snapshot also carries retired_uids. Adds inherent methods add_export(), remove_export(), update_export() on MultiExportFilesystem. All trait methods (Filesystem and ExportRegistry) clone the inner Arc<LocalFilesystem> out of the snapshot before any .await, never holding a guard across an await. This phase does not add any admin command; behavior to NFS clients is identical to the current state. Tests: existing multi_export.rs tests must pass; new tests exercise add_export / remove_export / update_export directly, including uid retirement. NFS — full nfstest required.

5. Admin commands: config show + exports list/add/remove/update + --dry-run. Wires the new inherent methods to admin commands. Adds Serialize impls on Config/ExportConfig/BackendConfig for config show. Tests: end-to-end exports add → list → remove → list; dry-run does not commit; retired uid rejection; exports update --read-only observable to a real Linux NFS client as EROFS. NFS — full nfstest required.

6. Audit log. New src/admin/audit.rs with append-only JSONL writer. SO_PEERCRED extraction via nix::sys::socket. New [audit] config section. Audit write failures emit a warning but do not block admin operations. Tests: every mutating command produces exactly one record; read-only commands produce zero; audit write failure path returns audit_warning in the response.

7. Metrics. Adds metrics and metrics-exporter-prometheus to Cargo.toml. Installs a PrometheusBuilder recorder at daemon startup. Instruments the NFS dispatcher (counter!/histogram! per-procedure), read/write handlers (per-export bytes), the RPC server (active connections gauge), and the dispatcher's error path (errors by nfsstat3). Implements arcticwolfctl metrics [--format json|prometheus]. Tests: counters/histograms increment as expected over a small NFS workload; both output formats parse.

8. Graceful shutdown + signal handling. Introduces tokio_util::sync::CancellationToken. Each RpcServer::serve_until(cancel) selects on accept() vs cancel.cancelled(). Active connections drain via a JoinSet before the task returns. tokio::signal::unix listens for SIGTERM and SIGINT. Admin shutdown command and signals both trigger the same path. Tests: admin shutdown, SIGTERM, SIGINT all cause the process to exit cleanly within the timeout; an in-flight large WRITE completes successfully. NFS — full nfstest required.

References

• Multi-export prerequisite: [FEATURE] Support multiple NFS exports #26 (closed) / PR config: introduce per-export configuration #27 (merged).
• MultiExportFilesystem trait split context: src/fsal/multi_export.rs.
• Cross-cutting refactor that may interact: [REFACTOR] FSAL typed error enum to replace substring-based NFS error mapping #28 (FSAL typed error enum — independent, no blocker).
• classify_handle_error already maps stale-uid lookups to NFS3ERR_STALE: src/nfs/error.rs.
• Existing Config::validate() invariants the new candidate must pass: src/config.rs.