feat: arcticwolfctl admin CLI — exports, audit, metrics, graceful shutdown (closes #25)

[amarok-60T]

Closes #25.

Adds a Unix-socket-backed admin server to the arcticwolf NFSv3 daemon, plus an arcticwolfctl client binary. End-to-end commands: status, version, log-level get/set, exports list/add/remove/update, config show, metrics. Includes structured audit logging, in-memory operational metrics, and graceful shutdown on SIGTERM/SIGINT.

No HTTP endpoint. No external metrics crate. No new runtime dependencies beyond arc-swap, vergen-gitcl, and tokio_util::sync::CancellationToken (already-included tokio-util 0.7).

Stack (9 commits)

#	Phase	Subject
1	Phase 1	admin: introduce Unix socket scaffolding with length-prefixed JSON
2	Phase 2	admin: implement status and version commands
3	Phase 2.a	refactor: drop dual lib/bin module compilation (cross-cutting)
4	Phase 3	admin: implement log-level get/set with tracing reload handle
5	Phase 4	fsal: refactor MultiExportFilesystem to ArcSwap snapshot
6	Phase 5	admin: wire exports list/add/remove/update and config show
7	Phase 6	admin: structured audit log with SO_PEERCRED capture
8	Phase 7	admin: metrics command with per-export and per-NFS-op counters
9	Phase 8	admin: graceful shutdown on SIGTERM/SIGINT

Each phase was independently reviewed under Opus 4.7 1M context; review findings (SHOULD FIX and NICE TO HAVE) were folded into the same commit via --fixup + --autosquash before progressing to the next phase. CI (Build / Test / Lint / NFS Integration Test) is green on every squashed SHA.

Key design decisions

• No HTTP surface. Metrics and audit are admin-command-only. Operators reach them via arcticwolfctl metrics and the JSONL audit file. This keeps the attack surface minimal and avoids the dependency overhead of an HTTP framework.
• In-memory runtime mutation only. exports add/remove/update mutate MultiExportFilesystem via an ArcSwap snapshot but do NOT write back to config.toml. v1 only allows mutating the read_only flag; structural changes (uid, name, backend, path) require config edits + restart.
• uid retirement. A removed export's uid is retired until daemon restart and cannot be reused, preventing stale handle/uid collisions across mutations. Names are reusable after removal.
• --dry-run on every mutating command. The dry-run path validates without taking the write lock or mutating state.
• Audit log in v1. Each accepted admin request emits one JSONL line with ts, peer (uid/gid/pid via SO_PEERCRED), command tag, verbatim request payload, result, error (if any), duration_ms. Sensitive-field redaction is deferred until the S3 backend lands (the local backend has no sensitive config fields).
• Metrics without prometheus. AtomicU64 counters with Ordering::Relaxed, exposed as a JSON snapshot via arcticwolfctl metrics. Covers rpc_requests_total / rpc_decode_errors_total / rpc_errors_total, all 22 NFSv3 procedure counters, per-export requests/bytes_read/bytes_written/errors, and per-admin-command counters via a pre-populated allowlist (to prevent client-supplied tags from growing the map without bound).
• Graceful shutdown. SIGTERM and SIGINT trigger a CancellationToken that stops the four accept loops (portmap, mount, NFS, admin). In-flight per-connection handlers are tracked in JoinSet and awaited. The whole drain is bounded by [shutdown] drain_timeout_seconds (default 30); on timeout, a warn is emitted and the daemon exits 0. A second signal during drain escalates to process::exit(128 + signum). Audit channel is flushed and admin socket file is unlinked before exit.
• One commit per phase. Each phase's review fixes were folded back into the phase commit via --fixup + git rebase -i --autosquash. The history is reviewable phase-by-phase, not fix-by-fix.

Configuration

New sections in config.toml:

[admin]
enabled = true
socket_path = "/var/run/arcticwolf/admin.sock"

[audit]
enabled = false                        # default off
path = "/var/log/arcticwolf/admin-audit.jsonl"

[shutdown]
drain_timeout_seconds = 30             # ceiling, not target; exit 0 on overrun

Config::validate() rejects audit.enabled = true without a non-empty path. See arcticwolf.example.toml for the full annotated example.

Test coverage

• 230+ library unit tests, including:
  • ArcSwap snapshot semantics (counters survive update_export, reset on remove_export)
  • per-NFS-op counter dispatch (22 procedures)
  • audit JSONL round-trip + RFC 3339 formatter (boundary-checked through year 9999 and Gregorian leap rules)
  • graceful shutdown signal handler mapping (130 / 143)
  • audit Mutex<Option<Sender>> shutdown idempotency
• Integration tests for status, version, log-level, exports, config-show, metrics, audit, graceful-shutdown — all driving the real Unix-socket wire format.
• CLI exit-code matrix for every subcommand (success / daemon-Err / unreachable-daemon).
• NFS Integration Test (make nfstest) green on every squashed SHA — verifies multi-export mount, EROFS enforcement, MNT NOENT, NFS POSIX against /data.

Known limitations / follow-up

• Idle per-connection handlers block drain until the timeout ceiling. The per-connection read loops in RpcServer::serve and admin::serve_with_shutdown do not check the CancellationToken between requests; an idle client (TCP keepalive on the NFS port, or an arcticwolfctl that connected and never sent anything) keeps its handler task alive until the drain timeout fires. Tracked separately — see follow-up issue.
• Sensitive-field redaction in audit payload deferred to the S3 backend follow-up.
• Per-export historical metrics — removing an export drops its counters. Persistent historical metrics out of scope for v1.
• macOS-only compile_error! in src/lib.rs is unchanged; this PR is Linux-only in practice (matches existing repo policy).

DCO

All commits carry Signed-off-by: trailers.

[gemini-code-assist]

Important

Installation incomplete: to start using Gemini Code Assist, please ask the organization owner(s) to visit the Gemini Code Assist Admin Console and sign the Terms of Services.

[amarok-60T]

Architecture overview as flow charts, in case it helps a cold reviewer orient.

1. Request flow — arcticwolfctl to daemon

flowchart LR
    subgraph client["Client"]
        ctl["arcticwolfctl<br/>(clap CLI)"]
    end

    subgraph wire["Wire"]
        uds["Unix socket<br/>length-prefixed JSON"]
    end

    subgraph daemon["arcticwolf daemon"]
        accept["admin accept loop<br/>SO_PEERCRED on accept"]
        dispatch["Dispatcher<br/>route by command tag"]

        subgraph handlers["Handlers"]
            h1["status<br/>version"]
            h2["log-level<br/>get / set"]
            h3["exports<br/>list / add / remove / update"]
            h4["config-show"]
            h5["metrics"]
        end

        subgraph state["Shared state (Arc)"]
            mef["MultiExportFilesystem<br/>ArcSwap&lt;ExportsSnapshot&gt;<br/>+ retired_uids"]
            cfg["Config<br/>(startup snapshot)"]
            metrics_obj["Metrics<br/>AtomicU64 counters<br/>+ pre-populated by_command"]
            reload["LogReloadHandle"]
            audit["AuditWriter<br/>mpsc to BufWriter"]
        end

        afile[("admin-audit.jsonl")]
        rpc["Portmap / Mount / NFS<br/>(RpcServer::serve)"]
    end

    ctl <--> uds
    uds <--> accept
    accept --> dispatch
    dispatch --> h1 & h2 & h3 & h4 & h5
    h1 -.read.-> cfg
    h2 -->|reload| reload
    h3 -->|add / remove / update + list| mef
    h4 -.read.-> cfg
    h5 -.read.-> metrics_obj

    accept -->|record event| audit
    audit -->|writer task| afile

    accept -->|bump counter| metrics_obj
    rpc -->|bump per-op + bytes| metrics_obj
    rpc -->|fs ops| mef

Loading

Notes:

• MultiExportFilesystem mutating methods (add_export / remove_export / update_export) are serialized by a Mutex<()> write lock; readers (every NFS trait method) are lock-free via ArcSwap.
• Per-export Arc<ExportMetrics> survives update_export snapshot rebuilds (refcount bump on the cloned ExportEntry), and resets on remove_export + re-add.
• config-show returns the startup config snapshot, not the live mutated state — the difference matters once runtime mutations land.
• AuditWriter::record is fire-and-forget; the writer task drains the mpsc into the file with per-line flush().

2. Graceful shutdown — Phase 8

flowchart TB
    sig["SIGTERM / SIGINT<br/>(handler installed right after tracing init)"]

    sig -->|1st signal| token["CancellationToken.cancel()"]
    token -->|notify| servers["Four accept loops<br/>stop accepting new connections"]
    servers --> drain["Drain per-server JoinSet<br/>(in-flight requests finish)"]
    drain --> wrap{"tokio::time::timeout<br/>(drain_timeout_seconds)"}
    wrap -->|complete| flush["audit.shutdown()<br/>drain mpsc to file"]
    wrap -->|ceiling| warn["warn: drain timeout<br/>(in_flight=N)"]
    warn --> flush
    flush --> unlink["unlink admin socket"]
    unlink --> exit0["exit 0"]

    sig -.->|2nd signal during drain| escalate["process::exit(128 + signum)<br/>SIGINT to 130, SIGTERM to 143"]

Loading

Notes:

• The Mutex<Option<Sender>> pattern inside FileAuditWriter lets shutdown() close the audit channel regardless of how many Arc<dyn AuditWriter> clones are still alive.
• Drain timeout is a ceiling, not a target — exit status remains 0 if the ceiling fires.
• The admin socket file is unlinked via a RAII SocketCleanup guard at the end of serve_with_shutdown, plus a best-effort fallback in main.rs for the timeout path.
• See follow-up issue admin: idle per-connection handlers block graceful drain until ceiling #30 for the known limitation that idle per-connection handlers block drain until the ceiling.