EPIC: one service account for every taOS-owned service (taos), finishing the /opt/taos move first

[jaylfc]

Problem

taOS-owned services currently run under three different accounts, and nothing enforces a policy:

service	runs as	install / data
tinyagentos (controller)	taos	/opt/taos
taosmd serve (memory + A2A bus)	jay	/home/jay/taosmd, /home/jay/.taosmd
qmd	jay	-
rkllama	root	/root/rkllama

This is not cosmetic. It produced a real production defect on 2026-07-20: the Library writes its collections handoff directory as taos under /opt/taos/data (mode 700), while taosmd indexes as jay, so taosmd could not traverse to the directory at all and every create failed with a PermissionError before any allowed_roots check. Patched with group traversal plus a setgid shared dir, which works but is a workaround for the account split, not a fix. The same class already bit us once before with the agent-commons mirror (relocated to /srv because taos cannot traverse /home/jay).

Root cause: cross-service file handoffs are only sane when producer and consumer share an account (or an explicit, documented group contract). We have neither by policy, only by accident.

Goal

Every taOS-owned service runs as taos, with installs and data under taos-owned paths, so file handoffs between services need no group gymnastics and deployment is repeatable rather than hand-tuned per service.

Hard prerequisite

Finish the /opt/taos move first (#156 is half-done). getent passwd taos still reports home /opt/tinyagentos. Relocating any service into taos-owned space before that is resolved bakes the stale path in deeper.

Sequencing (each step independently verifiable)

1. Finish Cross-node conversation summarization: idle CPU nodes compress old context in the background #156: taos home and any remaining hardcoded /opt/tinyagentos paths move to /opt/taos.
2. taosmd (highest value, ties to the Library handoff): relocate install + data dir + the 7.9 GB ONNX model tree out of /home/jay into taos-owned space; run taosmd.service as taos; set TAOSMD_ONNX_PATH explicitly in the service environment. Non-negotiable, see risk below. Revert the interim group-traversal patch (jay in the taos group, g+x on /opt/taos/data) once it is redundant.
3. qmd: same treatment, currently jay.
4. rkllama: currently root, which is the worst of the three. Needs its own assessment (NPU device node permissions may be why it is root - confirm before demoting).
5. Policy + docs: a short service-account contract in the repo, and every new managed backend registers under it by default. This is what makes the backend-service-management slice repeatable instead of per-service bespoke.

Risk that must be respected at step 2

Moving the ONNX model path is the exact trigger for taosmd store-mode poisoning: if the model does not resolve at the moment of the first post-move request, the vector store is permanently stamped embedder=qmd:qmd and serve raises on every later boot. Procedure: cold backup of the data dir, explicit TAOSMD_ONNX_PATH, then verify store_meta reads embedder=minilm-onnx after the first real request (recovery is deleting the marker row, but only clean if nothing was ingested in the poisoned window).

Related

• Cross-node conversation summarization: idle CPU nodes compress old context in the background #156 /opt/taos move (prerequisite, half-done)
• Backend service management (taosmd registration as CORE_MANAGED is the natural vehicle for step 2)
• feat(library): P1 core — LibraryStore, ingest pipeline, processors, collections handoff, app UI #2062 Library collections handoff (the defect that surfaced this)

[jaylfc]

Relocation procedure, from the taosmd side (@taOSmd-dev input)

Three additions that change how step 2 must be executed. The first is a genuine data-loss trap and outranks the model-path risk already noted.

1. The SQLite WAL is the biggest risk in the move, bigger than the ONNX path. The live data dir currently has archive-index.db-wal at 4.1 MB plus a -shm alongside it. That WAL holds committed data that is not in the .db file. A move that copies *.db and drops the -wal/-shm silently loses recent archive writes, and the archive is the zero-loss source of truth. Rules: stop serve first so a clean shutdown checkpoints the WAL, verify no -wal/-shm remain (or move all three files together per database), and never rsync a .db while serve is running. Applies to archive-index.db, vector-memory.db and knowledge-graph.db alike.

2. Strict ordering, because the wrong order re-poisons the store:

stop serve
move/copy the data dir (WAL rules above)
chown -R taos:taos the data dir AND the model tree
set TAOSMD_ONNX_PATH to the NEW model location in the unit env
start serve
make ONE real request
re-check store_meta   # mandatory - relocation is a SECOND poisoning window

If serve starts before the chown completes it creates root or jay owned files inside a taos-owned tree, which is worse than either starting state. The store_meta re-check is not optional: surviving the first poisoning window does not immunise the move.

3. The 7.9 GB model tree should NOT move with the data. It is read-only at runtime, so a shared taos-readable location (/opt/taos/models or /usr/local/share/taosmd/models) is cleaner than dragging it into the data dir, and it means a future data-dir migration never has to shift 8 GB again. Pin TAOSMD_ONNX_PATH at it explicitly; never rely on the implicit ~/taosmd fallback, which is exactly what would have bricked the deploy on 2026-07-20.

Separate risk raised in the same review (not this issue, but tracked here so it is not lost)

taosmd has no migration framework at all - no user_version, no version stamp, just create-if-missing plus three hand-rolled ALTER guards. That was fine for the July jump because every change was additive, but the first column ever added to collections.db will silently break every existing collections.db in the field. Internal-only problem today; it stops being one the moment taOS ships taosmd to users. Worth its own piece of work before the install base grows.

[jaylfc]

Relocation findings from the taosmd side (round 2, code-audited not from memory)

@taOSmd-dev had the codebase audited for home-dir, CWD and outside-data-dir assumptions. Several of these change the plan, and two are silent-failure classes we would not have caught in testing.

1. The data-dir default is a silent data-loss footgun. Resolution order is explicit arg, then TAOSMD_DATA_DIR, then expanduser("~/.taosmd"). A service account with a different or missing HOME resolves a different path, and api.py then does mkdir(parents=True, exist_ok=True) and initialises a brand new empty store. No existence check, no warning: the service comes up healthy and every memory appears gone. Mitigation: always pass --serve-data-dir explicitly AND set TAOSMD_DATA_DIR (belt and braces, and the explicit arg is visible in ps). Relevant to systemd: ProtectHome=yes or DynamicUser= leave HOME unset or /, so the defaults resolve to /.taosmd.

2. The cross-encoder reranker is a worse path hazard than the ONNX one. cross_encoder.py defaults onnx_path to the CWD-relative models/cross-encoder-onnx with no env override, and api.py constructs it with no argument. Worse, recipes.ensure_reranker_model() will snapshot_download roughly 600 MB from HuggingFace into that CWD-relative path, triggered at runtime by a live search request, whenever the active recipe wants bge-v2-m3 and the model is absent. Under a system unit with no WorkingDirectory, CWD is /, so that becomes /models/. Mitigation: set WorkingDirectory explicitly, pre-seed the reranker model there, and set HF_HOME + XDG_CACHE_HOME to writable paths.

3. TAOSMD_ONNX_PATH is MiniLM-specific, not a general model pointer. It is silently ignored for arctic-embed-s or any other embedder. The general answers are <data_dir>/models/<name> or TAOSMD_DIR.

4. Act on this before any copy: the current config contains server_url. /home/jay/.taosmd/config.json on the Pi has it, and http_server always passes an explicit data_dir so the config-file value is honoured even where the env var is ignored. Copying that file into a new data dir makes the new server proxy every request to the old host or to itself. Strip server_url, server_token, and the stale collections.allowed_roots from any migrated config.

5. Good news on the risk we were both bracing for: the store_meta marker is deliberately path-independent - it reduces the model path to its leaf directory name. Moving models/minilm-onnx under a new prefix will NOT trip StoreModeMismatch. It only breaks if the leaf directory is renamed. Relocation is safer than either of us assumed, provided the leaf stays minilm-onnx and the model resolves at all.

6. Layout decision (theirs, as recovery owner): data SEPARATE from install. Install at /opt/taos/taosmd, data at /opt/taos/data/taosmd or another tree entirely. Every store path is built from Path(data_dir)/..., nothing writes to the data dir's parent, and store_meta is path-independent, so the store travels cleanly. Separate trees mean the backup is the memory and only the memory.

7. Model tree: shared root, but pick exactly ONE home. _resolve_onnx_path checks <data_dir>/models/ before /models/, so a stale copy under the data dir silently wins and you would be serving a different model than you think. Do not leave copies in both.

8. Load-bearing unit environment:

User/Group=taos
WorkingDirectory=/opt/taos/taosmd          # REQUIRED, several paths are CWD-relative
Environment=HOME=/opt/taos                 # give the account a real writable home
Environment=TAOSMD_DATA_DIR=/opt/taos/data/taosmd
Environment=TAOSMD_DIR=/opt/taos/taosmd
Environment=TAOSMD_ONNX_PATH=<shared model root>/minilm-onnx
Environment=HF_HOME=/opt/taos/cache/huggingface
Environment=XDG_CACHE_HOME=/opt/taos/cache
ExecStart=... serve --serve-data-dir /opt/taos/data/taosmd

Note their own taosmd serve --install-service is unusable here: it writes systemd user units only, with no User= and no Environment=. Hand-write the unit.

9. Post-migration verification (catches both silent failures): memory count must match the pre-migration baseline (proves we did not open a fresh empty data dir), plus the store_meta check already in the plan.

This supersedes nothing in the WAL procedure above; it adds to it. Combined ordering for step 2 is now: stop serve, WAL-safe copy, chown, set the full env block, start, one real request, verify store_meta AND memory count against baseline.

[jaylfc]

Correction on item 4 (the server_url proxy hazard), verified rather than assumed: our live config does not have it. /home/jay/.taosmd/config.json on the Pi contains exactly admin_token and collections - I wrote it fresh during the 2026-07-20 deploy rather than inheriting the old one, and the pre-deploy config was an empty object (2 bytes). The backup tarball confirms the same: empty, neither server_url nor server_token present.

So nothing was stripped and nothing changed. The hazard is real as a class and stays in the checklist above, because the next person copying a config forward will not know ours happened to be clean - but it is not an outstanding action for this migration.

[jaylfc]

Additional check: DEV boxes, not just the deploy target

@taOSmd-dev found this on their own machine while verifying the server_url item above, and it is worse than the class-level hazard we were describing.

Their dev box config had server_url as its only key, pointing at the Pi on port 7900. So every taosmd CLI command run "locally" on that box has been acting on the live production store, not a local one. The trap is not a future copied config: it is an agent or developer believing they have a sandbox when every command is hitting production.

Adding to the migration checklist as a standing check, applied to every machine that can run the taosmd CLI rather than only the deploy target:

• On each dev/admin box, inspect ~/.taosmd/config.json for server_url. If present, confirm deliberately whether it should point at production.
• Prefer an explicit local instance for development, and treat pointing a dev CLI at production as a conscious act rather than a default.
• After the relocation, re-check every box again: a config that pointed at the old host is now either stale or, worse, still live.

They are repointing theirs at a local instance. Nothing to do on our side (our Pi config is clean, verified twice), but the check belongs in the procedure.