Skip to content

Commit 281bf9e

Browse files
authored
release: v0.8.0 — daemon cross-agent capture + ingest correctness
* feat: recall quality & project inspection (issue #56) - --whole flag for `smriti ingest file`: stores .md as single message (no paragraph splitting); warns without flag - `smriti projects <id>`: rich inspection report — sessions, messages, agents, tags, decisions, recent sessions - `smriti tags`: global or --project-scoped tag usage counts; --available mirrors category tree - `smriti status --project <id>`: scopes all stats (agents, categories) to a single project - 29 new tests in test/recall.test.ts covering all retrieval paths (full-doc, tags, project reports, multi-filter) * refactor: move memory.ts + ollama.ts out of QMD submodule QMD submodule is now a clean upstream fork (d58fedf, v2.1.0+) — no Smriti-specific code lives there. Future upstream syncs are conflict-free. - src/memory.ts: moved from qmd/src/memory.ts; imports updated to ../qmd/src/store.js and ../qmd/src/llm.js; uses QMD's Database type - src/ollama.ts: moved from qmd/src/ollama.ts; self-contained, no changes - src/qmd.ts: re-exports now come from ./memory and ./ollama - qmd submodule: bumped to d58fedf (upstream v2.1.0+34 commits of fixes) Upstream picks up: security dep bumps, db-transaction-type fix, embedding overflow hardening, sqlite-vec actionable errors, GGUF magic error fix, Windows home fallback, status device probe opt-in, and more. * fix(ci): add picomatch@4 as explicit root dep after QMD upstream sync * feat: knowledge density scoring (#62) and smriti digest (#63) Issue #62 — Knowledge Density Scoring: - Add density_score REAL column to smriti_session_meta (migration) - computeDensityScore(): composite 0-1 score from tool calls (25%), file writes (25%), git ops (20%), decision tags (15%), errors (10%), token volume (5%) - Hook into storeSession() so every ingest auto-computes and persists score - Blend density into recallMemories() ranking: final = score*0.8 + density*0.2 - smriti enrich --density: backfill scores for all existing sessions - smriti show <id> extension: --density flag shows bar-chart breakdown Issue #63 — smriti digest: - New src/digest.ts: generateDigest() aggregates sidecar signals for a time window, groups by project, surfaces top tools/errors/costs - formatDigest() and formatDensityBreakdown() added to format.ts - smriti digest [--days N] [--project id] [--synthesize] [--model name] generates work summary; --synthesize calls Ollama for narrative * refactor: SDK migration Phase 1+2 — createStore() init + kill llm.js import (#57) Phase 1: DB init via QMD SDK - New src/store.ts: cycle-free QMDStore singleton (setQmdStore/getQmdStore) - db.ts: remove initializeQmdStore() (duplicate of SDK's initializeDatabase) initSmriti() now calls SDK createStore() and is async - closeDb() delegates to closeQmdStore() Phase 2: Kill ../qmd/src/llm.js deep import in memory.ts - Replace getDefaultLlamaCpp() with getQmdStore().internal.llm - Replace insertEmbedding() call with getQmdStore().internal.insertEmbedding() - formatQueryForEmbedding/formatDocForEmbedding moved to ../qmd/src/store.js import (they are re-exported there; no longer touching llm.ts internals) Downstream: index.ts awaits initSmriti(); test/team.test.ts uses beforeAll for async init * feat: query expansion + reranking in recall pipeline (#58) Default-on quality mode in recallMemories(): - Calls store.internal.expandQuery() to generate lex/vec/hyde query variants - Runs FTS + vec search for each variant with 0.7 weight - Fuses all ranked lists via RRF (original queries at 1.0 weight) - Reranks top deduped candidates with store.internal.rerank() (60/40 blend) - --fast flag skips both steps for low-latency lookups Also fixes team-segmented.test.ts beforeAll to use async initSmriti(). * feat: smriti enrich --queries — retroactive query labeling (#60) Adds smriti_session_queries table + smriti_queries_fts virtual table. expandQuery() generates search aliases per session (lex/vec/hyde variants). searchFiltered() merges alias matches as 'query_alias' source results. storeSession() auto-enriches new ingests non-blocking (fire-and-forget). --dry-run shows generated aliases without writing; --project scopes batch. * feat: smriti ask — RAG question-answering command (#61) Multi-angle recall (expandQuery + rerank default-on) feeds top-N sessions to ollamaAsk() which returns a grounded answer with [N] citations. --no-synthesize returns ranked sources only; --json returns structured output. Graceful fallback to sources when Ollama is unavailable. * feat: --wide flag for cross-project knowledge routing (#64) smriti recall "query" --project X --wide searches all projects (bypassing project filter) and rerankss with intent "relevant to X project context" so the cross-encoder scores cross-project results against local needs. Results from other projects get project badge in output via session meta lookup. --wide without --project is equivalent to global unfiltered recall. * feat: smriti drift — temporal topic evolution command (#65) Recalls all sessions about a topic, sorts chronologically, and synthesizes an evolution narrative via Ollama showing decisions, reversals, refinements. --since <date> filters to recent history; --no-synthesize returns timeline only. --json returns structured timeline array. Graceful "not enough history" when < 2 sessions. * feat: --check-conflicts flag for contradiction detection in recall (#67) ollamaCheckConflicts() sends all top-N results in one batch to Ollama and parses CONFLICT [i] vs [j]: description responses. --check-conflicts on smriti recall flags contradictory pairs in output. --json includes conflicts array. No behavior change without the flag. SMRITI_CONFLICT_THRESHOLD env configures sensitivity (default 0.7). * feat: QMD SDK Migration Phase 4 — index sessions as QMD documents (#59) Dual-write on ingest: each session written to ~/.cache/smriti/sessions/<id>.md. initSmriti() registers smriti-sessions collection when dir exists. storeSession() writes markdown + fires background store.update(). recall() uses store.search() when smriti-sessions has docs; falls back to recallMemories() otherwise (backward compat). rerank=false when --fast. * feat: smriti clusters — semantic session clustering (#66) k-means over session embeddings, Ollama cluster naming, smriti clusters command, enrich --clusters, and recall --cluster <name> filter. * fix: sync tag roundtrip + team config.json with custom categories (#1, #2) #1: parseFrontmatter now parses tags arrays into string[]; syncTeamKnowledge restores all tags from meta.tags with isValidCategory guard; falls back to scalar meta.category for old exports. #2: new src/team/config.ts with readConfig/writeConfig/mergeCategories/ exportCustomCategories; share writes custom categories to config.json (v2); sync reads config.json and upserts categories before scanning files; SyncResult gains categoriesImported; smriti config show/add-category/ sync-categories CLI added. * chore(qmd): bump submodule to upstream main (ddbd6bd) Pulls 49 upstream commits via fast-forward merge. Key changes touching search behavior: - Fix hybrid RRF weighting by query type (#004714a) — expansion-derived lists no longer steal original-query 2x weight when inserted first - CJK FTS support (#d045a8b) — Han/Hiragana/Katakana/Hangul queries now searchable via char-level spacing of CJK runs in documents_fts (one-time migration on first qmd query after upgrade; Smriti's memory_fts is unaffected) - Embed collection filter honored (#5b9f472) - HTTP MCP rerank control (#e36ab96) - Forward candidateLimit through search APIs (#3b7e065) - Preserve docids across case-only renames (#dff6513) - macOS Metal cleanup abort mitigation (#60c75cb) Risk audit: all 11 QMD APIs Smriti imports (createStore, QMDStore, hashContent, chunkDocumentByTokens, reciprocalRankFusion, formatQueryForEmbedding, formatDocForEmbedding, RankedResult, insertEmbedding, initializeMemoryTables, Database) verified backward compatible. insertEmbedding gained an optional 7th param totalChunks (partial-embedding pending state), unused by Smriti. Also restore test scoping ("bun test --cwd ./test") so Smriti's test runner doesn't pick up QMD's own test/ files — two new upstream tests (cli-lazy-llm-import, local-config) hardcode cwd-relative paths and would otherwise fail when discovered from the parent repo. Same fix pattern as cef23f2 from the March 2026 sync. Full plan and verification at qmd/docs/UPSTREAM_MERGE_PLAN.md. * feat(daemon): scaffold server with PID-file single-instance + IPC socket First piece of the v0.8.0 daemon work (#72). Intentionally narrow: just the single-instance guard, IPC socket bind, and signal handlers. No watcher, no debounce queue, no ingest wiring — those are separate modules / commits. Implementation notes (from pre-impl smoke tests against Bun 1.3.6): - Single-instance is enforced via DAEMON_PID_FILE + kill(pid, 0) liveness probe, not Unix-socket bind contention. Bun's net.listen() silently succeeds on duplicate binds and steals connections from the original server — verified with a reproducer. PID-file pattern is the same one QMD uses for `qmd mcp --daemon`. - IPC socket is bound separately for the Claude Stop hook poke, with cleanup of any stale socket file from a previous crash. - SIGTERM/SIGINT install a graceful shutdown that closes the server, removes the socket file, removes the PID file, and exits with conventional 128+signo status for supervisor visibility. - detectRunningDaemon() handles three stale states: missing PID file (returns null), garbage PID file (cleans + null), dead PID via ESRCH (cleans + null). Live PID returns the PID; EPERM also returns the PID (process exists but is foreign — don't start alongside). 10 unit tests cover detectRunningDaemon() across the three stale states plus the live case, and startDaemon() across the happy path, contention path, stale-PID-recovery path, idempotent shutdown, and the onPoke wire. PRD also gains a new "Three pre-impl smoke-test findings" section documenting why chokidar was dropped, why socket-bind isn't the single-instance mechanism, and why ingest() will open a fresh DB handle per debounce flush. Refs #71, #72. * feat(daemon): recursive watcher with macOS-native + Linux walk-and-watch Second module of the daemon (#72). Wraps Node's fs.watch so the queue can subscribe to "anything happened under this root" with a single callback shape, regardless of OS-specific backend differences. - macOS: fs.watch(root, { recursive: true }, cb). Native FSEvents delivers a single watcher per root. - Linux: inotify doesn't implement `recursive`, so we walk the tree at startup and watch each directory. New directories are picked up on the fly by re-watching when we see a `rename` event whose target is a directory. - Windows: same code path as macOS (ReadDirectoryChangesW supports recursive natively). Event paths are normalized to absolute. Null filenames (some FS backends emit them under load) are filtered out. Errors on individual watchers are silently dropped rather than crashing the parent — losing one subdirectory is better than losing the daemon. 7 tests cover: non-existent root rejection, direct-child file creation, deep-subdirectory creation (recursion), content change, absolute-path normalization, close()-stops-events, and the watchedCount() topology assertion (1 on macOS/Windows native, N on Linux). Refs #71, #72. * feat(daemon): per-project debounce queue Third module of the daemon (#72). Coalesces bursts of "this project changed" signals into a single onFlush per project per quiet window. - schedule(projectId) resets the timer for that project. Repeated calls inside the window collapse to one firing — this is what makes a busy agent session not trigger 200 ingests as it writes JSONL. - flush(projectId) is the synchronous hook-poke path: fire onFlush immediately, cancel any pending debounce for that project. - Errors thrown by onFlush are caught and logged via the optional log callback rather than rejecting the timer's microtask. The caller (typically the daemon entry point) decides how to surface ingest errors. - close() cancels everything pending; subsequent schedule() calls become no-ops. Matches the lifecycle of the daemon process itself. Timers are unref'd so they don't keep Node alive on their own — process lifetime is owned by the IPC server, not by pending debounce timers. 9 tests cover: basic schedule/wait, coalescing across rapid schedules, per-project independence, immediate flush(), flush() with no pending timer, close() preventing pending fires, close() blocking subsequent schedules, error isolation from onFlush, and the isPending() inspector. Refs #71, #72. * feat(daemon): agent-root routing helpers Fourth module of the daemon (#72). Pure helpers that turn an FS path into the agent name responsible for it, and produce the default list of (agent, root) pairs the daemon should watch. For v0.8.0 the routing is intentionally coarse — by agent, not by project. A change anywhere under ~/.claude/projects/ schedules a single "ingest all of claude" flush, debounced. ingest() is already incremental at the session level, so unchanged sessions cost almost nothing per flush. A per-project resolution layer can replace this without changing the daemon's structure. getDefaultAgentRoots() filters by existsSync so we don't crash trying to watch a Codex or Cline install that isn't on this machine. Copilot is included only when COPILOT_STORAGE_DIR is set, since its location varies by OS and isn't auto-detected here. resolveAgentForPath() uses a strict prefix-with-separator check to avoid the classic ".claude/projects" matching ".claude/projects- archive/" bug. 6 tests cover the four match cases (exact root, child path, no match, sibling-prefix non-match) plus multi-root dispatch and the empty-root handling. Refs #71, #72. * feat(daemon): lifecycle client for stop / status Fifth and final module of the daemon core (#72). Powers `smriti daemon stop` and `smriti daemon status` without going through the IPC socket. The deliberate choice not to go through the socket: lifecycle commands need to work even when the daemon is wedged in a way that makes it unresponsive on the socket. Working through the PID file + signals is the most robust way to inspect and shut down a process. - getDaemonStatus() reads the PID, probes liveness via the existing detectRunningDaemon() helper, and includes a startedAt timestamp derived from the PID file's birthtime (falls back to ctime on filesystems that don't track birth). The PID-file races (file disappears between detect and stat) report as not-running rather than crashing. - stopDaemon() sends SIGTERM and polls for the PID file to disappear (the daemon's signal handler is responsible for unlinking it as part of graceful shutdown). Three result states: stopped, not- running, timeout. Callers — typically `smriti daemon stop` — decide how to escalate on timeout (could SIGKILL, could surface to the user). 6 tests cover both functions across no-daemon, stale-PID, and live cases. The timeout-path test temporarily swaps out the harness's SIGTERM handler so receiving the signal during the test doesn't kill the test runner. With this commit, #72 has all five core daemon modules: server, watcher, queue, handlers, client. Wiring them into a top-level daemon entry point and the CLI happens in subsequent commits. Refs #71, #72. * feat(daemon): runDaemon() entry point wiring all five modules Top-level wiring for the daemon (#72). Connects watcher → resolveAgent → queue.schedule, plus hook poke → queue.flush("claude"), plus the default onFlush that opens a fresh SQLite handle, calls ingest(), and closes the handle (per smoke-test finding 3). Dependency-injection-friendly: tests pass a mock flushAgent so they can verify the wiring without invoking real ingest() against the user's real DB. Production callers (the CLI) accept defaults and get the real ingest path. A few intentional choices: - One log function flows through every module. Defaults to console.error so foreground daemon output goes to stderr; in production the LaunchAgent/systemd unit redirects stderr to DAEMON_LOG_FILE. Tests pass () => {} to silence. - "No agent roots found" is a soft warning, not a fatal error. The daemon still runs (the hook poke still works for Claude if Claude later writes session files). Avoids the case where installing on a fresh machine fails because no agents have written logs yet. - defaultFlushAgent catches and logs both DB-open errors and ingest errors. One bad flush should not crash the daemon — the next FS event will retry. - shutdown() is idempotent and closes watchers and queue before the server. This guarantees no FS event arrives at a torn-down queue (which would be a no-op but log a misleading "closed" warning). 6 integration tests cover: single-flush via watcher, coalescing across rapid writes, hook poke wired to claude flush, multi-root routing, error isolation from flushAgent, idempotent shutdown. With this commit, #72 is structurally complete. Next step is the CLI wiring (#74) so `smriti daemon` actually invokes runDaemon(). Refs #71, #72. * feat(daemon): LaunchAgent + systemd-user installer (macOS + Linux) Implements #73. Generates the platform-appropriate service file, registers it with the system supervisor, and exposes inverse operations for uninstall. macOS: - Writes ~/Library/LaunchAgents/dev.zero8.smriti.plist - Registers via `launchctl bootstrap gui/<uid> <plist>` (modern) - Falls back to `launchctl load -w` on older macOS where bootstrap isn't available - Treats EEXIST / "already loaded" as success, not failure — that's the idempotent re-install case - Uninstall calls `launchctl bootout`, falls back to `launchctl unload`, then removes the plist Linux: - Writes ~/.config/systemd/user/smriti.service - Registers via `systemctl --user daemon-reload && systemctl --user enable --now smriti` - Service includes Restart=on-failure + RestartSec=5 so a crashed daemon comes back automatically, plus Nice=10 + IOSchedulingClass= idle so background indexing doesn't fight foreground work - Uninstall calls `systemctl --user disable --now`, removes the unit file, then daemon-reload to flush systemd's view Pure template generators (generatePlist, generateSystemdUnit) are exported for unit testing without spawning real launchctl/systemctl. Real-world interaction goes through a RunCmd abstraction that the default install path implements with Bun.spawn — tests inject a recording runner instead, so they can assert which commands would have been called without actually registering anything with the host's service manager. 13 tests cover the plist + systemd-unit generators (incl. XML escaping and ExecStart quoting), the install happy paths (macOS bootstrap + load-fallback, Linux daemon-reload + enable), the idempotent re-install path, the EEXIST-as-success case, error propagation from systemctl, and both uninstall paths. Manual integration testing (real launchctl bootstrap on this machine) will happen during the release-readiness work in #75. Refs #71, #73. * feat(cli): wire smriti daemon subcommands Implements #74. Adds the user-facing entry points for the daemon that #72 and #73 built. Six subcommands: smriti daemon Run in foreground (debugging, systemd target) smriti daemon install LaunchAgent (macOS) / systemd unit (Linux) smriti daemon uninstall Reverse install smriti daemon status PID, uptime, watched agents smriti daemon stop SIGTERM the running daemon smriti daemon logs tail -F the daemon log file Dispatch happens BEFORE initSmriti() because the foreground daemon opens its own DB handle per ingest flush rather than sharing one. Sharing a long-lived connection across many ingest calls was ruled out by pre-impl smoke test 3 (Bun segfault at ~6.8 GB peak RSS). Each subcommand uses lazy imports — the daemon module graph isn't loaded for unrelated commands like `smriti search`. Keeps the hot path cold-start unchanged. `smriti daemon status` formats the uptime in the largest-fitting unit (seconds / minutes / hours / days) so the most common state ("running for 2 days") reads naturally without grep. Logs follow tail -F semantics so the command keeps working across log rotation, which both LaunchAgents and systemd will do over time. HELP text gains a "Daemon options" block alongside Ingest, Search, Recall, etc. Manual verification: $ smriti daemon status daemon: not running PID file: /Users/zero8/.cache/smriti/daemon.pid $ smriti daemon banana Unknown daemon subcommand: banana Usage: smriti daemon [install|uninstall|status|stop|logs] smriti daemon (run in foreground) Refs #71, #74. * chore(release): bump to v0.8.0; document daemon commands in CLAUDE.md - package.json: 0.6.0 → 0.8.0. (v0.7.0 was tagged in git without a matching package.json bump; we skip past it directly to 0.8.0 since the daemon is the headline change.) - CLAUDE.md quick-reference gains a "Daemon (v0.8+)" block covering all six subcommands, plus the recommended Stop-hook template that pokes the socket when the daemon is running and falls back to lockf when it isn't. Refs #71, #75. * docs(release): add release-flow + v0.8.0 release notes Two reference docs to make the v0.8.0 tag a five-minute event: - docs/internal/release-flow.md captures the four-phase release process (feature branch → staging on hardware → checklist → tag). Intended to be reused for every future release, not just v0.8.0. Includes the upgrade-restart gap that will become v0.8.1, the "what lives where" table, and the explicit list of things we do not do (no CI release pipeline, no RC channels, no release branches kept alive past tag). - docs/internal/release-notes-v0.8.0.md is the canonical body for the GitHub release. Written in the "what an engineer would tell a colleague about" voice that we agreed releases should land in. Pulls together the headline (cross-agent capture), the three design constraints that shaped it (smoke-test findings), the recommended Stop-hook update, what's deferred to 0.8.1, and the postmortem provenance that got us here. Issue #75 now contains the real-hardware acceptance checklist that gates tagging — once those boxes are green, the commands in release-flow.md execute the tag. Refs #71, #75. * fix(ingest): parse Codex rollout response_item format Codex CLI (codex_cli_rs >= ~0.40) wraps messages as {type:'response_item', payload:{type:'message', role, content:[{type:'input_text'|'output_text', text}]}}. The parser only understood flat {role, content} entries, so every modern session parsed to zero messages and was silently skipped. Also filters injected context blocks (AGENTS.md, environment_context) recorded as user messages. * fix(ingest): support VS Code .jsonl chatSessions and new field shapes VS Code now writes chatSessions as JSONL with a {kind:0, v:{...session}} snapshot line (single- or multi-line). Request text moved to message.text; response items carry markdown in .value with a kind discriminator (thinking/progress/tool kinds are skipped). Discovery glob now includes *.jsonl. * fix(ingest): honor original timestamps on backfill; bulk-ingest enrichment kill switch addMessage() now accepts options.timestamp and uses it for message and session created_at — backfilled history keeps its real dates instead of collapsing to ingest day (live hook ingests are unchanged: default now). SMRITI_INGEST_NO_ENRICH=1 skips per-session LLM query expansion and collection sync during bulk backfills: 482 queued local-llama inferences pegged the CPU for 20+ minutes. Run smriti enrich/embed explicitly after. * feat(ingest): Cursor globalStorage ingest — real chat history from state.vscdb Cursor's actual history lives in globalStorage/state.vscdb (cursorDiskKV: composerData:* + bubbleId:* keys), not project .cursor/*.json. Adds: - read-only SQLite discovery of all composers (inline conversation and headers+bubble formats; bubble timestamps fall back to composer createdAt) - composerId -> workspace folder mapping from workspaceStorage for project resolution; CURSOR_STORAGE_DIR override; Linux/Windows paths - smriti ingest cursor now works without --project-path (legacy .cursor JSON path retained behind the flag) - --force re-ingest now deletes memory_messages first (was appending duplicates) Recovers 482 sessions / 22k messages on this machine. 14 new tests. * chore(scripts): zero-dependency local server for markdown reports * docs(release): v0.8.0 notes — ingest correctness section + retrospective link
1 parent 74922d2 commit 281bf9e

49 files changed

Lines changed: 7858 additions & 1063 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

CLAUDE.md

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -134,8 +134,34 @@ smriti embed # Build vector embeddings
134134
smriti categorize # Auto-categorize sessions
135135
smriti share --project myapp # Export to .smriti/ for git
136136
smriti sync # Import team knowledge
137+
138+
# Daemon (v0.8+) — cross-agent capture in the background
139+
smriti daemon install # Install LaunchAgent/systemd unit; auto-start at login
140+
smriti daemon status # PID, uptime, watched agents
141+
smriti daemon stop # Graceful shutdown
142+
smriti daemon logs # Tail the daemon log
143+
smriti daemon uninstall # Reverse install
144+
smriti daemon # Run in foreground (debugging)
137145
```
138146

147+
### The Claude Stop hook with the daemon
148+
149+
When the daemon is installed, the Claude Stop hook becomes a 5ms socket
150+
poke instead of a full ingest invocation. Recommended template:
151+
152+
```bash
153+
#!/bin/bash
154+
SOCK="$HOME/.cache/smriti/daemon.sock"
155+
if [ -S "$SOCK" ]; then
156+
: | nc -U "$SOCK" 2>/dev/null
157+
else
158+
/usr/bin/lockf -t 0 /tmp/smriti-ingest.lock smriti ingest claude 2>/dev/null
159+
fi
160+
exit 0
161+
```
162+
163+
The `lockf` fallback keeps the system working when the daemon isn't running.
164+
139165
## Project Structure
140166

141167
```

bun.lock

Lines changed: 8 additions & 809 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

docs/internal/daemon-prd.md

Lines changed: 162 additions & 0 deletions
Large diffs are not rendered by default.

docs/internal/release-flow.md

Lines changed: 134 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,134 @@
1+
# Release flow
2+
3+
How a Smriti version goes from "code on a feature branch" to "tagged release that downstream users pick up via `smriti upgrade`." Written after the v0.8.0 work; intended to be reused for every future release.
4+
5+
## Versioning
6+
7+
Standard semver. The general rule: features bump minor (0.7.0 → 0.8.0), bug fixes and polish bump patch (0.8.0 → 0.8.1), and we will reach 1.0.0 once the daemon has been running on real teams for a month without anyone hitting a "this is broken in a load-bearing way" issue.
8+
9+
A note about local drift: `package.json` has occasionally lagged the git tag (v0.7.0 was tagged without a corresponding `"version"` bump). Try to keep them in sync; if they drift, fix it during the next release rather than rewriting history.
10+
11+
## The four phases
12+
13+
### Phase 1 — Feature branch development
14+
15+
All work for a release lives on one feature branch, named `feat/<headline-shape>` (e.g. `feat/daemon-core` for v0.8.0). One feature branch per release, even if the release contains several modules.
16+
17+
The branch accumulates commits as we go. We don't try to keep the branch always-rebased-to-main during development — that creates more friction than it solves for a solo developer. Instead we squash-or-merge when we're ready to ship.
18+
19+
Each commit on the branch should be independently testable: `bun test` passes after each commit. This makes bisecting later much cheaper.
20+
21+
### Phase 2 — Staging on real hardware
22+
23+
When the branch is feature-complete and unit-tested, we install it on real hardware and exercise it. There's no separate "build artifact" — the source IS the build, and switching to staging is `git checkout <feature-branch> && bun install`.
24+
25+
For the developer (running from `/Users/zero8/zero8.dev/smriti/`):
26+
```bash
27+
cd /Users/zero8/zero8.dev/smriti
28+
git fetch origin
29+
git checkout feat/<branch-name>
30+
bun install --frozen-lockfile
31+
bun test # sanity
32+
```
33+
34+
For downstream users (running from `~/.smriti` as a git clone):
35+
```bash
36+
cd ~/.smriti
37+
git fetch origin
38+
git checkout feat/<branch-name>
39+
bun install --frozen-lockfile
40+
```
41+
42+
If the release adds a long-running process or service-file install, the staging step includes those too — e.g. for v0.8.0:
43+
```bash
44+
bun src/index.ts daemon install
45+
bun src/index.ts daemon status # verify it's running
46+
```
47+
48+
To leave staging, `git checkout main && bun install --frozen-lockfile` (and remove any service-file installs).
49+
50+
### Phase 3 — Release-readiness checklist
51+
52+
Every release has a tracking issue with a checklist of real-hardware verifications. The checklist is release-specific (a daemon release tests reboot + soak; a search-quality release tests recall against a fixture set; etc.) but the shape is consistent:
53+
54+
- Per-OS verification rows that have to be done on actual machines
55+
- Soak / endurance rows where time itself is the test
56+
- Idempotency rows (running install twice, etc.) that catch state-corruption bugs
57+
- A "previous CLI still works" row to catch regressions
58+
59+
When all rows are ✅, we tag. When a row fails, we fix it on the feature branch with a small commit and re-run the row — same branch, just more commits.
60+
61+
The tracking issue for v0.8.0 is #75. Future releases should clone its structure.
62+
63+
### Phase 4 — Promotion to release
64+
65+
```bash
66+
# 1. Final sanity check
67+
cd /Users/zero8/zero8.dev/smriti
68+
git checkout feat/<branch-name>
69+
bun test # all green
70+
bun src/index.ts <whatever-needs-spot-check> # smoke-test the headline feature
71+
72+
# 2. Merge the PR
73+
gh pr merge <PR-#> --squash --delete-branch # squash if many commits and you don't need the history
74+
# --merge if you want the commit-by-commit story preserved
75+
76+
# 3. Tag
77+
git checkout main && git pull
78+
git tag -a v<x.y.z> -m "v<x.y.z> — <one-line headline>"
79+
git push origin v<x.y.z>
80+
81+
# 4. GitHub release with notes
82+
gh release create v<x.y.z> \
83+
--title "v<x.y.z> — <one-line headline>" \
84+
--notes-file docs/internal/release-notes-v<x.y.z>.md \
85+
--latest
86+
```
87+
88+
The release notes file lives in the repo (`docs/internal/release-notes-v<x.y.z>.md`) as a draft from Phase 1, gets polished during Phase 3, and is the canonical source for the GitHub release body in Phase 4. After tagging, the file can stay in the repo as historical record — it's small and useful when someone asks "what landed in 0.8?"
89+
90+
### Optional Phase 5 — Daemon / long-running-process restart
91+
92+
For releases that ship changes to a long-running process (the daemon, future MCP server, etc.), users who upgrade need to restart that process to pick up the new code. Today this is manual:
93+
94+
```bash
95+
smriti upgrade # git pull + bun install
96+
smriti daemon stop
97+
launchctl kickstart -k gui/$UID/dev.zero8.smriti # macOS; KeepAlive=true will respawn it
98+
# or: systemctl --user restart smriti # Linux
99+
```
100+
101+
A v0.8.1 polish release should teach `smriti upgrade` to detect a running daemon and restart it automatically. Tracked separately — not load-bearing for v0.8.0 itself.
102+
103+
## What lives where
104+
105+
| Artifact | Location | When updated |
106+
|---|---|---|
107+
| Release-tracking issue | GitHub issue (one per release) | Created at start of Phase 3; closed when tagged |
108+
| Release notes | `docs/internal/release-notes-v<x.y.z>.md` | Drafted Phase 1, polished Phase 3, used in Phase 4 |
109+
| Version | `package.json` `"version"` | Bumped in the same commit as the release notes finalisation |
110+
| CHANGELOG | We don't maintain one. The set of GitHub Releases is the changelog. ||
111+
| Reference doc per major change | `docs/internal/*-prd.md` | Drafted alongside the feature; stays in the repo as historical record |
112+
| Postmortems / reflections | `docs/papers/` | When something is worth telling as a story |
113+
114+
## What we deliberately don't do
115+
116+
- **No CI release pipeline.** Releases are small enough and rare enough that automating them past `gh release create` adds more failure modes than it removes. If we ever release multiple times a week, revisit.
117+
- **No release candidates or beta channels.** Staging on the feature branch IS the RC. If a release needs longer soak time before tagging, just leave it in Phase 3 longer.
118+
- **No release branches.** `main` is always the latest stable; feature branches are everything else. Branching off a tag for a hotfix is fine, but we don't keep a `release/0.8.x` branch alive after tagging.
119+
- **No version-skipping for ceremony.** If v0.7.0 was tagged without a `package.json` bump, the next release just skips ahead in `package.json` — we don't go back and re-tag 0.7.1 to fix the drift.
120+
121+
## When something goes wrong post-release
122+
123+
If a release ships with a regression bad enough to revert:
124+
125+
1. `git revert <merge-commit>` on main (creates a clean revert commit)
126+
2. Tag v<x.y.z+1> from the revert
127+
3. Push tag, create release marked as a regression revert
128+
4. Users on `smriti upgrade` pick up the revert via the normal flow
129+
130+
For less severe issues, a regular patch release (v<x.y.z+1> with the fix) is preferred over a revert.
131+
132+
## Source of truth for the current release
133+
134+
Always the GitHub release for the highest tag. If `package.json` disagrees with the tag, the tag wins. If a doc disagrees with the code, the code wins. We are explicit about this so future-us doesn't get confused by stale documentation that says we shipped something we didn't.
Lines changed: 97 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,97 @@
1+
# Smriti v0.8.0 — Cross-agent capture, finally
2+
3+
The headline: a long-running `smriti daemon` that captures your sessions across every coding agent in the background. Open Cursor, Codex, or Claude — the daemon watches the filesystem, debounces, and ingests automatically. You stop having to remember which agent you used yesterday, or whether you remembered to `smriti ingest`.
4+
5+
This is the release the postmortem [`docs/papers/stop-hook-never-stopped.md`](../papers/stop-hook-never-stopped.md) gestured at — the daemon shape the lockf mitigation pointed toward. It also reuses everything Smriti was already doing for Claude (the Stop hook continues to work, just as a 5ms socket poke now instead of a full ingest).
6+
7+
Before tagging, we dogfooded the whole pipeline: pointed Smriti at a year of our own sessions across all four agents and had a fleet of analysis agents mine what the developer actually learned. The result is the [Builder Retrospective: May 2025 → June 2026](https://zero8.dev/blog/builder-retrospective-may-2025-june-2026) — and the exercise itself found and fixed real bugs in every non-Claude connector, all included below.
8+
9+
## What you get
10+
11+
- **Cross-agent capture.** Sessions from Claude, Codex, Cline, Copilot, and Cursor are picked up automatically as they're written. No `smriti ingest <agent>` to remember.
12+
- **Auto-start at login.** `smriti daemon install` writes a LaunchAgent on macOS or a systemd-user unit on Linux. Daemon comes back after every reboot; restarts itself on crash.
13+
- **Per-project debouncing.** A busy session in project A doesn't delay project B. Each project gets its own 30s settle window.
14+
- **Six new commands** (`smriti daemon install / uninstall / status / stop / logs`, plus the bare `smriti daemon` for foreground debugging).
15+
- **`smriti share` continues to work** with its existing sanitization — unchanged.
16+
17+
## Recommended Claude Stop-hook update
18+
19+
When the daemon is running, the Stop hook becomes a 5ms poke. Update your `~/.claude/hooks/save-memory.sh` to:
20+
21+
```bash
22+
#!/bin/bash
23+
SOCK="$HOME/.cache/smriti/daemon.sock"
24+
if [ -S "$SOCK" ]; then
25+
: | nc -U "$SOCK" 2>/dev/null
26+
else
27+
/usr/bin/lockf -t 0 /tmp/smriti-ingest.lock smriti ingest claude 2>/dev/null
28+
fi
29+
exit 0
30+
```
31+
32+
The `lockf` fallback keeps capture working if the daemon isn't running. No flag day; the old hook continues to work too.
33+
34+
## Quick start
35+
36+
```bash
37+
smriti upgrade # pull the new code
38+
smriti daemon install # register the LaunchAgent / systemd unit
39+
smriti daemon status # PID, uptime, watched agents
40+
```
41+
42+
That's it. Open your agent of choice, work for a while, then `smriti search` for whatever you did. The session will be there.
43+
44+
## What's in the box
45+
46+
- 6 daemon modules (~1500 LOC of code, ~1200 LOC of tests)
47+
- 57 daemon tests passing (full suite: 1174+ tests)
48+
- 9 commits on `feat/daemon-core`, all individually testable
49+
- 1 dedicated PRD (`docs/internal/daemon-prd.md`) documenting both the design and the three pre-impl smoke-test findings that shaped it
50+
51+
## Design discipline (the boring details that matter)
52+
53+
Three constraints came out of pre-impl smoke tests against Bun 1.3.6, and each one shaped a design decision that would have silently bitten us in production:
54+
55+
- **Single-instance enforcement uses a PID file + `kill(pid, 0)` liveness probe, not Unix-socket bind contention.** Bun's `net.createServer().listen(path)` silently succeeds on duplicate binds and steals connections from the original server. We use the PID-file pattern QMD already uses for `qmd mcp --daemon`.
56+
- **FS watching uses native `fs.watch({ recursive: true })`, not chokidar.** Chokidar 5.0.0 under Bun fires zero events; native `fs.watch` works correctly on macOS (recursive native) and Linux (walk-and-watch).
57+
- **DB connections are per-flush, not per-daemon-lifetime.** Repeatedly calling `ingest()` against a single long-lived SQLite handle inside one Bun process climbed to 6.8 GB peak RSS and segfaulted Bun. Opening a fresh connection per flush sidesteps this entirely; the ~30ms cost is invisible inside the 30s debounce window.
58+
59+
All three findings are documented in [`docs/internal/daemon-prd.md`](daemon-prd.md).
60+
61+
## Ingest correctness — found by dogfooding
62+
63+
Every non-Claude agent connector had silently drifted from the current on-disk formats. All "Sessions found: N, ingested: 0" failure modes, all fixed:
64+
65+
- **Codex** — modern rollouts wrap messages in `{type:"response_item", payload:{...}}` envelopes; the parser now unwraps them and filters injected context (AGENTS.md, environment blocks).
66+
- **Copilot (VS Code)** — chatSessions moved to `.jsonl` with `{kind:0, v:{...}}` snapshot lines, and text moved to `message.text` / response `value` fields. Discovery and parsing updated.
67+
- **Cursor** — the big one. Real chat history lives in `globalStorage/state.vscdb` (`composerData:` + `bubbleId:` keys), not project `.cursor/` dirs. New read-only SQLite discovery with composer→workspace project mapping; `smriti ingest cursor` now works with no flags. Recovered 482 sessions / 22k messages on the dev machine.
68+
- **Backfill correctness**`addMessage()` honors original message timestamps (history no longer collapses to ingest day); `--force` re-ingest deletes prior messages instead of appending duplicates; `SMRITI_INGEST_NO_ENRICH=1` skips per-session LLM query expansion during bulk backfills (482 queued local-LLM inferences previously pegged the CPU for 20+ minutes).
69+
70+
## Platforms
71+
72+
- ✅ macOS 14+ (Apple Silicon and Intel)
73+
- ✅ Linux (systemd-user supported)
74+
- ⏸️ Windows — deferred to a later release. Bun's Windows daemon support is rough and named-pipe semantics differ enough from Unix sockets that we want to ship them separately rather than half-build them now.
75+
76+
## Upgrading
77+
78+
If you're coming from v0.6.0 / v0.7.0:
79+
80+
1. `cd ~/.smriti && smriti upgrade` (or your equivalent — wherever your smriti install lives)
81+
2. `smriti daemon install` if you want the daemon. Optional — if you skip this, Smriti continues to work exactly as it did before via the existing Claude Stop hook.
82+
83+
If you do install the daemon and later decide to roll back, `smriti daemon uninstall` removes the service file and stops the daemon. The PID file and IPC socket are cleaned up automatically. There is no other state to migrate.
84+
85+
## What's not in this release
86+
87+
A few things explicitly deferred to keep this release tight:
88+
89+
- **No real redaction pipeline.** `smriti share` still does the basic sanitization it always has. Real redaction comes in v0.8.1 / v0.9.0.
90+
- **No read-side routing through the daemon.** `smriti search` and `smriti recall` are still one-shot CLI invocations. The daemon doesn't speed them up.
91+
- **No auto-restart on `smriti upgrade`.** After upgrading, you'll want to `smriti daemon stop` followed by `launchctl kickstart -k gui/$UID/dev.zero8.smriti` (or `systemctl --user restart smriti` on Linux) so the daemon picks up the new code. v0.8.1 will teach `smriti upgrade` to do this automatically.
92+
93+
## Thanks
94+
95+
This release came out of a debugging session that found 42 stuck `smriti ingest` processes consuming 9 CPU-days. The lockf mitigation that stopped the pile-up is still in place as the fallback path; the daemon makes it usually-unnecessary. Both stories live in `docs/papers/`.
96+
97+
Refs: #71, #72, #73, #74, #75. PR #76.

0 commit comments

Comments
 (0)