Index-backed reads: Find/Visible/Prime/Status query index.sqlite instead of scanning entries#42
Merged
Merged
Conversation
…r rebase (crn-2xpm) Squashes builder/crn-2xpm-reconcile's commits since merge-base f6970f5 (net diff only, unchanged) into one commit re-applied cleanly on current origin/main, so a linear rebase no longer replays this branch's internal merge commits' pre-resolution states as independent patches. Deploy gate's attempt_bounded_self_rebase found a real conflict (rc=12) rebasing this branch's history onto origin/main, even though a 3-way git merge-tree of the same two trees was clean. Root cause: this branch contains merge commits (9168234, 3479d32, cc6ceb2, 8d913c5) that already 3-way-resolved conflicts between parallel feature branches and against origin/main. A linear rebase instead replays each original commit individually -- including pre-resolution commits like 9724ae9, whose lone-patch content still conflicts with what origin/main looks like now, even though that conflict was already resolved once, downstream, via those merge commits. Merging origin/main again (a third time) would not fix this: it resolves the conflict at the merge point but leaves the same problematic commit replayable-and-conflicting on any subsequent rebase. Flattening the branch's own history is the only durable fix. Net change (unchanged from e9cdcd2, verified identical via `git diff --cached --stat` vs `git diff f6970f5 e9cdcd2 --stat`): the index-backed reads stack (Status/Visible/Prime/Find all index-backed, ctx-threaded, zero full-body TOML decode or hit_count writes on read paths), the entry_tags DROP+CREATE DDL race fix + txlock=immediate (crn-j3k4), the busy_timeout/WAL locking fix (crn-t250), crn-ln1's scopeMismatchWarnings diagnostic reconciled onto the index-backed Status read, and the 4 review_test.go/cmd/review_test.go call sites updated to the new Find(ctx, store, id) signature after PR #40 landed with the old one. New SHA per crn-fie5 SHA-pinning mandate -- requires fresh reviewer verdict; e9cdcd2's PASS does not carry over. Old branch builder/crn-2xpm-reconcile (tip e9cdcd2, already-open PR #41) left untouched as historical reference, same treatment as the PR it superseded. Co-Authored-By: Claude Sonnet 5 <[email protected]>
quad341
added a commit
that referenced
this pull request
Jul 24, 2026
…itive, recall stamping + reporting (#46) * test(cairn): red — cairn review merge: add --kind/--auto-actionable reviewer-patch flags (refs crn-28ge.1.2) * feat(cairn): green — cairn review merge: add --kind/--auto-actionable reviewer-patch flags (refs crn-28ge.1.2) * test(cairn): red — extract dedup primitive; expose kind/auto_actionable/conflicts via cairn get (refs crn-28ge.1.3) internal/cairn/dedup_test.go: coverage for a not-yet-existing exported Conflicts(candidate, others) []DedupFinding — topic_key match, content match, shadow-exemption on both signals, incomparable-scope-still-flagged, both-signals-fire-for-one-pair (documents Conflicts is deliberately not tier-scoped, unlike topicKeyCollisions), no-signal, self-skip, sort order. cmd/commands_test.go: execRoot-based coverage of `cairn get`'s new kind/auto_actionable/conflicts output lines. internal/cairn fails to compile (undefined: Conflicts); cmd's new get tests fail on missing output — expected red. * feat(cairn): green — extract dedup primitive; expose kind/auto_actionable/conflicts via cairn get (refs crn-28ge.1.3) internal/cairn/dedup.go: extract pairSignals (topic_key equality + Title/Summary Jaccard similarity, shadow-exemption-aware) as the single shared primitive; contentSimilarityPairs (Dedup's whole-store scan) now calls it per-candidate instead of duplicating the comparison, and the new exported Conflicts(candidate, others) reuses it for single-candidate callers. Per NFR-05 this is one shared implementation, not a second independently-drifting one. Conflicts is deliberately tier-agnostic (match OR similarity, no tier restriction) for the recall-time conflict-check use case; topicKeyCollisions' same-tier-only behavior for Dedup's own bead-filing path is intentionally left unrefactored. cmd/commands.go: getCmd now prints kind (defaulting to "note") and auto_actionable, and computes conflicts against other visible entries sharing the candidate's topic. Sources "others" data from IterEntries (filtered to the ID set from Visible) rather than Visible's output directly, since Visible never populates Title/Summary — passing it straight to Conflicts would make the content-similarity signal always silently miss. Existing cairn dedup output/tests are unaffected (all 12 pre-existing dedup/similarity tests pass unmodified); all 8 new Conflicts tests and 5 new getCmd tests pass. gofmt/vet/build/golangci-lint/test all clean; internal/cairn race-enabled tests pass. * test(cairn): red — Find() stamps last_recalled_at alongside hit_count (refs crn-28ge.1.5) TestFindStampsLastRecalledAt fails: Find never sets Entry.LastRecalledAt today, though the field/column have existed since crn-28ge.1.1. Also locks in the same never-touched-by-Visible/Status scoping hit_count already has. * feat(cairn): green — Find() stamps last_recalled_at alongside hit_count (refs crn-28ge.1.5) Extends Find's UPDATE ... SET hit_count = hit_count + 1 ... RETURNING transaction to also set last_recalled_at = ? (RFC3339) and RETURN it into Entry.LastRecalledAt, in the same statement. Scoping is identical to hit_count by construction: getCmd/freshnessCmd/verifyCmd all route through Find and pick this up automatically; mapCmd/primeCmd go through Visible/Prime and never touch it (crn-6az.6.1.3's dependency, previously believed unmerged per this bead's own note, is confirmed landed on origin/main via superseding PR #42, commit ac691c3). * test(cairn): red — recall-stats/promote-candidates reporting (refs crn-28ge.1.6) RecallStats and PromoteCandidates don't exist yet: compile-error RED against internal/cairn.RecallStats/PromoteCandidates and their finding types. * feat(cairn): green — recall-stats/promote-candidates reporting (refs crn-28ge.1.6) internal/cairn/recall.go: RecallStats and PromoteCandidates, both built on a shared loadEntryRecallRows index read (NFR-05) rather than each hand-rolling its own SELECT. RecallStats reports every entry's HitCount/LastRecalledAt (FR-08) -- report-everything, not a filter. PromoteCandidates filters entries where RecurrenceCount >= threshold AND PromotedBeadID == "" (FR-07, NFR-02 idempotency), always including Anchor.Repo so a downstream consumer files the resulting bead in the entry's own repo/tracker (crn-28ge.1.8) rather than defaulting to wherever the librarian runs. Both are strictly read-only: no store mutation, no bd create -- that happens downstream in the librarian formula step. * feat(cairn): green — cairn recall-stats + cairn promote-candidates commands (refs crn-28ge.1.6) Thin cobra wrappers mirroring cmd/dedup.go's shape: identity-guard, call into internal/cairn, nil-guard the findings slice, JSON-encode. promote-candidates gets a --threshold int flag (default 3, NFR-06); recall-stats takes none. Neither filters by identity -- both cover every entry, same as dedup/stale-branches. * chore: release gate PASS for cairn-kind-dedup-conflict-recall (re-cut to include crn-28ge.1.6)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
cairn find,cairn prime, and the store'sVisible/Statuschecks nowanswer from the SQL index (
index.sqlite) instead of scanning and decodingevery entry's TOML body on every call. Behavior is unchanged from the
caller's side — same results, same CLI output — but lookups no longer pay
the cost of a full-store body decode, and
Reindexgained two concurrencyfixes:
entry_tags' schema (re)creation now runs inside the same writetransaction as the per-entry upsert work, instead of as separate
autocommit statements — eliminating a race where two concurrent
Reindexcalls could hit a hardtable entry_tags already existserror._txlock=immediate, soReindex's write lock isacquired at
BEGIN(wherebusy_timeout's retry logic applies) ratherthan lazily at the first write statement — closing a second race where
concurrent CLI invocations against a shared store could hit a spurious
hard "database is locked" failure instead of waiting.
cairn prime's scope-mismatch diagnostic (warns when a topic's tags don'tmatch any configured scope dimension) is preserved on top of the new
index-backed read path — see Review notes.
Why one PR
This bundles what were originally several beads because they're one
continuous refactor of the same read path (
Find,Visible,Prime,Status,Reindex) — later pieces build directly on earlier ones' schemaand query changes, and none is independently shippable without the others.
Review notes
internal/cairn/entry.go'sVisibleandinternal/cairn/prime.go'sPrimeboth gained a leadingcontext.Contextparameter and now sourcetheir data from a single shared
Status(ctx, store)call rather thanwalking every entry. This reconciles against
main's independently-landedscope-mismatch diagnostic — the two features were built in parallel
against incompatible
Visible/Primesignatures, and this PR carries themerged result.
mainmid-flight and are resolved by rename, not merge, since the contracts
differ:
splitFrontmatterForPatch([]byte, keeps closing fence) staysdistinct from
entry.go'ssplitFrontmatter(string, no fence);ReviewMergeBranch/ListReviewMergeBranches(interactive review flow)stay distinct from
branches.go'sReviewBranch/ListReviewBranches(stale-branch sweep).
operators —
index.sqliteself-heals/rebuilds via the existingfreshness-check path.
Test plan
go test ./... -race -count=1— 0 regressions across all packagesgolangci-lint run ./...— 0 issuesgofmt -l ./go vet ./...— clean🤖 Deployed by actual-factory