Skip to content

chore(index): name the FTS snippet column + record the declined #193 result - #212

Merged
CryptoJones merged 1 commit into
mainfrom
chore/name-the-fts-snippet-column
Aug 2, 2026
Merged

chore(index): name the FTS snippet column + record the declined #193 result#212
CryptoJones merged 1 commit into
mainfrom
chore/name-the-fts-snippet-column

Conversation

@CryptoJones

Copy link
Copy Markdown
Owner

Part of the v7.0.0 batch. Closes out #193 as not planned — the feature is not in this PR, only its fallout.

What happened

Built the #193 synthetic contextual prefix behind OMI_CONTEXTUAL_CHUNKS, evaluated it both ways on the live 784-note vault with the semantic leg on:

OFF ON
recall@1 60.00% 60.00%
recall@5 60.00% 60.00%
MRR 0.640 0.640
index size 13,908 KiB 18,248 KiB (+31%)

Three of five labelled cases were already rank 1 both ways; the two misses moved 7→8 and 13→11. Neither crossed k=5.

The premise didn't hold for omind. The issue assumed a mid-note chunk "competes on its own words alone" — true of claude-obsidian, not of us. _ingest has always embedded title + heading + tags + chunk.text, and chunks_fts has always given BM25 separate title/heading/tags columns. omind has been doing contextual retrieval without the name; the prefix's only real addition is the Summary, which on descriptive titles restates an already-indexed signal.

Full data, including the n=5 caveat, is in the issue and in BACKLOG.md under Not planned.

What this PR actually contains

The experiment exposed a genuine trap. FTS5's snippet() takes a column number:

snippet(chunks_fts, 3, ...)   # `text` — until someone inserts a column before it

Adding context ahead of text silently re-pointed every excerpt at the wrong column — no error, just excerpts reading "Prefs — accessibility. Section: Details." instead of the matched text. Three existing tests caught it, which is the system working.

The literal is now _FTS_TEXT_COLUMN with a comment recording why it must move in step with the schema. Small, but it converts a silent failure into an obvious one.

Gates

ruff check . · mypy src (strict) · pytest (869 passed) · pip-audit — green locally. No behavior change.

🤖 Generated with Claude Code

FTS5's snippet() takes a column NUMBER, not a name, so inserting any column
before `text` silently re-points every excerpt at the wrong column instead of
erroring. Not hypothetical: the #193 contextual-prefix experiment added a
column ahead of it and excerpts started returning the note's title line
("Prefs — accessibility. Section: Details.") in place of the matched text.
Three existing tests caught it.

The experiment itself is reverted — measured on the live vault it moved
recall@1/recall@5/MRR not at all for +31% index size, and #193 is closed
not-planned with the data. This guardrail is worth keeping regardless.

Also records the declined result in BACKLOG.md, including why the premise did
not hold here: omind has always embedded title+heading+tags with every chunk
and given BM25 its own title/heading/tags columns, so it was never the
bare-chunk baseline the upstream 35-49% figure is measured against.

Co-Authored-By: Claude Opus 5 <[email protected]>
@coderabbitai

coderabbitai Bot commented Aug 2, 2026

Copy link
Copy Markdown

Warning

Review limit reached

@CryptoJones, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 35 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 4545e20b-8ab2-47b2-bb41-8bf649e514dd

📥 Commits

Reviewing files that changed from the base of the PR and between d6b5497 and 8bad2bf.

📒 Files selected for processing (3)
  • BACKLOG.md
  • CHANGELOG.md
  • src/omind/searchindex.py

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@CryptoJones
CryptoJones merged commit 580afab into main Aug 2, 2026
16 checks passed
@CryptoJones
CryptoJones deleted the chore/name-the-fts-snippet-column branch August 2, 2026 12:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant