Skip to content

feat: complete the hybrid retrieval backlog - #182

Merged
CryptoJones merged 1 commit into
mainfrom
feat/retrieval-backlog-batch
Jul 27, 2026
Merged

feat: complete the hybrid retrieval backlog#182
CryptoJones merged 1 commit into
mainfrom
feat/retrieval-backlog-batch

Conversation

@CryptoJones

Copy link
Copy Markdown
Owner

Summary

  • add a labelled retrieval-quality harness, local top-k reranking, temporal validity, generated-note weighting, adaptive scopes, and dynamic core-memory promotion
  • throttle index refreshes, quantize embeddings to int8, and serve lint/duplicate detection from the index with fail-open fallbacks
  • add search-index doctor diagnostics and reviewed omind consolidate proposals with stale-source protection
  • introduce unified graph(op=...) with a one-release compatibility bridge; alias removal remains tracked in Remove deprecated graph-* MCP aliases after the compatibility release #181

Validation

  • ruff check .
  • mypy src (strict)
  • pip-audit
  • pytest -q — 839 passed
  • forced fallback retrieval/lint/consolidation set — 109 passed, 1 index-only assertion deselected

Closes #167
Closes #168
Closes #169
Closes #170
Closes #171
Closes #172
Closes #173
Closes #174
Closes #175
Closes #176
Closes #177
Closes #178

Proudly Made in Nebraska. Go Big Red! 🌽 https://xkcd.com/2347/

@coderabbitai

coderabbitai Bot commented Jul 27, 2026

Copy link
Copy Markdown

Review Change Stack

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: a92c8035-ace8-4e3a-bd48-e01962122968

📥 Commits

Reviewing files that changed from the base of the PR and between 2009685 and 4a2a85a.

📒 Files selected for processing (26)
  • BACKLOG.md
  • CHANGELOG.md
  • docs/retrieval.md
  • src/omind/access.py
  • src/omind/bench.py
  • src/omind/cli.py
  • src/omind/consolidate.py
  • src/omind/hooks.py
  • src/omind/lint.py
  • src/omind/merge.py
  • src/omind/paths.py
  • src/omind/recall.py
  • src/omind/searchindex.py
  • src/omind/seeds.py
  • src/omind/server.py
  • src/omind/store.py
  • tests/test_access.py
  • tests/test_bench.py
  • tests/test_cli.py
  • tests/test_cli_integration.py
  • tests/test_consolidate.py
  • tests/test_hooks.py
  • tests/test_lint.py
  • tests/test_searchindex.py
  • tests/test_server.py
  • tests/test_store.py

Cache: Disabled due to Reviews > Disable Cache setting


📝 Walkthrough

Summary by CodeRabbit

  • New Features

    • Added reviewed near-duplicate note consolidation with proposal and safe apply workflows.
    • Added supersession metadata for notes and improved handling of outdated information in search.
    • Added unified graph queries with path, orphan, dangling-link, and statistics operations.
    • Added dynamic session context based on recently accessed notes.
    • Added search-index health diagnostics and retrieval-quality benchmarking.
  • Improvements

    • Improved search ranking, excerpts, refresh efficiency, and index storage size.
    • Enhanced doctor, linting, and changelog guidance for search and graph features.

Walkthrough

This PR reworks omind's hybrid search index (int8 quantized vectors, throttled refresh, adaptive query scope, reranking, and generated/superseded-note weighting), adds note supersession metadata, access-based dynamic core promotion, an omind consolidate near-duplicate merge workflow, index-backed lint, search-index health diagnostics, a quality benchmark harness, and a unified graph MCP tool.

Changes

Retrieval Quality & Search Index Overhaul

Layer / File(s) Summary
Quantized vector storage
src/omind/searchindex.py, tests/test_searchindex.py, docs/retrieval.md, CHANGELOG.md
Embeddings are stored as int8 vectors with per-vector scale/residual instead of packed float32; schema version bumped to 4.
Throttled index refresh
src/omind/searchindex.py, src/omind/store.py, tests/test_searchindex.py
Adds a cheap directory/sync-signal signature to skip redundant refreshes across bursty queries, while writes always signal invalidation.
Adaptive query scope
src/omind/searchindex.py, tests/test_searchindex.py
Query-complexity-derived depth/limit/excerpt budgets are threaded through BM25, vector, and excerpt generation.
Fused reranking and weighting
src/omind/searchindex.py, tests/test_searchindex.py, docs/retrieval.md
Reranks the fused top-k using whole-chunk embeddings and applies ranking penalties for superseded and auto-generated notes.
Lint via search index
src/omind/lint.py, src/omind/searchindex.py, tests/test_lint.py
Lint loads notes/links from the index (with a fence-stripping link extractor) and uses vector-based near-duplicate detection, falling back to a full scan when unavailable.
Search-index health & doctor
src/omind/searchindex.py, src/omind/cli.py, tests/test_cli_integration.py, tests/test_searchindex.py
Adds a read-only Health report and wires FTS5/semantic/index staleness/corruption reporting into omind doctor.
Quality eval harness
src/omind/bench.py, src/omind/cli.py, tests/test_bench.py
Adds a labelled query dataset and run_quality() computing recall@1/@5 and MRR via omind bench --quality.
Backlog/changelog updates
BACKLOG.md, CHANGELOG.md, docs/retrieval.md
Documents the completed retrieval-quality/efficiency work and updated index build metrics.

Estimated code review effort: 4 (Complex) | ~75 minutes

Note Supersession, Access Tracking & Consolidation

Layer / File(s) Summary
Supersedes metadata contract
src/omind/store.py, src/omind/merge.py, src/omind/cli.py, src/omind/server.py, tests/test_store.py
Adds supersedes/superseded_by fields to NoteFields, parse/render support, merge preservation, CLI flags, and MCP create/edit-note support.
Access tracking & dynamic core
src/omind/access.py, src/omind/hooks.py, src/omind/recall.py, src/omind/server.py, src/omind/paths.py, docs/retrieval.md, tests/test_access.py, tests/test_hooks.py
New SQLite-backed access module records read frequency/recency; SessionStart promotes up to three earned notes into a bounded dynamic core.
Consolidation workflow
src/omind/consolidate.py, src/omind/store.py, src/omind/cli.py, src/omind/paths.py, docs/retrieval.md, tests/test_consolidate.py, tests/test_cli.py
Adds omind consolidate propose/apply, create_and_disable_sources, and version-checked disable_note for reviewed near-duplicate merges.
Backlog update
BACKLOG.md
Clears the completed "Memory shape" open backlog subsection.

Estimated code review effort: 4 (Complex) | ~60 minutes

Unified graph MCP tool

Layer / File(s) Summary
Graph tool consolidation
src/omind/server.py, src/omind/seeds.py, BACKLOG.md, tests/test_server.py
Adds a unified graph MCP tool with an op selector; legacy graph-path/graph-orphans/graph-dangling/graph-stats now delegate to it and are marked deprecated.

Estimated code review effort: 2 (Simple) | ~15 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Client
  participant SearchIndex
  participant SQLiteIndex
  Client->>SearchIndex: search(query, limit)
  SearchIndex->>SearchIndex: _refresh_if_needed()
  SearchIndex->>SearchIndex: _scope(query, limit)
  SearchIndex->>SQLiteIndex: keyword and vector legs (depth)
  SearchIndex->>SearchIndex: rerank fused head
  SearchIndex->>SearchIndex: weight generated/superseded notes
  SearchIndex-->>Client: materialized results (excerpt_chars)
Loading
sequenceDiagram
  participant CLI
  participant Consolidate
  participant OmiStore
  CLI->>Consolidate: propose(limit)
  Consolidate->>Consolidate: find candidate pairs, build draft
  Consolidate-->>CLI: plan.json + draft.md
  CLI->>Consolidate: apply(plan_id)
  Consolidate->>Consolidate: validate plan, check source versions
  Consolidate->>OmiStore: create_and_disable_sources(draft, sources)
  OmiStore-->>Consolidate: ApplyResult
  Consolidate-->>CLI: merged filename + archived sources
Loading

Possibly related PRs

  • CryptoJones/omind#179: Also modifies the hybrid retrieval/search pipeline in src/omind/searchindex.py, touching the same index/query schema and fused ranking logic.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/retrieval-backlog-batch

Comment @coderabbitai help to get the list of available commands.

@CryptoJones
CryptoJones merged commit c05fce2 into main Jul 27, 2026
14 of 15 checks passed
Comment thread src/omind/hooks.py
# Fixed operational/persona notes above remain pinned; generated and
# credential-looking notes are never promoted into always-on context.
try:
from omind import access, retrieve
Comment thread src/omind/hooks.py
# credential-looking notes are never promoted into always-on context.
try:
from omind import access, retrieve
from omind.store import OmiStore, parse_note
Comment thread src/omind/hooks.py
f"===== OMI capsule: {name} (dynamic core) =====\n{digest}"
)
added += 1
except Exception:
@CryptoJones
CryptoJones deleted the feat/retrieval-backlog-batch branch July 27, 2026 07:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment