feat(rag): index the Obsidian memory vault into Qdrant (semantic memory search)#77
Merged
Merged
Conversation
…ry search)
Phase 2B of the memory system: vault notes written through the memory-vault
MCP become semantically searchable via qdrant_search (same nomic space).
- plugins/rag: mount the vault READ-ONLY at /watch/memory-vault, nested
inside the existing watch root so the recursive watcher/scanner ingests it
with zero ingester config (same ${MEMORY_VAULT_PATH} the MCP mounts).
- rag-ingestion: hidden-path rule — skip any path component starting with a
dot, keeping vault internals (.obsidian/ workspace JSON, .trash/ soft
deletes, .gitkeep stubs) out of the index. Applies to both the periodic
scan and watchdog events.
- plugins/rag: raise llamacpp-embed --batch-size/--ubatch-size to 2048.
Root-cause fix for a live 500 ("input (556 tokens) is too large ...
physical batch size (current batch size: 512)"): embeddings require each
input to fit in ONE physical batch, and a 400-word markdown chunk
routinely exceeds the upstream n_ubatch=512 default.
- tests: 8 new (render: vault ro-mount, contract unchanged, CPU shape,
embed batch flags; ingester: hidden-path unit + scan + refusal paths).
Validated live on ordo-v2: vault ingested (points 1 -> 6, .obsidian/.trash
excluded), semantic query "how should dashboards be validated" returns the
agent-written vault note at 0.74 via the qdrant-rag MCP. llamacpp chat
block byte-identical (GPU-97fe65ee pin unchanged); only rag-ingestion +
llamacpp-embed recreated.
Co-Authored-By: Claude Fable 5 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 2B of the Obsidian memory system (follows #76): notes the agent stores through the
memory-vaultMCP become semantically searchable viaqdrant_search— one nomic-768 space shared by ingest and query.What
v2/plugins/rag/plugin.yaml— mounts the vault read-only at/watch/memory-vault, nested inside the existing watch root, so the recursive watcher/scanner picks it up with zero ingester config. Same${MEMORY_VAULT_PATH}var the memory-vault MCP mounts (one vault, two interfaces).rag-ingestion/ingest.py— hidden-path rule: any path component starting with.is skipped (.obsidian/workspace JSON,.trash/soft-deleted notes,.gitkeepstubs), on both the periodic scan and watchdog-event paths.v2/plugins/rag/plugin.yaml—llamacpp-embedgains--batch-size 2048 --ubatch-size 2048. Root-cause fix for a live 500 caught during validation: embeddings require each input to fit in one physical batch, and a 400-word markdown chunk routinely exceeds the upstreamn_ubatch=512default (input (556 tokens) is too large to process. increase the physical batch size (current batch size: 512)— CONVENTIONS.md, live).Validation (live on ordo-v2)
--gpus allcontainer from--source out/ordo.yaml: compose diff = only the vault mount line + the 4 embed batch-flag lines; llamacpp chat block byte-identical (pinGPU-97fe65eeunchanged). Applied withup -d --no-depstorag-ingestionandllamacpp-embedonly.documentspoints 1 → 6 (5 vault notes + smoketest);.obsidian/.trash/.gitkeep.mdexcluded; the pre-fix 500 on CONVENTIONS.md cleared on the first scan after the batch fix (Ingested memory-vault/CONVENTIONS.md).memory-vault/memories/user/dashboard-validation-by-content.md, the note Hermes itself wrote via the memory-vault MCP in Phase 2A.🤖 Generated with Claude Code